I tried using Blu-Rays with VLC, and have the following results.
1. As long as the media on the HD is an exact duplicate of the disc (with sub directories and files), VLC plays most Blu-Rays happily.
2. Irrespective of what you do, VLC does not play an ISO copy of a Blu-Ray. Even if you mount the ISO as a drive.
3. In spite of whatever preferences you set, VLC sends audio as PCM which the receiver sees as Multi Channel. When the movie is playing and your right click, you get a set of options for audio, video, etc. In that if you choose audio device as 'A/52 over S/PDIF', you get bit-stream with full DTS and Dolby Digital. Every time you change the movie or shut down and restart VLC, the audio device resets itself back to stereo and PCM. These could be machine/driver specific and would need more research.
4. I inserted a copy of 2012 into the Blu-Ray drive. After shutting down PowerDVD that auto started, I loaded VLC and asked it to 'open a disc'. It refused. I then asked it to 'open a folder' and pointed it to the Blu-Ray drive. It happily played the disc. I think when you ask VLC to play from a drive, it looks for a DVD as the error log said it could not find and Audio_TS or Video_TS directories/files.
5. You do not get the initial menu nor any advertisements. VLC starts playing the movie straight away. Next, Previous, Pause, Stop - all work.
I tried the following Blu-Rays successfully on VLC:
Ayan
Flight Of the Phoenix
Iron Man
King Kong
Dark Knight
The Rock
Om Shanthi Om
Rocket Singh
I-Robot
Zorro
Live Free Die Hard
Mission Impossible
Night At The Museum
Spiderman 3
Transporter 2
The only movie that I had as a standard sub-directory/file structure that VLC did not play was Magadheera. I have to see if I can get a log file and see what happens.
Please note that I have not installed any special codec packs. What I have installed are the following:
1. 10.12 version of ATI Catalyst
2. VLC
3. KMP Player
4. AnyDVD HD
My HTPC has an ATI 5770 card connected to my AVR through an HDMI cable.
I think VLC is well on it's way to be capable of playing Blu-Ray in all it's glory. If they introduce Menu capabilities, and native bit-stream of HD Audio, you may not need any other player. Tomorrow I will download and show the media information that VLC sees when it is playing a Blu-Ray.
Cheers