Hi there,
Firstly I'll go over the player settings, then the amp and speakers.
Player settings
================
Digital Output, via optical for none HDMI amps:
------------------------------------------------
PCM - (poor)Not really worth using as you can loose multichannel output, but great if you are using a HDMI equipped amp.
Digital (re-encode) - (good) This will re-encode the source audio to a DTS stream always. Even if the source is Dolby Digital (DD) 5.1
*Digital (audiphile) - (best) Pipes the audio source to the amp.
I found out that I can use the audiophile setting and any DTS-HD, Dolby TrueHD etc comes through fine! How my amp copes with these I'm not sure, but it could be because the formats are backwards compatible?
PCM Down sampling:
------------------
On - output is @ 48Khz
*Off - output is @ 96Khz (if source audio supports it otherwise 48Khz)
I have this turned off as my amp can cope with that rate.
Dynamic Range Compression:
---------------------------
Off - quiet speech will be quiet and loud noises very loud. Can lead to turning up and down the volume on the amp during watching the film.
*On - Pulls the quiet parts of the film up a bit so less volume twiddling.
Note your amp may support this to, sometimes it is called "Night mode". If you have it turned on in your amp, turn it off here.
Audio Source:
-------------
When playing a film it is *very* important to make sure you have chosen the right source. I watched The Matrix last night and it was playing in DD 5.1, not the Dolby TrueHD track that was available. I tested it again this morning with the TrueHD track and audiophile. The sound is much clearer and less "muddy" I found, seemed to be a better dynamic range.
It seems all films I have played default to the DD 5.1 track or PCM. Bit like DVDs always do and you had to select DTS explicitly (surely this could be a player setup, much like consoles specify desired audio output?).
Amp & Speaker Settings
=======================
Amp mode
---------
I set my amp to the optical source input, 6.1 on (it processes the signal to make a 6.1 field for 5.1 speakers, makes the sound field a bit wider imho).
Set the DSP to Sci-Fi or adventure etc, depending on the film.
Speakers
---------
I have some mission cinema speakers which are really nice
However to get the sound balanced rightly I did the following:
An audiophile friend of mine said the front L/R speakers should be at ear height when you are sat down, and the rears at ear height when you are stood up.
This seems to work well for me and my room which is a square and I have 1.5 metres behind my head to the rears. You may have to twiddle depending on your speakers locations.
To balance the sound I played the test tone so it cycles through the 5 speakers. I then adjusted each speaker so that from where I was sat they had the same perceived level. I then added a slight notch up on the centre, to try and cope with quiet speech parts (The Dark Knight, I'm looking at you!)
Observations
=============
bluRay discs seem to have a slightly quieter audio than DVDs, so I have to turn the amp up a bit more than normal. Also due to dynamic range compression you may notice some audio sources sound "punchier" than others (esp compared to PCM and HD sources). This is an artefact of the encoding since the quiet sounds have to be brought up in volume before encoding, otherwise they will get "lost" due to bit accuracy. (bit like having a nice 24 bit colour picture of outside with a blue sky with slightly off blue bits in it (the speech), then converting it to a Jpeg, the detail in the sky will get lost unless it is "enhanced" first.
Secondly the re-compress seems to make the sound a bit muddy for my liking, but this is as I'd expect if you are re-encoding on the fly.
For good films to test it against,
The Matrix - Big gun fight scene and Neo fights Agent Smith in the subway.
Wall-E - May "only" be DTS but some good surround effects.
Probably more but not had chance to test them yet! 
Hope this helps!