The internal mute is for the television speakers only. The RCA feed is live - always - and is not controlled by the volume. The best way to mute, in your case the commercials, is by muting the amp/receiver that is controling the sound.
The mode not supported is showing up because you've already muted the internal speakers, and you can't mute the volume AND the internal speakers at the same time. The television sees that command as redundant and displays the message for that reason.
I have the LN40A550 40" LCD HDTV, which I currently have configured with the built in speakers turned off. I am using the analog RCA outputs of the LN40A550 to feed an input on my Yamaha stereo receiver. When I press the mute or volume buttons on the remote for the LN40A550, a "Not Available" warning graphic comes up on the screen.
Is it possible to configure the mute and/or volume functions on the remote to control the RCA audio outputs? I have tried a number of different configurations on the LN40A550's audio settings and I can't find a way to accomplish this. Help! I want to be able to mute the obnoxiously loud commercials without using a separate remote!