I also have a PS3 and an Xbox 360 both connected via HDMI. Maybe try changing the HDMI ports they are connected to. Also for some reason i remember i was getting the same issue but i wasn't choosing the correct HDMI port. My settings are HDMI1=PVR / HDMI2=360 / HDMI3=PS3 with no issues. It HAS to work out...there shouldn't be any issues.
I have a new LN46A850 series TV and it is sweet. I also have a PS3 (latest firmware) connected to it via HDMI cable in HDMI 1 port on TV. I have had problems with the TV getting the "Mode not Supported" screen.
I have got this during the following times:
1. while playing a game on the PS3
2. while downloading on the PS3
3. and when I first start up the PS3.
Note: It has not done this during Blu-Ray movies
I have went into the PS3 display setup screen and have only selected 480P, 720P, and 1080P. As far as the HDMI cable I have it is one from BestBuy that they recomended I use. I am at a loss here and can use some help. Please share your thoughts and helpfull tips to resolve this problem.
Thank You!!

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic