What resolution are you sending from the Blu-Ray player and the PS3?
Are you getting "No Signal" or "Mode Not Supported" on the screen? "Mode Not Supported suggests that the TV detects a signal that is an unsupported resolution. "No Signal" suggests that the TV isn't receiving a signal at all, or a weak signal (or a bad cable).
You can try to change the settings of each player with both connectors attached in most cases, so change a setting in the player, and then switch from AV to HDMI to see if the picture appears.
I recently started having problems with my PS3 and Samsung (LN52B530) TV.
Basically, the PS3 will not work when connected to the TV via HDMI. The PS3 works fine when connected via the regular A/V cables. Additionally, my Sony Blu-ray player also does not work via HDMI (but does work via A/V cables).
I know it is not the HDMI cable (I have tried 4 different cables) and I know the HDMI still works on the TV (my xbox360 works fine via HDMI on any of the 3 ports).
The PS3 and bluray player themselves work fine via HDMI on my other TV so there isn't a problem there.
It doesn't have anything to do with output settings because I have tried manually adjusting them to basically all possibilities.
Any other ideas what it could be? I tried calling Samsung tech support and they weren't very helpful.