The BDP-3600, as well as Samsung's other models, have numerous unresolved issues... the HDMI issue you are having is among the most annoying. You should be able to restore picture by unplugging the HDMI from the player and then plugging back in... this isn't a sure fix, but it does work about 60-70% of the time. I own the 3600, and I have sold/installed dozens of them and this issue is a constant pain in the neck. I also have run this unit to Samsung TV's, with the same result, and 'recycling' the hdmi has worked for me. I speculate that the issue is an authentication problem within the player... for whatever reason, the TV and the player become unsynced and lose their handshake.
Some other reasons the 'mode not supported' message would appear would include: unsupported resolutions/frame rates, a damaged HDMI, etc... the player's specifications output supported resolutions and frame rates for all Samsung TVs, and nearly all manufacturers within the past two years. I suppose one or more of the video processing chips may be malfunctioning, but that is tougher to diagnose. The conceptual design of the 3600 is great... when it works, it is one of the best players at it's pricepoint - unfortunately, the problems are impacting the performance and leaving a negative impression on the player and Samsung. In my opinion, the players were rushed into production and need improving. If they can get them to operate, without major issues, then that would be great... but I'm afraid that it will not happen with any '09 products... historically, this is very true for all manufacturers... all electronics will have issues, sooner or later, it's their nature to eventually fail; but Samsung '09 products have caused me more to have more service calls than any other in the past... let's just hope that '10 models opperate to full compacity with fewer issues.
The tech gave you some advice that I would not follow... the fact that you are running the HDMI to your receiver is not causing this issue, for I have installed dozens of these units, direct to TV, with the exact result. The problem is within the player, not your receiver/TV - I discovered that months ago. I would absolutely not settle with an optical cable to your receiver from the BDP3600... you will lose out on all of the audio benefits that blu-ray has to offer... optical cables can NOT support lossless audio formats such as DD True HD or DTS HD Master Audio - and for any home theater owner, these formats are a must! If your receiver does not have the decoders for the HD audio formats, then definately use the 7.1 analog audio outputs from the player to the receiver - you will get full HD Audio benefits because connecting this way utilizes the players HD Audio Decoders, one nice feature of the player.
The bitstream (audiophile) option is designed for HDMI transferring the audio signal... it really doesn't do anything more for you with optical than the bitstream (re-encode), because the audiophile option will eliminate the secondary audio track, and use it to 'reassemble' the signal to reproduce the lossless audio track... when set to audiophile, the audio signal bypasses the secondary audio decoders all together... but the optical can't transmit the assembles 'bit-for-bit' lossless signal. I hope this was helpful, good luck!