Have you checked the list of formats that the 360 supports? It doesn't support a very wide array of formats unlike XBMC for the original Xbox.
I am having problems trying to set my vista media center to stream ripped movies into my Xbox 360. I edit my registry on my vista computer to play ripped movies but I am unable to play them on my 360. I?m trying to stay away from converting my movies to WMV or any other format and stream them from my Zune software, but anything will do as long as they look good on my 1080p TV.
I have my movies ripped as VOB files?

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic