There was a lot in your post but playback here of almost everything is fine with VLC PLAYER. It tops what I get from a blu ray player in my opinion and my PC is a lowly laptop. It seems odd to read about all that horsepower in your machine and you have issues with video playback.
I'm using these two to playback my Anime & other movies, and I was always very satisfied with the performance. I used to use CCCP for my playback, but I heard it is "outdated", and the performance on madVR really is better.
However, I've a new PC now and encountered an issue I never had before - randomly without any clues I can recognize, my screen fills with a standing picture of vertical lines, all in one color, though the color is not the same brightness for each line, in other words theres enough contrast between them to clearly make out each line to the next. It always happened during playback of my Anime so far, which are residing on an external WD hard drive. They always resided there, and I never had this problem on my old PC.
My old PC was using the same playback method and was playing back Anime from external WD just fine. So I'm really dumbfounded.
My monitor is a Flatron IPS235, my GPU a Radeon HD 7870. I was using a Radeon on my old PC as well.
What did change was the OS. I was using Vista 64 bit, now I'm using Windows 7 64 bit.
I was always following this guide closely to set up MPC + madVR: http://haruhichan.com/wpblog/index.php/205/hi10p-info-guide.html
Also, I didn't encounter any problems while playing games.
The computer remains responsive, so far as I still hear sound in the background, and pressing "space" will pause or resume the playback. Only the picture wont change and I'm always forced to do a hard reset.
Any help would be greatly appreciated, I don't even know how to analyze this problem, is there some log file for the playback during that duration? Though I guess it would stop being responsive if it encountered an error. Monitor at fault then? But why only with Anime, how can I find out what he doesn't like about it...
At this rate I might try out CCCP again, though I would really prefer to keep using madVR.
I also tried disabling all Energy Managing options in Windows, like USB power save etc. It seemed to improve the issue at first, I didn't run into any problems for more than 20 episodes, but today I had the problem again twice in two successive episodes.