Go slow.

I would go here.

I would get a 2GB kit of pc6400.
Remove old ram...install new ram....test.
If all ok...I now have more ram and faster ram.

Vid card.
I would go here.

There are many charts to help guide you.

Be aware that as you move up the vid perf level some of these cards need serious power and produce serious heat.

So there is the power and cooling issue to be considered.
Bigger psu?....more fans?