Lots of video cards, APG and PCIe, to choose from.
Once you narrow it down...double check the PSU requirement and go from there.
Re RAM...check you system DOCs carefully and by all means look at sites like CRUCIAL.COM. Sometimes it best to add RAM in pairs to maximize system performance.
Before you go too far...check to see if buying a new unit may be better/cheaper than upgrading...adding RAM, adding new video card and maybe a new PSU.