Guys, before removing your Ram cards, u should first find out if your system runs Dual channel. Usually u can tell just by the ram card slots on the MB. Since u guys have 4 ram slots, they usually color code the ram slots. 2 slots will be 1 color while the other be a different color. More then likely, u guys have a system that can run Dual chn mode if this is the case. It will only run Dual chn mode though if u have eith 1 pair fille or both pairs filled. If u take out only 1 card, your system will run single chn which will make your system run slower.
As for the Video card problem with the 9600GT, I believe u need a PCIe 2.0 (New PCIe) slot and by looking at your system, u only have a PCIe 1.0 (original PCIe). If this the case, u are possibly choking the card as it may require that PCIe 2.0 in order to work properly. Choking meaning it needs the bandwidth that PCIe 2.0 has which is 16 GBps. PCIe 1.0 is at 8 GBps I believe. So basically u are only using half the card if this is the case. Surprised it even works if u have a old PCIe slot.