At first I thought it might be a limitation of the motherboard. Mine only takes 8 GB... But that's not the case here, yours should take 16 GB. As you may have noticed, there's quite a few BIOS updates mentioning memory compatibility improvements.
According to the manual, sticks up to 4 GB are supported. Funny aside, the compatibility listings only show sticks up to 2 GB, the larger amounts being kits of 2 or 3 sticks. You suspected one 8 GB stick possibly being bad, so I assume you have 2x8. The manuals have been published in 2009 in English, the German update from 2011 showing similar data, so 8 GB sticks might be supported, too. As Oboe said, it's important that they are seated in the A1 and B1 slots, especially with the Lynnfield processor you have. Oh, and in the specifications I found there's a MemOK! button on the motherboard for fixing memory boot compatibility issues. It's located at the upper end of the ATX power connector.
As for your suspicion for reserving memory for the onboard video, there's no such thing on your motherboard. I scrolled through the manual for each BIOS option until I looked at the image and the specifications.
If you have any doubts about potential issues with the Windows.old folder possibly interfering your current setup, reinstall Windows for your piece of mind. At this point it still isn't too much of an effort.