.NET incurs a lot of overhead. In the white paper he also noted the biggest gain in the move to 64bit was to allow the operating system to grow to the 8GB level in a couple of years, which is where MS sees it will be.
Anyone remembering the transition from 16bit to 32bit will also remember 32bit applications ran slower than the 16bit counterparts for quite some time.
Bigger is not always better. The bigger the datum, the larger the application, the more likelihood of a CPU cache miss which is where most performance gains are to be had.
Quite frankly, you would not believe how much 16bit code there still is inside of Windows XP.
Right now, 64bit is enjoying a lot of marketing hype, with some specific benchmarks done to show it is potentially faster, but those benchmarks have been virtually hand coded and do not represent the final product as virtually all developers will see it.