It could possibly slow it down.
HiTech
To be honest your statement surprises me.
I've done many benchmarks on particular issue and worked with lots of SW it almost never the case
The real performance advantage of x86_64 isn't the ability to use more than 4G RAM.
But significantly improved instruction set, more registers, update calling convention
For example in 64 bit first 6 parameters of function call are passed in registers
like in most RISC architectures while 32 bit version passes all parameters at stack
So unless you did tests and discovered the opposite, as general rule at x86 Vs x86_64 architecture the 64 bit wins with a good margin especially in computation intensive applications