Anyone considering collecting WWII battle rifles should consider that it is not an inexpensive hobby.
Where to start? Mosin-Nagant 1891/30. Inexpensive, and even during the current ammo shortage, 7.62x54R ammo is plentiful and low cost. These are crude, but rugged and reliable rifles. The 7.62x54R cartridge is on par with the U.S. M1906 (.30-06).
http://www.aimsurplus.com/product.aspx?item=A76254R&name=Russian+7.62x54R+147grn+FMJ+440rd+Can&groupid=40Next? There's still plenty of Mausers on the market. You'll pay more for a wartime 98k, but Czechoslovakian vz.24 Mausers are plentiful and inexpensive. A good rifle to get one's feet wet with. I own three Mausers, all variants of the 98k.
Enfields... I own three. Not as inexpensive as they used to be, but great shooters. No.4 MkII models are probably the least expensive. WWII vintage No.4 MkI types are more costly, with the U.S. made No.4 MkI* rifles commanding high prices. There are a fair amount of Lithgow No.I Mk111* SMLEs out there. I own one dating to 1916, and another dating to 1917. Beautifully made, they are superb, shootable collector rifles. One of mine was traced to a Light Horse unit in the Middle East. I also have a Savage-Stevens made No.4 MkI*, Stamped U.S. Property on the receiver. Arsenal refinished just after the war, this rifle has all matching serial numbers and is to nearly new condition.
Of course, there are many different rifles you can buy, from mint condition expensive Garands to dirt cheap Chinese made M44 Mosin-Nagant carbines. However, all are fun to shoot and provide a certain historical ambiance when shooting them.