Have Fun

Make sure you get some instruction so you learn how to do things right and safe from the start.
.22s can be fun, but (IMO) mainly if its walking around in th great outdoors and plinking at stuff. I've found 22's get pretty old on the range after a while. Though marksmanship target shooting is likely a different story.
If you are limited to bolt actions, then a nice Enfield would put you in touch with the greatest bolt action battle rifle ever made and the historic staff of British might through two world wars. A Russian Mosin Nagant is probably the only WW2 rifle that still has abundant cheap ammo.
If it even remotely allowed, an M1 Garand would be it, or a Carbine.
If Bermuda is even remotely as cool as
France you could even go with a dreaded AR-15 platform. Easy to shoot, cheapish ammo -- a great civilian range rifle.
Charon