it's not cooler. solids transfer heat more efficiently than gasses, so while the rock is the same as the air, both are cooler than your body temp. the rock takes heat away from your body faster than the air, so it feels cooler.
try this one. put your rock in a sauna for a few hours, pump the heat in there up as much as you can stand it above your body temp, (it'll have to stay that way for awhile for the rock and air to be the same temp) go in there and the rock will feel hotter than the air.