Icecores, tree rings, fossils, soil & seeds, and in later history human accounts is the core of what we have about the climate in ancient times.
Oddly enough, some folks from the warming-denialist camp have been mocking the scientific effort to gather information.
As for the average temp in various places, where I live it is uppedy-up. So are the plants that we can suddenly grow outside. We're even growing plants that didn't thrive in the warm medival period.
By the way, to analyse what was actually being grown in medieval times up here, a team of scientists has been looking into soil from canals etc, - ash layers from volcanic eruptions give us a very good and "striped" calendar down into the earth. So, they look for seeds from certain depth. Accounts of Barley and Rhye being grown are now backed up, but today we are doing wheat. In ICELAND!