Yes, a nuclear reactor is a heat source. Various means have been used to get the heat away from the reactor such as liquid metals and other fluids. Space based reactors have considered using heat pipes and two-phase fluids. Water (as low tech as that sounds), which has been deionized and removed of impurities, has a half life of about 10 seconds and makes a great coolant.
At one time in the 1950's there was a nuclear aircraft being designed and built for the cold war. The idea being that it could stay airborne for months with resupply from other aircraft, similar to the concept of nuclear submarines. Unfortunately (or fortunately), water and lead are very heavy and the aircraft could never fly.
Basically, a nuclear reactor is a steam generator. The only 'trick' to GE and Westinghouse made nuclear reactors, aside from the waste we don't know what to do with, is making sure they have adequate cooling. That was the problem at 3 mile island in the 1970's. The night shift operators heard the cooling pumps cavitating due to low water levels caused by a stuck open valve and shut them off. Fortunately, the day shift arrived a few hours later and turned them back on, but it was too late for the reactor because part of it melted. Chernobyl was a completely different accident due to reactor design. Chernobyl exploded. Naval reactors are the same basic design as those used at 3 mile island, btw.
Regards,
Malta