I do not think they should have shot it out with them. the Iranian force should have been anihilated for crossing the border into a war zone before they even knew they were engaged. One laser designated JDAM should have sufficed, no need for a small arms shootout at all.
Iran and Britain are not at war, nor indeed is the United States at war with Iran. Crossing an imaginary line in the ocean is not a good reason to start a war. Equally the Iranians had no business taking the British prisoner whether the transgression is real or imagined.
I suspect this had been an embarrassing incident for the Iranian government despite their public stance. Already the Iranian ambassador to Britain has asked the British to help mend international relations with Iran. From other reading it may be that the whole incident was engineered not so much to get at the British but to cause problems for the President of Iran. He is not as universally popular in Iran as many seem to believe.
A war, even a limited one would be disastrous for Iran and the government knows it. The US does not need to invade to reduce the infrastructure to ashes. I'm quite sure their military has fully briefed Ahmadinejad about the consequences of hostilities. Also there seems to be a power struggle in Iran between the extremists and the moderates. This time the moderates won.
Whatever the reason, this has to be seen as a climb down by the Iranians. Despite Krauthammers nonsense. There was in fact a concerted campaign by other countries, including Syria, Turkey and other middle eastern countries which seems to have had an effect. One of the problems Saddam's Iraq had was no friends. So when the Americans came it ended quickly. That is a lesson Iran has learned.
Far too many people love the JDAM method of diplomacy. We all see how well that worked in Iraq!