I always thought it was 6 sent, however 2 of them were diverted to what became The Battle of the Coral Sea. This is from memory. The 2 diverted, I forget the names, were sunk or heavily damaged. I cant remember. Unfortunately for the IJN 3 of the 4 CVs they lost at Midway were considered, for them, large CVs. The USN describes such things named ESSEX, but for the IJN losing 3 fast, large CVs was pretty much the end of offensive naval air operations.
When people ask me when the Japanese really lost the war I always answered at Midway. Yamamoto knew it to.
My answer is four CVs.