The Ki-84 is respected because it's competitive enough with the late war planeset, but it has some nasty traits that require the pilot to pay attention and spend some time learning what the plane can do.
If you pull too much AoA, it will snap roll without much warning (compared to our other aircraft).
If you dive too fast, you will lose parts...quickly.

You will sustain more pilot wounds in the Ki-84 than in the N1K or Ki-61.
The elevator becomes ineffective a higher speeds that are still slow enough not to cause structural failure.
That said, if you learn to work around these weaknesses, it is very competitive.
As a fighter, I used to think the Ki-84 was superior to the N1K, but after flying them more I've changed my mind. I believe the N1K is superior because it doesn't have all of the weaknesses listed above. You lose a little bit of speed, but the ability to dive without fear of losing parts is a big plus, and the control surfaces remain effective past 400mph ias. The truckload of 20mm doesn't hurt, either.

The Ki-61 would be my favorite out of the three if it were faster. When I fly the 61 it seems like there's a "shoot me" sign on my back. A lot of people will drop what they're doing to attack it. Despite its superior diving qualities, it frequently happens that after a diving BnZ pass on a bandit I find a Spitfire XVI closing on my 6, someone who has dropped 10k ft of altitude for a shot at a slow target. The last time that happened I sent the XVI to the tower, but was ganged like crazy immediately afterward.
The A6M series is great for low CV battles, but it's not very gratifying to score most of your victories over enemy who only turned to avoid your faster countrymen. In this way it's a lot like the Hurri IIC: too slow to be effective on its own, scavenging on those who are already forced defensive.