I'm guessing you weren't around for the M4's introduction. It had almost entirely supplanted usage of the Panzer IV. This was before we got realistic optics as well, so the Panzer and M4 were on a more even footing. The M4 also shot faster back then.
But mostly it came down to armor. While the Pander has 80mm on the hull front, the M4 has 93mm with slope. This can be further increased if you angle the hull. And turret armor is much better, and the M4 is much easier to go hull down in.
Personally I attribute the M4's apparent lack of awesomeness to generally poor usage.