I'd go one farther and suggest that anything smaller than a 37-mm be ineffective against large structures and ammo bunkers
There's a much more realistic way to deal with it, but it involves looking at ammunition types.
1) Solid-shot weapons -- machine guns and AP rounds -- are going to punch nice, round holes in things if they penetrate, but even with a light wooden-framed house, it's going to take a
lot of 3" diameter holes to even have a chance of knocking down a house. Machine guns should not damage structures. AP rounds
might be able to damage a house if they can hit something sufficiently structural -- not likely, maybe 1% chance every hit. There's one exception, though -- fuel tanks. An AP round is going to punch right through the tank and start a leak; any tracer round hitting a fuel tank with a leak, or any round striking a spark off the tank, would cause the tank to ignite and burn. (Actually, ammo bunkers would be an exception, too, but in the other way; the earth berm typically raised over the concrete shell of an ammo bunker would stop penetration of an AP round.)
2) Burst-effect weapons -- all weapons that carry a bursting charge for their primary effect -- have their blast effect applied against the structure's integrity; when the integrity reaches 0, the structure collapses. The blast effect is proportional to the weight of the bursting charge, and diminishes according to the cube of the miss distance.
A town building requires a 250-lb bomb to destroy it. HE bombs typically had about half their weight as a bursting charge, so the bursting charge will be about 125 pounds. The US 37mm M54 HE projectile had a bursting charge of 0.1 pounds; in order to get the same accumulated blast effect as one 250-lb bomb, it would take 1250 37mm shells. You can extrapolate for sturdier buildings.
Unfortunately, this is likely to be unpopular, as it pretty much castrates the Ostwind as a tool to take down a town for capture. On the other hand, in the pictures I've seen from WWII, where ground troops are moving through a destroyed built-up area, the destruction has been caused by either aerial bombing or bombardment by heavy artillery -- not by a light FlaK gun.