Well, remember, AH is an all or nothing damage model. If every single bullet hole was graphically represented on your airframe, your plane would probably have been riddled with holes. However, only your ailerons sustained enough damage to pass their "breaking" threshold. The aileron "falling off" is really just a graphical representation of your aileron becoming so damaged (either the surface itself, or the linkage system for it) that it loses all authority as a control surface. If we had a gradual damage model system, more than likely your ailerons would have probably have lost over 80% (or whatever AH considers as "breaking") of their authority while other control surfaces, if hit, would have lost a much lower % of their authority.
At least that's how I've always understood the AH damage model.