What confuses the heck out of me is hitting him with a wing and only the aileron falls off. Did you lose just your rudder, or rudder and vstab? If you were pulling or pushing, I could imagine a scenario where just your tail would clip the guy but if it's just the control surface, sounds similar to what I was talking about.
Wiley.