Lazs, incest directly creates things like retardation as a result of things like recessive genes coming to the front, XXY pondings, etc. It's genuinely harmful to the child, and thus directly negatively affects others.
Is there an equivalent to homosexual marriages with which I'm unfamiliar?
BTW, you use the example of "those are the rules, tough". I'll note that until partway into the 20th century, women couldn't vote. Those were the rules too. Using the existance of a rule as rationale for the correctness of the rule is a circular argument.