I think the crux of the problem is fans of programs like the one Berkeley seems to be using and affirmative action in general will point out that it is unreasonable to expect a rich white kid and a poor black kid (to generalize) to get the same fundamental education and that a lower SAT score is not an indication of anything other than less opportunity for academic growth in K - 12 education. And that they should not be excluded from higher education because of essentially where they were born.
Naturally anyone who did get a decent education prior to college loathes the idea that someone who is less qualified in their minds getting something instead of them when they've worked hard towards that goal for years and years.
and there is no easy answer.