My thoughts:
Baseball as a whole isn't hard -- hitting a baseball is hard! Being a really good pitcher is hard as well.
Golf isn't a sport, it's a game, albeit a game that requires a lot of skill. It's not hard to become a 10-15 handicap, but to become a scratch golfer, that's a lot of practice and skill. It's also not a physically demanding game in the sense that you can be very out of shape and still be good at it.
Car racing and other motorsports are physically demanding and require great reflexes and coordination.
Soccer requires a lot of skill with the feet. Being in the peak physical condition comes with playing the game. But the ball control, while being whacked and pushed around, is a tough skill to master.
My picks would be gymnastics (all aspects), hockey (ice skating, handling a puck, shooting) , and polo (master horseback riders that hit a ball with a mallet while galloping around??)