I think a lot of it has to do not with what kind of activities are being done so much as the attitude of the teacher. The first 2 gym teachers of my schooling were horrid - our elem. teacher was a sexist pig (the girls had to do basketball layups while they boys played an actual game of b-ball, plus don't even get me started on some of his comments), and our junior high gym teacher was just a bitch. With both of them, if you were athletic you were the pet, if you weren't, they treated you like crap.
My senior high gym teacher was an altogether different story. As long as you showed up and were making an effort (got dressed, participated), even if you fell on your butt 9/10 of the time, you could get a good grade. I actually liked gym then, and I am NOT athletic. I think the whole key was the fact that she was sensitive enough to realize not everyone is Mary Lou freaking Retton!!
As Iowa Hawkeye said, we shouldn't take away competitive sports - there should be a mix of both. Plus, for some kids that struggle in academic classes every day, gym is the only place where they really get a chance to shine.