Josh, you had a great experience and learned something important as well. Part of maturity is looking beyond the stereo type!
These labels apply to fraternities as well, you just substitute the term "fat" with nerds, Mama's boys, sketchy etc.
There is an LSU fraternity (actually several because there are so many) that has a negative label attached to them. Yet when it comes to a party and treating the girls like treasures, they are top notch! My daughter always speaks highly of them as do MOST of the DGs- (You will always have a bit of snobbery in any given organization).
Now, you can help them. When you hear a crude remark, speak up and say something positive. Maybe, just maybe some of the people around you will take a deeper look.