I agree with ktsnake on the point that 50-60 isn't a big difference versus 80-90.
Don't go for a house just because it is big or small. Go for the whole package. Find where you fit in best. Just because a bunch of girls are calling the smaller house "dorks" doesn't mean that's what they really are or that you wouldn't fit in there.
To somewhat relate as best I can, my freshman year I hung out with a "big" fraternity on my campus. They were the jocks and the big men on campus, and I was friends with a bunch of them because they lived on my floor and I was the only girl on the soccer team with them. I was invited to their Christmas date party and had a pretty good time hanging out with the guys.
A week later, I was asked to another fraternity formal. This fraternity didn't have the best reputation on campus. They were known for always having the top GPA, being crappy at sports, and having members who always sat in their room having LAN parties. I went despite barely knowing the guy who took me and my 2 friends, but I had a BLAST. I literally knew 2 people other than my date and co-dates, and I had 10 times more fun than I'd had with the "cool guys".
Moral of the story? Sometimes it's more fun to be a geek than the big men on campus?
Not exactly. But I found a group of friends where I had a great time, and found out that they weren't exactly the stereotype. And their stereotype has changed. They're still smart guys who actually manage to graduate in 4 years, but they've gotten better at sports and have some great times.
Figure out where you fit in. I can tell you that I wouldn't be happy in a sorority where I got all the prestige but felt I couldn't relate to my sisters. If the big house is the full package, go for it. But don't be afraid to go for a group just because it's got 20 less members and a bad rep with a few chicks.