I didn't think too much about it (obviously, because I ended up pledging a colony!), but a lot of fellow PNMs that went through with me really noticed and immediately discounted the smaller groups. I suppose they thought small = weak, but that's not necessarily true--but still, a lot of people think that way. My chapter is actually middle of the road when it comes to size on our campus, but we're that way for a reason--we're sort of picky about who we give bids to, and we operate on the idea that quality builds quality. We want quality girls, who will attract more of the same. So we're not the biggest on campus (we've got about 20 or 30 less members than the largest), but I think we've proven that that doesn't matter as our house is full of trophies from Greek events.
So I guess the bottom line is that all PNMs don't worry about numbers, but quite a few of them seem to.