When I joined my local chapter of KDR we had about 18 or so members, then I became rush chairman and made it grow to 30 something members(which is one of the larger houses on my campus). now I'm runing for president on the platform that I built the house for the most part.

so my advise, is join where you can contribute the most. If that means smaller, then that means smaller.