Being a member of a colony, interest group, or a newly-founded local is a lot of work. You have to set up your group's governing infrastructure, policies and procedures, etc. and (at least in the case of a colony) also complete your new member program. Sometimes people join, then realize how much work it is and drop out. Other people see that it will be a lot of work and decide to join a more established group instead, without even considering the colony/interest group.
If it takes a while to get colonized/chartered, people may get frustrated with the wait and drop out for that reason. The reason for the wait may not even have anything to do with the local group - it could come from lack of interest from the national org(s) you want to affiliate with, or from a Panhellenic Council refusing to allow expansion (for NPC orgs) - but that doesn't make it any less frustrating. Plus, you lose the momentum.
Even if no one drops out, numbers can be a problem. AEPhi told us we had to have 25 members to be chartered, but they gave us lots of help to get there. Some orgs (as CC1GC said) won't even colonize you until you have a certain number of members, which means you have to recruit up to that threshold entirely on your own, with no assistance from your intended national org.
Anyway, lots o' luck