I hope we didn't scare you away.
This is a thread that has some pretty good advice about dynamic recruitment and it links to another thread that also has a lot of good advice.
The keyword here is
dynamic recruitment. You need to meet as many guys as you possibly can and befriend them before you even bring up the subject of fraternity. I think you're getting a lot of "not my thing" responses because you're rushing the idea of a fraternity on them. Make friends with people and show them what you're all about (it's a lot easier than telling them) and by the time you invite them to rush events they already know you well enough to think that it could be their thing.
Also, I highly reccomend
Good Guys a book by
Phired Up Productions. Check out their website - they have a lot of excellent advice!