Shaun man, don't let it get to you.
I go to a school with a 1% greek population, like yourself all my freinds and family hate fraternities. I could not understand it and got in some real heated arguments over it.
Unfourtunately for me, my greek life has to be segregated from my other non-greek freinds. I'm sorry to say, this point of conflict may always be there with your freinds since it hasn't gone away with mine yet
Take everything in stride, try to get them to understand that this is important to you and that it won't change your relationship with your freinds.