For all intents and purposes, you are a college student, so it is understandable if you're not wanting to spend as much time with people who are still in high school. Take this time to make some friends of those freshman ladies, and especially sorority members.
It takes a very long time to build up a positive reputation, but you can earn a negative reputation in the blink of an eye. Your conduct at the fraternity houses may be completely innocent, but if you're getting the evil eye from freshmen sorority women, then they are getting a less-than-desirable impression of you. Maybe you're getting too chatty with someone's love interest or ex-boyfriend. Maybe your conduct looks unladylike. Maybe you ticked them off somehow when you attended high school together. Honestly, it doesn't matter: if they are snubbing you, they probably don't like you.
Their impression of you will most certainly affect your recruitment, as these ladies will represent that majority of the chapter make-up in their sororities next fall.
Find another weekend activity and save the fraternity parties for next fall. You can maintain your friendships with the fraternity members, but do it over lunch at the Student Union, going to a movie or cheering them on at the intramural fields.
Also, on an unrelated note: Think about applying to a variety of colleges, not just the one in your hometown. It's a wonderful growing experience, and you can always come home to take classes at the local U in the summertime.
