This definitely happens, my ex was in a local fraternity and they all treated me like crap. I put up with it, but eventually got too frustrated (with him and the frat). KSUviolet is right, you and his fraternity guys don't have to love each other, but that does not give them the right to say such things.
He won't be in this place for the rest of his life. and I'm sure that you love each other, but you even said that you are different in terms of social circles and the like... are there other things that are different as well? 3 years is a long time.
I'm not saying these things to make you paranoid or upset...just pointing them out...?
