You don't have much in common with them?
That's what Greek Life is all about! Fraternal bonds with people who have different backgrounds than yourself!
My chapter had guys from suburbia, farmer's sons, guys raised in the 'not so nice' parts of town. Heck, we had a Japanese guy for awhile (hell of a guy).
The point is that no, I didn't have a damned thing in common with most of them except for Sigma Nu -- and that's enough!
If you go through life trying to tune out anyone that's not like yourself, you'll lead a boring life. Stick with it and in several years' time, you'll not regret your decision.
Oh and if you haven't noticed, many of the folks on this board have been with their respective organizations for years. Don't think you'll get a lot of folks telling you to cut ties and try to switch