My advice:
It would be highly unlikely for you to get along with or even be good friends with EVERYONE in your chapter. There will almost always be a few people that just aren't your favorites.
I'm sure you rushed this house for a reason.... I would say if this is the fraternity that you connected with, then you should give it a shot. Don't let the fact that there are two guys who you don't like keep you from doing something that YOU want.
I hope my opinion helps.
