My pledge class had a girl who dropped out after two pledge meetings (I think...it might have been less than that!). I still see her on campus and she wishes she had given it more of a chance but she's going to rush again during formal rush in the spring.
I would hang in there for a few weeks, even I who got the house I wanted was a little apprehensive about whether or not I fit in with the Pi Phis! It happens to everyone I think, and you just gotta try it out and see what is best for yourself. Good Luck with your decision!
Jess