I agree with what's been said here... lots of good advice.
Depledging/deaffiliating is a big decision, not to be made lightly - just as pledging is a big decision.
As James pointed out, you have two big things to consider: the 4 years of college, and the rest of your life after college. It's not worth making yourself miserable for 4 years in order to become an alum and tell your chapter where to stick it when they start asking for donations.

But only you can decide if you would be unhappy ... if it's just a couple of guys, the benefits of being in a GLO and the friendships you'll form with your other brothers may outweigh the effects of these guys.
I had some big problems with a couple of my chapter sisters, and thought about deaffiliating more than once. But I realized that by deaffiliating, I would be throwing out the good with the bad. I would be severing relationships with those of my sisters who were my good friends (they would still be my friends, of course, but not my sisters), and I would be giving up the opportunities for social activities, charity work, networking, etc. that being in a GLO brings. So I stayed, and I'm glad I did.
Have you talked to your pledge trainers about depledging? Would you want to join a different fraternity? If so, what are the odds that you'd be able to? (Some fraternities at my school just *do not* take anyone older than a freshman.)
Oh, and your pledge brother who called your girlfriend a bitch, needs an ass-kicking in the worst way