It's tough to put my finger on just one thing. Being a part of any organization is a challenge.
When I was a rho chi, I had to explain to my new (non-greek) boyfriend that I had to pretty much disappear for a week for rush because I had to be available for my PNMs and "on duty" during parties. He didn't quite get that just because I wasn't squirreled away in a rush room didn't mean that I could hang out with him any time.
It was also tough deaffiliating and being unable to speak to any of my sisters or any non-freshman women on my dorm's floor (pretty much all the women on my floor were in sororities).
The year after I graduated, I came back to help with rush, and was treated very badly by some of the actives. I didn't have anything to do with my chapter, or AEPhi at all, for some years afterwards.
Probably the toughest thing of all was watching my chapter go through a reorganization. I was an alum, but I was still around for grad school. Some of my dearest sisters were asked to take early alum status; others deaffiliated outright (by their choice). My little was among them.
The chapter is stronger now than it was then, but it was still painful to watch.