You probably don't even need to be the one to bring it up during rush conversation-- one year there was a girl on my campus who depledged from Chi O for reasons she made very public, then rushed the next year and wound up a TriDelt (both are hugely popular houses at my school). Anyway, towards the later rounds, most people recognized her as the girl who depledged the year before, but it really didn't have any bearing at all on how my house, or others, saw her.
Also, I believe she spoke the the greek advisor and was excused from going back to her old house during round one when every PNM goes to every house. If you are seriously uncomfortable about visiting your old chapter, you could talk to the people in charge at your school to see if this is an option.
Good luck!! I'm looking forward to your story
