I'm pretty protective towards my letters, but so is everyone else I know, so I never thought of it as a big deal. As far as the party favor thing goes, my boyfriend has LOTS of Pi Phi stuff from date parties and formals, and so do the dates everyone else brought to those things. Those favors kind of represent the event they were at, however, and not the group hosting the event.
Likewise, I have about six Sigma Chi date party shirts and several more Derby Days shirts that I wear all the time. They don't have letters on them (this particular chapter only allows members and lavaliered girlfriends, plus their sweetheart, to wear their letters). But, like our party favors, they represent the event, not the organization. Maybe that's weird logic, but that's just how it is in my mind.
When my roommate was named Sigma Chi Sweetheart, I went out and ordered her a Sigma Chi lettered sweatshirt that said "Sweetheart" on it. I remembered some older Pi Phis who had been Sigma Chi Sweethearts having sweatshirts like this. My dear roommie was rather hesitant to wear it, and even checked with a bunch of guys in the chapter to make sure it was okay.
When they said absolutely, she began wearing it with pride practically nonstop. Now, in my mind, her wearing their letters was an okay thing because she had been selected to represent their organization as their sweetheart.