I'm 33 and I wear t-shirts and sweatshirts with my letters to the library (I'm a grad student) or any other place I feel like other than work (not because of the letters, but because I don't wear any casual clothing to work). I do reserve the coaches jackets/windbreakers for alumnae events and other events for which they would be appropriate.
I've never had anyone say anything about my age and my letters. Usually, I would get something like "Latinas join sororities?" if anything. But oftentimes, someone will come up to me and say "my cousin is in that sorority!" or something positive. I have even run into sisters around town who I did not know yet because they saw me in letters. That's always fun.
Maybe it is different for NPC sorority members, but I love seeing ladies well into their 40s, 50s, and beyond wearing their sorority letters when out and about. It would look funny at work, though.