I still wear my letters occasionally. I just turned 30, but still look 20 apparently because that is what people often think when they first meet me.
I mainly wear letter sweatshirts here and there, but not to relive glory days. It's just part of my wardrobe--if it is clean, warm, and hanging in my closet or folded in my drawer then I'll wear it. I do still wear some rush and party shirts around the house or to the gym or when running early morning errands. So while I still wear my letters, I dont make it a point to wear them everyday or several times a week like I did when I was an active collegian member
As others have stated, if people giving you crappy looks about it, then the problem lies with them, not you.