A lot of people can't read Greek letters and I don't think I could before I started seeing fraternity and sorority stuff when I was in high school. I assume your shirt spelled out Kappa Sigma, so those people have no excuse.
As for the sorority formal shirts, it took me posting on GC to figure out why GLOs made event shirts for nonmembers to wear--regardless of whether there were Greek letters or the organization was spelled out.
It was very rare for BGLOs to make event shirts for nonmembers to wear. I didn't even notice a lot of the nonBGLOs wearing event shirts when I was on college campuses. When I did, I probably assumed they were just borrowing their boyfriend/girlfriend's shirt--which I thought was a silly thing to do.