You're going to get a bunch of "it depends" answers in this discussion.

I myself was pushing 20 when I rushed and pledged. A woman in my pledge class was already 24, but only a sophomore because she too took a break from school.

Basically you know where you're at in life. Putting aside how competitive rush is at your school, ask yourself if you could see yourself "meshing" well with these straight-outta-HS freshmen. I know you won't be much older than those girls, but you'll most likely have more life experiences than they will. Your relationship with the sisters in the house is in important, but the bond a NM has with her NM class is unlike any other.

If you want to read about other experiences from "non-18 year old PNMs" there are a lot of them in the Recruitment Stories forum.