If they wanted to hire someone right away, it could be possible that they already filled the position. It has nothing to do with your credentials; they just needed to fill the spot as soon as possible.
You need to talk to the manager directly. Call the manager and say, "Unfortunately, I am unable to attend the group interview sessions. I do not want this to hurt my chances of receiving this position because this job seems really interesting to me (or some kind of BS
). Is there anyway that you can possibly meet with me at another time?"
Hopefully, the manager will be willing to meet with you at a later date.