I don't think your boss meant anything by it. If he has otherwise been polite and respectful towards you, I'd say it was innocent, if a bit misinformed. Maybe you ought to just say "Hey when you say things like that it makes me feel awkward. I know you may not realize it but I feel like I should tell you so that you know in the future." And then leave it at that. If he continues to say things like that, then maybe it's not so innocent.
Threaded Mode