All in all, what it boils down to is what you feel is right
If the only way you'll wear his lavalier is with yours and he doesn't accept it, that's bad
He should take into consideration of your feelings toward the issue.
BTW, I like the idea of the charm bracelet!