At the least, it's disrespectful to your marriage to talk about you negatively to other people. At the worst, it's a sign that he might be emotional cheating.

Either way, it needs to stop, and you two need to work on your other problems.

Most people really wouldn't care about this, OP, and say you're making a mountain out of a molehill.

But you seem to be very afraid of getting replaced by a female colleague. No one knows your husband like you do, so we can't tell you if he's likely to do that. But from where we're sitting, it's not probable!