Why anyone in their right mind would say something negative about your spouse in your face?
For one, if he did something to them and they come to you for some sort of intervention then it would be reasonable to listen and act accordingly.
If they just don't like him and keep expressing it to you in different ways, it is not only inappropriate but damaging.
I am pretty sure you are an adult and you know where to seek for help or to confide when you need to but otherwise
them saying things at random is not something you should allow happening. Just let them know in a matter of factly manner that you find it offensive and inappropriate and you won't allow them to talk to you about your husband in such a manner so they have to choose their words wisely or you will stop interacting with them.
One exception is mother. ..

moms tend to overstep sometimes but they often do not mean ill and often one can take it with a grain of salt but also sometimes they can see something that one can not but even so it is never acceptable to speak in front of the husband and has to be done in most polite and gentle manner not rude and direct.
This is your marriage, your husband and when they offend YOUR husband they offend you. They do not respect what is yours.