My ILs live in New York. In October, I went up to New York for a much-deserved weekend off. Took my 3.5yo daughter and dropped her off at my ILs' house in the New York suburbs, then took the train into Brooklyn to see my college friends. The last day of the trip, I was supposed to come mid-afternoon to pick my daughter up and drive home. MIL was home alone that day and called at 9:00 a.m. to tell me that my husband's cousin's wife had just gone into labor and she was going to the hospital to wait with them. She had called her cleaning lady, who agreed to come to the house and watch my daughter. The cleaning lady was on the way, MIL was on her way out the door. I was pretty upset. I made my MIL cancel the cleaning-lady-as-babysitter plan and wait at her house until I could get there by train. I arrived 1.5 hours later or so, at which point I was polite, but then bundled my daughter up, got in the car, and came home.
MIL has had the same cleaning lady for a long time. I've met her once. Still, I think, unacceptable. (And this is setting aside whether the cousin's wife REALLY wanted husband's distant cousin's mother to be at the hospital while she labored!)
This is the most extreme example that MIL is always making weird error judgments when she watches our daughter: locking them out of our house, setting off our house alarm and not telling us so that the cops show up, not putting DD down to nap all day (this was when she was <1yo), etc. Once, she brought a snack for DD that had peanuts in it even though she knows DD is allergic; she just didn't even think of it. Fortunately, FIL was there that time and caught her. And this is not to mention the little things like failing to comb hair or get daughter dressed, feeding her nothing but snacks all day, etc. After the no-nap incident, we didn't leave DD alone with them until she was around 18mo or so. And we do the no-peanut talk each and every time we see them to make sure it sinks in.
Now ILs are agitating again that I bring daughter up for a weekend with them. Also, I'm pregnant with #2 and they will want weekend trips with both kids eventually.
I'm trying to decide how wary to be and what conversation (if any) to have with the ILs. I think we've addressed the peanut thing so that she gets it. Other than that, I just can't even figure out how to approach the conversation because, well, how do you anticipate and head off these bad judgment calls? Clearly, they have really weird judgment, but (peanut thing aside) ultimately, I don't think my child is any physical danger. And my ILs are good people; I liked them a lot before we had kids.

So it doesn't seem to me that cutting them off from babysitting is the right answer. It's just this sense of impending stress that each and every time MIL watches DD, SOMETHING non-inane will go awry.