I don't have legal advice for this I'm afraid. But it sounds like you have at least 2 kids? Is one older than the 4 year old? I'd talk to them both about safety, and enlist the older child to work with pedestrian safety as well. Have the kids hold hands with one another at least. It's not fail proof of course, but they are going to have to learn to look after themselves better if their dad isn't.
Because you are right. You can't make him responsible. It sounds like he's had odd issues like this while married so it's not new? Supervised visitation may be the way to go.
Why are kids afraid to ride with him? Has he done something while driving and/or had an accident, or are they feeding off your anxiety (or someone else's)?
What kind of car and/or car seats does he have? Are they generally safe? Is he good about buckling them in properly?