She might be depressed, which can sometimes manifest as anger. If you're close to your brother, talk to him about suggesting therapy for her and suggest parenting books and classes for both of them (frame it as kids are really tough and my own really drive me nuts sometimes, here are some ideas that might help). Try to take the kids frequently so SIL can get a break.