It's your brother's fault, OP, not your SIL's.
You need to tell him that he's neglecting you, and that you're really disappointed. You need to point to factual examples (SIL's ignoring you, forgetting you and your kids and going to eat with other people, etc). Tell him you're sad you're not more important in his life, because he's your only family, and you used to be close. But that you cannot be disrespected in this way again, so unless he makes more of an effort to prioritize your family, you refuse to be at his beck and call.