No you are not crazy if you let this relationship go OP.
You are a Father, a parent to TWO young children who are going through a tough transition right now. They need you now more than ever + they must be your primary focus right now.
It would be too unhealthy for them to reside w/another Female who cannot give them the attention & care that they deserve at this tender age/time. They are probably already traumatized by their mother's actions, to feel not worthy in the eyes of another woman would be too destructive to their self-esteem now.
I say either keep dating your girlfriend, but not move her in or if you are truly looking for a positive "Mother"-type figure for your kids, it is time to give her her walking papers and look for someone new.
There are plenty of women out there who love kids and could love yours too.
Best of luck to you and the kids.
