I don’t think you need to break up, but I do think with many men you need to set hard boundaries around timelines. Many come around once they see you’re serious and will leave.
I’d figure out the timeline you’d like (when do you want to be engaged, married, have your child, etc) and let him know if he can’t meet that, you need to find someone who will. Not in a threatening way, but just a “I’m not getting any younger and this is what I need to be happy”.
How’s the relationship otherwise? If you’re having problems besides this, then bailing may be the right answer.