I think he rushed the relationship thus pressuring you to give him an answer immediately.
A 2nd date is much too soon to expect commitment from someone.
You're absolutely correct OP.....Relationships are a huge effort to maintain as well as a time suck.
You shouldn't have to sacrifice your future goals for a relationship that was created on the whim.
Talk to your boyfriend and let him know you are not ready to commit yourself to him yet. Tell him your plate is just too full now and it would be unfair to be someone's girlfriend when you acknowledge you really cannot invest what is needed to maintain a romantic relationship.
If this guy is worth his weight in gold, he should understand your dilemma and accept what you had to say.
Tell him you guys can still date & have fun together.