First, there are some "lovely decent men" who are white collar professionals and also amazing lovers. Don't judge a book by its cover.
Second, are you emotionally ready to be in a long-term relationship? If so, my view is that you need to be open to having a relationship for it to happen, and so long as you're fooling around with this guy that's not going to happen.
Third, sounds like he may be wanting more than you, and if so it's unfair to string him along.
-Signed, another 40-something divorced woman in the dating scene