OP - I never had sex on a 4th date (usually it happens in a month or two, eg. 7-8 dates). But I would absolutely want to visit his place, to see if he's tidy, doesn't have a GF or a wife, and just spend some time in private. You are not teenagers to kiss in pubic all the time.
I've always been open with men telling them that I'm not ready for sex. But it doesn't need to be sex - there are things in between, too!