Could you offer to come a bit early and phrase it as you won't be on the clock until the requested start time, but you were really just hoping to get to hang out with him for a bit?