Don't forget that your son is used to going to the octor merely because you say so: that's what he's doing every time he has a well child visit. We took our child (5 at the time) to two dev peds. The first time I probably over-explained and said that just like she goes to the docotr to get her body checked out every year, at age 5, she has to get her brain checked out. For the second, I just said that [name of doctor] was going to play with her for a bit, and she never asked why. I called that doctor Ms. [name] rahter than doctor, because she didnt look like a doctor (no white coat, etc.), so there no reason to introduce that.
As far as your son knows, everyone does this. He doesnt have your adult concerns.