Tell him that you think the real fun part is going up to neighbors doors. I can tell my son is so scared the first few doors, and then wams up and then by the end is just so happy. And then is so interested in the neighbors afterwards.
Also, tell him the plan to keep warm. Maybe he is more concerned that your child will get too cold, but won't want to admit it. Maybe plan to dress up well - or take breaks - or whatever works. Just make it clear that it will be safe.