You're normal, OP.
In a situation where you're crossing and a vehicle is barreling down on you, you need to pull your children to safety. Hence the hand-holding.
It's a safety thing that everyone should implement with children who cannot evaluate and decide in a split-second. Which means all the way up to teens, basically
For red lights with no turn on red, this may not be such a problem. But we regularly use a pedestrian crossing in Bethesda without traffic lights, and here my children absolutely have to hold my hand tightly. One car in a lane will stop, and the danger comes from the second lane, since the car in that lane often cannot see us. So I am hypervigilant when using that crossing.