Children at that age need schedules and routines. They don't adjust to changes very easily. If the gym was something that was not normal or usual, it's going to be scary and overwhelming for a lot of kids.
Kindergarten on the other hand, may be difficult the first week, but once he learns his schedule and routine, he'll adjust and be okay. Then he will probably have problems when you do something off-schedule or outside of the normal routine. But all of this is a part of the maturation process. Each time, he'll adapt and as time passes, he'll have fewer issues with changes and adjustments.
You can't use a one-off situation that is new and different to set your expectations.