Both have their challenges. When visiting, you don’t have the comforts of home and are largely dependent on the generosity and planning of your hosts. When hosting, the pressure is on and you can’t ever really relax.
I think a big component of this is that no one in anyone’s family, nor do we, have a large home. I like to think that when we have more space, hosting will be more relaxing, or at least feel easier.