That would drive me insane. Maybe draw up a schedule with breaks. My kids take a 30-60 minute break around lunch time. And then they are often done by 2pm with other classwork. We have the opposite rules - have to be outside for at least an hour a day to get tv. If you want to take a longer break - 2-3 hours to do an activity then your son could do schoolwork 9-12, outside 12-3, homework 3-6.
Yes, school is almost over as many posters have pointed out but this problem lies deeper to me. Have you told your husband how difficult you find that? Did he used to work outside of the house? Maybe he is missing having minions to control so transferring it over to you? You need to stand up calmly for your parenting and talk to him about all of it. A good marriage requires painful communication
