Don't worry. It's never too late to change the habits.
But you need to have clear rules. And then you need to stick to them. Even when it's grossly inconvenient to you and DH.
On that note, we all know that TV is sometimes the best "babysitter" when you need to take a call or hop in the shower or have an adult conversation. But if you spontaneously offer TV because it's helpful to you, you're setting yourself up for disaster.
Kids that age REALLY need clear rules and boundaries so they can adjust their expectations accordingly. "We only watch TV sometimes" is about the worst thing you can tell them. They will always want to know (or try to convince you) that NOW is the time for it.
So . . .
I would scale back slowly but with very clear rules.
(1) DH and I decided on a new rule for TV. You can watch one show in the morning when you wake up. But no TV when you get home from school. That's the rule.
(2) If you whine and cry about TV, you lose your show for the next day. [This includes whining to watch TV when it's not the scheduled TV time, and also includes throwing a fit when it's time to turn the TV off. Specific and direct consequence for whining and crying about TV -- no TV the next day.]
(3) Get a timer that he can see and use. If there are other times you decide to let him watch TV (weekends, random times etc.), ALWAYS set the timer. Tell him that when the timer goes off, the TV goes off, too. AND . . . if he whines and cries about it, he loses his TV time the next morning.
This is the timer we like best. We use it for A LOT of things, including helping them be patient while they wait for us to finish something else:
http://www.difflearn.com/product/Time_Timer_PLUS/timers_counters_clocks
(This version seems way more pricey than ours. But the fact that it counts backwards visually is KEY. The kids can easily see how much time they have left so they're not surprised when it beeps and the TV goes off.)
The most important thing here is that you set a SPECIFIC limit on each instance of TV watching, every time BEFORE the TV goes on. Not "You can watch a little TV while I XYZ." And not "Ok, it's time to turn it off now" without having told him up front how much he can and can not watch.
And the second thing is that you stick to what you say, every single time. No changing your mind when he asks nicely for "Just one more show." Or when you realize you need more time to finish something. He needs to learn that when you set a TV limit, you stick to it. EVERY SINGLE TIME.
And finally, here's the phrase that truly works in our house: "When you whine about X, you lose X." Or "When you throw a fit about Y, we take away Y." EVERY SINGLE TIME.
Good luck.