My DH (also ADHD/depression/anxiety) and my oldest (6th grader/ADHD/anxiety) struggle with what you do. I strongly encourage to you work with a CBT counselor (Cognitive Behavioral Therapy). Your brain is wired differently. It's easy to say things like, ignore it, turn off your phone, etc.) but your brain is pre-disposed to go to the worst possible end (at least, it is if you're like my DH and DS). You have to train your brain not to take the negativity path. The part of your brain that is responsible for reasonableness is weak. The part that is responsible for fear/worry is over-trained and over-strong. You need to get some balance back into your brain and to do that you have to exercise the part that's weak and rest the part that's overly strong. That's what CBT will help you do.
If you don't already have someone to work with, you might contact the Ross Center
http://www.rosscenterdc.com/ . If you're not near them, they can likely recommend someone closer to you. I know how much this impacts your life. I see my DH and my DS struggle with it and it kills me. I'm happy to say that they're both making great progress. You don't have to live like this.
One more thing - don't discount medication. We had to put my DS on Prozac for about 9 months because his 'worry' was too consuming. It had gotten to the point where it was controlling him. The medication suppressed it (not eliminated) enough that he could use the techniques he'd been taught. Once that cycle was interrupted, he made great improvements. I wouldn't hesitate to use it again if his anxiety dominates his life again. Hugs.