I wouldn't. I have two children, both a result of ICSI, and both have mild issues. We don't have these issues in either of our families for as far back as we know so we think it was ICSI that caused it. My son has ADHD and sensory issues and my daughter seems to have the same (she's still young but am seeing signs of it already).
Not to be argumentative, but why tie the ADHD and sensory issues back to ICSI specifically? Maybe they were caused just by going through IVF, or the drugs you took during IVF, or whatever health concerns existed that caused the infertility in the first place, or environmental exposures in utero or after birth, or even just coincidence.
I'm using IVF too, so I'm in the same boat and not trying to be critical of that at all. But there's so many other factors that are a part of fertility treatment and just giving birth to a baby that I don't know if we can point the finger at ICSI every time a child has a health issue. Particularly ADHD, which is so common now regardless of a child's method of conception.
With that said, for the OP I'd say that maybe you shouldn't do it, not because of risks to the fetus but because it's just another thing to pay for. It sounds like you're already against it, so if you are okay with the small risk that you may have fewer/no eggs fertilize, skip it. It's your money and your eggs.