You'll be FINE. I was a 26/28 and never needed 2 seats - I carry a lot of weight in my belly/hips and could get the arm rest down fine. Crowded into my hip, but I wasn't spilling over -- it more went out forward on my body.
When I got down to around a 20/22, I no longer needed a belt extender on any plane (and some planes didn't need a belt extender at a bigger size). Again, I carry it in my hips so if you carry it differently, you may even be smaller.
Travel is a zillion times easier now that I'm average sized (12), but it is not that big of a deal at a 20. More where things are hard for you (scooting by people, bending over in a full row if you're in the window to reach something far under the seat, etc), not where you're inconveniencing others.