I telework 3 days per week, and up until recently, dressed in my old ratty clothes or what I wore to bed most of the time, just as you're suggesting. The yoga pants I was wearing were 10 years old and huge. Everything else I owned had holes in it - it was time.
I went to Target (yes, Target, I'm cheap!!) and invested in yoga pants "outfits". Now I have several fun pairs of comfortable yoga/running pants and matching hoodies, etc. They are cute enough outfits that if I leave the house for an appointment or something, I look like I at least "got dressed". It's super comfortable to sit around in, but on the off chance that someone does see me, I at least look presentable. I typically do a morning workout and then quick shower, no makeup on my telework days.
In summer, I suppose I will buy a couple of sporty t-shirts and capri legging/yoga pants.
It does make me feel better that I am dressed like an acceptable human being, yet still super comfortable. No need for jeans or makeup!