I recently bought this lightweight jacket from Evereve. The material is soft and stretchy and it's a nice summer weight, and it's slightly cropped so I'm able to wear it with higher rise pants/shorts or a dress without looking dowdy. Check out their jackets/blazer sections, you might find something there. Or maybe a "lady jacket" type sweater made of 100% cotton--those are all over the place.
Here's the jacket:
https://evereve.com/collections/blazers-jackets