1. Are there enough swingers in your community that being mistaken for one could truly be problematic?
2. Even if someone misinterpreted your dress somehow, is there any reason you couldn't just say "oh no thank you, I just liked the pattern"?
Btw I think it's dumb when subcultures decided to appropriate common symbols as their own. Pineapples have long had a social meaning of warmth and home -- it used to be a common gift at a housewarming party and pineapple imagery in door decor was a way of conveying welcome to guests. If swingers want upside pineapples to mean something different, okay, but you can't expect it to also *stop* meaning what it has always meant.
So I think it would be dumb for you to stop wearing your dress over this.