Of course you can be overdressed. I wear a suit every day at my agency (GS-15) because I like the way they make my body look and I tend to get cold easily, and I do think as a senior manager it's important to convey a professional image, but most of my peers don't wear suits. I think it all depends on your role, level, and field. Bing overdressed by "a bit" is fine at any level (e.g., wearing tailored slacks when jeans are the norm), but if everyone in your office wears jeans and casual blouses and you're coming in a conservative skirted suit and pantyhose, you'll stick out in the wrong ways.
Still, I'd rather be known as the overdressed person than the underdressed one.