Go. Easy flight so good for a weekend. Note that it is the south, so lots of things are closed on Sunday so plan accordingly. We rented a car when we went, so we did Charles Towne Landing State Historic Site on Sunday. Depending on what you want to see and how you plan to pack it in, the Charleston Heritage Passport may be worth it. It's a ticket you buy at the visitor's center that gets you in to many of the popular sites in the area. Be sure to visit the plantations in addition to the historic homes in town.
We went in January a few years ago - for the most part, I only needed my fleece coat and we were fine walking around outside. It's less crowded that time of year.