Ride the ferry to Sausalito (and then check out PP's suggestion of the children's museum).
One of the sourdough bakeries has a visitor's center on Fisherman's Wharf - you can watch them baking bread, then eat samples. Touristy, but I think a toddler would love it.

The Children's Discovery Museum just across the golden gate bridge in Sausalito. Fantastic for toddlers.

The Exploratorium.

DO NOT miss the SF Zoo. I was there with a 2 1/2 year old, and it was so fun for him. You get much more access to the animals than the DC zoo, so even little ones can get close enough to see and there's a great petting zoo.

Head to the beach. The SF Zoo. Golden Gate Park. A baseball game?

There are so many great parks for kids in SF. I swear they are everywhere. We also took our kids down to Fishermans Wharf, rode the (doubledecker!) carousel, watched the sea lions, and took a cable car ride.

I will be going to SF during spring break with my toddler alone (26m). We will spend 5 days, including a weekend - I have been there twice, but although I love it, I don't remember many details (and I was single). Anyway, what can I do for fun with a toddler in town? I want to have a great time with my daughter in this mini vacation.