Depending on how old your toddler is, you could do the secret door tour at the Mansion on O street. It would only work if you could put him in a carrier because otherwise there's way too much stuff s/he could get into.
Great Falls either side. Md side you can walk to the look out and back. VA side you can do the lookouts and do a short hike through the woods part. Its about a 30 min look. Not sure how old your toddler is, my kid could do the hike at 3.