National Gallery's policy is officially 12 (
https://www.nga.gov/visit/visitor-policies.html) to visit alone. However, I once sent my 12 year old there - we live within walking distance - and the guards wouldn't let him in, saying incorrectly that he had to be 13. But if you're actually in the building, I can't imagine that a well-behaved 10 or 11 year old would have any problems. At the worst, if a staffer asked, he'd just walk back and find you.
I don't know if the Smithsonian has an official policy, but I think similar ages should apply.