Not at all. Why would it be crazy?
Train is a very nice way to travel with toddlers, since you can walk around on the train. Outbound from DC you'll be at the first stop on the line, so you can sit where you like. Coming back, if it's a busy time (like a Sunday evening), you may have some trouble sitting together (but try the cafe car, sometimes that has seats).
In NY, where will you be staying and how are you planning to get around? Their subway doesn't have elevators, so a stroller isn't the easiest, unless you have something lightweight.