The commute from Annapolis to the Pentagon is horrible.
I knew a person who commuted from Annapolis. he took a commuter bus, but he said it only works if you have very predictable hours. The computer bus picked up/dropped off at ONE time, so it's not like a train or metro, where there multiple times.
And I don't know where you think you might find work closer, unless you are considering working at Ft. Meade or the Naval Academy. There really isn't a huge job market in Annapolis. There are some law firms, some professional firms (architects), but there aren't many big employers. Most white collar people I know who live in Annapolis either work in law, the medical field, Ft. Meade/NSA/ Naval Academy or they commute to somewhere else.
I would wait until you actually get work closer to Annapolis to make the move. Because for two people, one commuting to DC and the other to Virginia, Annapolis does not make sense.