York is less than two hours by train and lovely if you are looking for history but want fewer tourists than Oxford or Bath. If you actually want to live in a castle, this is your best bet
http://www.landmarktrust.org.uk/
Don't go to Dover unless you are catching a ferry to France.
Have fun, Dan (a londoner moving to DC)