The Castle is near Goshen (Rte 17)-- maybe 15-20 minutes off of 87 north? Min-Golf, go-karts, climbing wall, arcade games, and the Goshen diner has pretty decent food. It would probably take you 4.5 hrs nonstop/no traffic to get there and then another 2 hrs to Albany so not really halfway. You can also get there by taking 81 north to 84 east so avoiding 95.
http://thecastlefuncenter.com/