I would def. try to stay on Mt Desert Island somewhere in SW Harbor or a rental overlooking Somes Sound because it would be less crowded/trafficky than Bar Harbor. But BH has lots of restaurants, boat tours, fudge shops, cute homes etc. so it wouldn't be TOO bad. We enjoyed a BYOB sunset tour of Frenchman's Bay on the Margaret Todd windjammer.
We stayed in a b&b on an inlet in Ellsworth because I wanted a central place from which to check out the Wooden Boat School in Brooklin, ME and the beautiful islandy area around Stonington, the gateway to Isle Au Haut.