I'm from SB. Bacara is very far from downtown (it's really in the suburb of Goleta), and while it is lovely (my husband and I stayed there for a kid-free evening last year when we were visiting my parents), it is remote. The Biltmore (as everyone in town calls the Four Seasons Biltmore) is beautiful and right on the ocean but still a (short) drive to downtown SB. There is also the El Encanto, which will be reopening shortly-- in fact, having just checked the website, I think perhaps it has already reopened after a major renovation-- and it will be exquisite. But it is up on the Riviera and still requires a 10-minute drive down to the commercial area. Another lovely high-end hotel is the San Ysidro Ranch, in Montecito. None of the above are walking distance to downtown.
Or, my sister owns a beautiful VRBO property just a few blocks from the beach and a few blocks from downtown--
http://www.vrbo.com/454896. For complete privacy, a lot more space than a hotel room, and a lot less money spent, I'd probably stay there instead of a high-end hotel!
Have a good time in my favorite place in the world.