Could you go on your own and leave the baby and your husband at home? And fly to the nearest airport and rent a car?
My dad lives in a similar type of place (rural with little to do) and I always just waited for him to come up here and visit us, which was maybe a couple of times per year. We did a family vacation for a week in July and I had no further plans to visit for a while. My stepmother called that he wasn't "doing well" and was in the hospital and I decided to go visit this coming weekend. I got another call the day after Thanksgiving that he was being discharged on "hospice and palliative care" and there was nothing else to do for him. Needless to say I was in shock. I dropped everything, loaded up my kids and flew down there. We spent Friday-Monday with him and he passed away two days later

It's been just over a week since he died and I still cannot believe he is gone. I have to say, the ONLY thing that has kept me from completely going off the deep end is that I flew down there and had four days with him to say goodbye. I keep thinking if I had not changed my flight, this weekend would have been too late and I would have never been able to get past the guilt.
I too suffer from anxiety and depression and feel overwhelmed at the thought of being out of my routine, travel, I HATE flying- like I literally spend the entire flight thinking we are about to crash and it is awful. But I am so glad that I somehow was able to put it all aside and just go down there. If you can make it happen, make it happen. Don't put it off and don't wait.