OP, there is respite care for families of both children and adults with SN for this exact reason. You are this woman's friend and I'm sure she and her family treasure that. You need to set some limits so that you and your family don't start to resent her position in your life, instead of seeing it as a positive. Talk to the family about respite care so that they can get the alone time that they need to recharge and make a concerted effort to change your thinking about this situation. Feeling as if you are being taken advantage of and that you should "allow them to have whatever time they want" is not a friendly relationship. This is a guilt/pity relationship. If it doesn't work for you, say so. Conversely, if there's a time that you'd like to see your friend, call them up.
If you happen to live in a MD county, here's a brochure to get you started.
http://www.dhr.state.md.us/documents/Brochures/Adult-Services-Flyers/Respite-Care.pdf