Is it time that you could do something meaningful with? For example you are not needed / paid by current family on Fridays, so you could pick up a nanny / tutor job on Fridays? Or they only need care 9-2pm so you could do after school help / care. Or is it that you would have 11am - 1pm free and you couldn't really do much with that?
If you can pick up other jobs, that seems reasonable (and might actually have your current family reconsider because when you say "no, sorry I can't do that I have another job then". They might decide having you on standby even if they don't need you is worth it). But, if the time is not usable, then I would move on.