I wonder if you can effectuate "alone time" with each parent while still having the kids all sleep in the same place--e.g. if the kids are with you Sunday-Weds, one Monday Dad picks kid A from school to do something alone. The next Monday Dad picks kid B up to do something alone. The next Monday, Dad picks kid C up. You do the same thing once a week when kids are with dad. The kid having "solo" time with one parent still sleeps in the same house as the other siblings.
Also, you know your kids. Do what's best for your family! Don't listen to everyone saying mean things. Divorce happens. It's not pleasant, but the fact that you and your ex are working together to find a good system for your kids is great!