UWS all the way. UES is weird - it's either billionares or just out of college kids. UWS is a little "newer" and "hipper" (as hip as the UWS can be) and definitely is more solidly UMC (other than the few projects left...). Ridiculously kid friendly - you can't escape Uppababies and scooters on every corner.
Great private schools and public schools but a lot of families send their kids to privates on the east side.
GREAT restaurants. Way more amenities (Citarella, two of the best Whole Foods in the city, Trader Joes, Fairway, Zabar's). Tons of gyms and workout classes, tons of after school activities for the kids. You've got Lincoln Center, Columbia, Riverside Park, Central Park, playgrounds on every corner, Columbus Circle, the amazing new Nordstrom
Both are great though. $5 million will get you a lot farther on the east side unless you're willing to go way far north on the UWS.