People definitely do. I am surprised that at our UES private there are many families from downtown. It’s not the majority, but in our grade there are 4 families I know of who live between Chelsea and Tribeca.
The main issue is really with getting together outside of school. You will really need to be the one to make an effort to come up to where most kids live for play dates, birthday parties, etc.