Neighbors in our old neighborhood hosted one each year. Here is what I gleaned from watching them:

1) Don't make the time frame too long or you will burn out. 3-6 or 4-7 works if you have children.
2) Have someone whose JOB it is to entertain and play with the children. 2-3 middle or highschool students would be great. Get a bunch of easy crafts at Michaels and have a room set up for children to do crafts (decorate an ornament, paint a model of a menorah, etc). Pay them to stay and clean up this play space after the open house is over
3) Space food out so that you don't put everything out in the beginning and then it's either gone or old by the second half when other people will come.
4) Make the food easy and well in advance (use warming trays or crock pots to keep things warm) so that you're not running around stressing. You want to be able ot socialize with whomever comes in at any time
5) Plan with a couple of sets of close friends to have them come early or later -- so you are sure of having people who will be lively conversationalists both at the beginning and in the second half

We really enjoyed attending their open house over the years and met a lot of neighbors and others of their friends, whom we otherwise would not have met. The main thing you need to do is to make this as simple and easy for yourself as possible, so that you can enjoy having your friends together.

Plan the open house AFTER your kids naps.

I've been to drop-in holiday parties where kids are invited, usually in the 5-8 PM time frame.

Informal.

We are finally at a point where I can actually start daydreaming again of hosting more get togethers. We have a new house and out of the toddler stage with our kids. I was thinking an open house for the holidays might be nice. But, I've never been to one and never hosted one! So, for those who have, I have a couple of questions: how formal, are kids usually invited, what time, how long, efc. Also, my kids still nap so what do you do about that? Any tips or advice would be most welcome!