Any time is a good time! Temperatures inside the caverns should be pretty stable year round, this summer we just had to put on a sweater once we got there. No biggie.
On a sunny holiday, it will get crowded so get there early with tickets bought online!
Also, the food offerings there are bad and I was particularly disappointed with the inn across the road, who made us wait for an hour to boot. Next time we go I was thinking we should either pack a picnic or try going into Luray itself to find another restaurant.
We liked the ropeway adventure thing and the car museum. You can get a discount if you buy the tickets to the ropeway with the Cavern tickets, I believe. Your 4 year old might be limited to the baby course, though.