I have PTSD from volleyball in gym all winter every year. The worst. Everything mentioned above. Horrendous.

You can watch volleyball for beginners on YouTube and practice with her.

My daughter tried volleyball for the first time at 11. While she wasn’t completely afraid of the ball, she didn’t really go after it either. One way she got better was to lie on the floor and set a slightly deflated ball to herself. She’d listen to music or have a show on in the background. I’d also toss it to her and have her try to bump it back to me. Simple drills that did not take much effort.

She also didn’t have the upper body strength at the beginning to get a serve over, but worked at it all season and by the end was getting her serve in.

Give it time and if she’s interested, try to practice with her. If she’s not, it’s a rec season so it should be short!

Does your DD care? Does she want to go to practices and games? If she isn't enjoying it I might just cut your losses and stop going. Presumably it didn't cost that much if it's rec? If she doesn't hate it and is saying she wants to get better, then I would look up some drills she can practice at home. It doesn't sound like practicing with the friend who's really good would be very helpful for making her feel better about herself.

I hated volleyball in PE. Serving was about all I could do properly. Funny thing is I apparently look like a volleyball player (tall and athletic build, but not super muscular). I still get asked in middle age if I played!

I coach volleyball for CYO girls. It is a lot of fun and most girls come into it without any previous experience.
We spend a lot of time on serving. It really doesn’t take much strength to serve with an underhand. Overhead is another story. Most of my 3rd graders can manage it > 50% of the time after mid-season.

Just encourage her to have fun and make contact with the ball. Any contact. Agree practice at home can be helpful. Even though you don’t play you can help her by just tossing the ball to her.

Get her a chess set

If she wants to stick with it, there is really no way to develop skills than to practice. And practice a lot more outside of whatever she's doing with the team. There are YouTube videos that break down the serve, set, spike etc. that are incredibly helpful.

if she doesn't want to stick with it, but would still like to play a sport, I can relate. My DD tapped out of volleyball in 9th grade because her skill level wasn't there and she didn't like the sport enough to improve. She isn't a kid who likes sports that have a "you're on!" aspect, or anything where you're face-to-face with an opponent, fighting for control of an object. If yours is similar but you'd still like her to try something athletic, maybe cross-country? If she's a waterbaby, sailing or rowing [lots of running involved there] or even outrigger? I know it's not available everywhere [it does exist in the DC area though] but we've found the outrigger community to be the most positive and supporting of any sport my kids have tried.

Third option -- don't do a sport outside of school PE. Not everybody has that interest, and she'll turn out fine.