It may be too that your husband has a lot of emotion and thoughts about the whole ttc process but doesn't really have a forum to express them as he needs to be sweet and caring and take care of you all the time. He doesn't want to add additional stress to you and he is probably already walking on eggshells so he bottles it all up. Then in the moment that is the hardest for him, it all comes out as anger.
I think you need a frank discussion at a calm time to make sure this is what he really wants. He he really on board with this whole process, does he feel he could say he wasn't comfortable with it if he wanted to. How far does he really want to go with TTC. It may be hard for him to see what it is doing to you and maybe he needs a break from the ttc rollercoaster.
I strongly second this. My husband is FULLY on board with the whole TTC process, including IVF - but he's removed from it in a weird way. Not on purpose, but just because, let's face it - the woman has to go through most of the crap. So, at times, my husband "acted out" in weird ways that I didn't understand until we were able to have a really honest conversation about it. That didn't happen by me nagging him or by some calculated strategy - just eventually we got to the point in our (very long) struggle that he finally was able to talk about it.
In the end, TTC has brought my husband and I closer together - but to get there, we had many weird moments / times when we just weren't able to communicate well and had some of these same issues. OP - I know it's so challenging to humor your husband for getting pissy at YOU when you're doing most of the work - but I do urge patience. If your husband is anything like mine, he's struggling with this too and just doesn't know how to express it.