Yes, it's fine. On my 5th round of IVF, I did my normal (moderate) workout with weights 2 days after transfer, because I had become totally jaded about success. My DD from that round is now a first-grader.
You are absolutely fine. My RE used to say that all of the restrictions were mostly about what I would feel guilty about afterward, not about what would actually impact my cycle.