I think that we do worry too much, and that OTs make it their job to find something to "work on." Couldn't we all use greater core strength? There are also a lot of conflicting messages, like fine motor is related to gross motor, so if your kid has poor fine motor skills, then we need to work on gross motor. But if your kid has good gross motor skills and is able to carry through with multiple steps (e.g., hitting a moving baseball!), and is able to handle small manipulatives, then maybe the problem is that he *simply* is not motivated to write letters. In many countries with higher literacy rates, kids don't start writing letters until they start reading, around age 6-7.
FWIW, my DS couldn't be bribed to hold a crayon until he was almost 5. But, we knew he great fine motor skills (he could put together advanced Lego kits with tiny pieces) and that he had good gross motor skills. Fortunately, DS had a teacher that recognized that he just didn't want to write. Fast forward 5 years later, and he is an amazing artist, has great handwriting--and could care less about Legos.
I think that if all the other indicators suggest that there is no lack of motor skills, fine or gross, encourage it, but don't force it. And talk to his teacher about other ways to maintain fine motor skills.