You contractor used cheap anchors, likely the plastic ones that install in holes in walls and expand with the mounting screws. The metal anchors like this:
http://www.homedepot.com/p/E-Z-Ancor-Stud-Solver-7-x-1-1-4-in-Phillips-Zinc-Plated-Flat-Head-Drywall-Anchors-50-Pack-25316/100391938
are so much more reliable.
To use the new anchors, you fill in the old holes, paint over the patches, and move the towel bars a couple of inches left or right. If you don't want to paint and the wall looks to be in good shape, you may be able to screw the new, bigger anchors into the old holes.