I think wearing it to a game is a bit out of place. But then again I get weird looks and often critical comments when I wear my Red Sox cap while jogging. I think wearing it out of a stadium is fine, but in a game might best be avoided.

Maybe OP is not American? When I bring my family to games they have stupid questions like this.

Wear the gear of one of the teams that is playing, or just wear normal attire. People who wear fan gear from teams other than the ones who are playing just look ... odd.