This attitude I find strange albeit it seems not uncommon but only with particular tops. Before I go on, I'm older that the OP but I wear a Gers top playing football and going to Rangers games. I sometimes wear a top when on holiday but then I don't usually take holidays in the popular resorts so nobody takes much notice. .
Now back on my response to this poster FSM, I would doubt very much if you would be so anti-football top if it was some unknown or even just lesser known and less obvious team. People say nothing about other sports club tops such as US football, baseball and ice hockey.