Ok, lets put it another way.
Its not about the hoody itself, your all missing the point...its about what it is associated with...hooligans and rowdy teens.
Even if they were wearing a suit & tearing the place appart, i would associate suits with hooligans...but also buisiness men.