The adjuster mechanism on whichever side (in the drum) is either full of shit or worn out.

You might get away with cleaning everything up but whilst in there plan on replacing the shoes.