Appologies then I was confusing you with someone else.
It could be a bearing, they get worn with age and use, then start to rattle round a little, then wear out the carrier where they sit. This then needs milling out and a bigger bearing putting in if its on the inner case, or parts replacing if its on the shaft.
Only real way to tell is to strip it and have a look around. If the box is off the car how much play is in the shaft ?