Its not about being stung with mysterious accidents outside circuits but a lifestyle/type of driver thing. In the same way they link your occupation (and hence character judgement) to risk, going on trackdays in thier eyes implies you like speed and thus are a higher risk on the roads.
I wonder how long before they start making us do proper psychometric testing before doing car insurance so they can risk load your premium based on your type of character? at the moment they only have occupation and relationship status to go on for that.