What nige said is the way it is, however there only angle is on health & safety grounds which they will need to prove, the y cannot just say it and expect u to follow it. I suggest u contact ur union rep if u have 1 or have a free consultation with a solicitor and see what he/she thinks.

Good luck oh and dont forget u have the right to have a colleague/friend/solicitor with u for any formal hearing, if its informal then there is nothing they can say as its ur word against theres.