Mine was reading hot all the time recently and i was overriding the fans to keep the temps down, also when parked up it took a while for the fan to go off... Ive just fitted a 2nd hand rad tho, so it might not be greatly efficient! I solved the heat issue by changing the temp sender, turns out it was running around 88-90degrees when it was in the red block - faulty sender!
I'd get rid of the swirl pot, as the thermostat housing is a swirl pot in itself, bleeding via the small pipe at the top.
What exactly are you concerned about, because taking ages to go above halfway once up to running temp isn't a great cause for concern. But getting upto normal temp in the 1st place is.
How long does it take to warm up and then for the fans to kick in?