Wouldn't that be a bit like refusing to work with a Muslim on the grounds some Muslims are fanatical? Isn't the problem not the belief - racism, religion, Arsenal footie club - but the fact that a significant proportion of people want to have an excuse to destroy the lives of others?

No. The belief - racism - is exactly the problem here because it's incompatible with public service. The BNP is not a jolly group of reasonable people saddled with a few rotten apples who happen to have a thing for rocket launchers. The whole stinking barrel is full of people who want an excuse to destroy the lives of others, or have the state do it for them while they cheer: that is the only reason for the BNP's existence. I hear your plea for tolerance, but when fascists are tolerated, people get killed.