There is one clear answer here.

Who has had any more fighters pull out or delay against them than Tyson Fury?

You may look at him and see what you think you see but actual pro boxers look at him and see a very difficult opponent to beat and a threat to their career.