Once the fight is announced, whoever pulls out will be the one who didn't want the fight.

Simple really.