It wouldn't suprise me if Taylor passed on the rematch for two reasons:

1. A fight against the Kessler-Calzaghe winner would be a big pay day and Taylor does want to fight at 168.
2. He dished out his best shots to Pavlik and still got beat (KO'd in fact). Must have been frustrating as hell. I doubt he wants Kelly again.