This fight won't happen right away for 2 reasons.

1. Lee will take a few more fights before he fights someone like Duddy.
2. Duddy isn't going to risk messing up his chance for a big money fight with Pavlik.