Not fighting an elite fighter in 5 years is hurting Dirrell