Well did you get in his face about it?
There's probably not a whole lot you can do but just let him come whenever he feels like it.