Ive never read the Hannibal Lecter series, just watched the films, the films seem to suggest that it was his own patients that eventually drove him mad, is this explored further in the books