There are actually 3 Dexter books..

They are :

Dearly Devoted Dexter
Dexter in the Dark
Darkly Dreaming Dexter

My girlfriend has read all 3 and she said that they compliment the show perfectly, as the tv show has a lot of different stuff going on and isn't based specifically of the events in the book. So it's worth reading them.
She also said that the people come across exactly the same in the book as they do in the tv show, especially Dexter. So even with already having an image of Dexter from watching it on tv, it fits perfectly with the book's Dexter and you can picture everything in the book as it happens on TV... Usually she's the same, she doesn't like watching movies then reading the book of it, or the other way around. But she said reading the books just makes her love the tv show even more, and watching the tv show makes her love the books even more..

If I had time i'd get stuck into the books as well..