First thing, condolences to you and your family.

I have three children myself and I would take them. I would just sit in the back for an easy escape (if need be) and it would limit their ability to see what is happening. If you take them to the grave site, again just stay in the back. It limits what they can see, plus if the 4 yr old will probably play with the grass and the bugs. I wouldn't worry about it too much, like some people have already said they are young enough to not remember much about the actual funeral. Besides if the wife is saying to take them you will end up in a huge fight if you don't.