The problem begins with people who are of the belief that transgender people have a 'sexual problem' instead of having a gender issue then associating it with some form of sexual depravity instead of a missing gene,chomosome variation, dual genitalia for some or any mixture of many other related issues.
The one issue that gets raised is sexual depravity and is not linked any more with trans people than it is among so called straight people or gay people.

Usually the ones who speak up uneducated, secretly believe others past teachings that God doesnt make mistakes even through nature and the laws governing should reflect that, so they buy into a whole belief system of basic generalizations concerning anybody out of the ordinary having some evil undertone.

The issues run much deeper than any of their judgments could imagine as you can see in the link below.

TransGenderCare.com : Guidance/Transition : What is Gender