While I am not an expert on this subject by any means, these are a few things that you might want to keep in mind:

Any moisture in your gloves will freeze causing the water to expand. This may accelerate breakdown of the padding in your gloves as the ice expands in betweend padding cells and stretching them out. If your gloves are leather, the natural fibers and cells in the leather may be damaged causing cracking to happen sooner. If your gloves are synthetic, exposure to the cold will make the material become more brittle and, again, cause cracking to happen sooner. The only good thing I can think of about leaving your gloves in the cold is that it will retard the growth of any potentially smelly bacteria in your gloves. If it's not too much trouble, I would take the time to keep them at around room temperature.