From:
White-tailed deer - Wikipedia
The white-tailed deer is highly variable in size, generally following Bergmann's rule that the average size is larger farther away from the Equator. North American male deer (also known as a buck) usually weigh 68 to 136 kg (150 to 300 lb),[9] but mature bucks over 180 kg (400 lb) have been recorded in the northernmost reaches of their native range, specifically, Minnesota and Ontario.
The Florida Key deer are quite small. Also from wikipedia:
Key deer - Wikipedia
This deer can be recognized by its characteristic size, smaller than all other white-tailed deer. Adult males (known as bucks) usually weigh 25-34 kg (55-75 lb) and stand about 76 cm (30 in) tall at the shoulder.
IDK, but I've heard that it has more to do with body heat. In cold climates stocky builds retain heat better than skinny ones, in hot climates, skinny ones can shed heat better than stocky ones.