Although I've posted on latex allergies before, my knowledge of this type of allergy is minimal, having no personal experience with it. So when I read this article from ProcessingTalk, I learned quite a few things.
Apparently, sometimes people who think they have allergies to gloves, are actually allergic to substances they've been into contact with before wearing gloves. Wearing gloves, or occlusion of the hands, just exacerbates the problem because the irritants are trapped and thus resulting to prolonged contact.

Of course there are those who are truly allergic to latex and rubber gloves, and this poses a big risk particularly to healthcare professionals who have high exposure to natural rubber latex materials. Aside from the rubber proteins, allergic reactions may also be triggered by other components such as the glove powder and rubber accelerators.

In powdered gloves, the rubber proteins which migrate to the surface of the glove material during Curing and become adhered to the glove powder. The glove powder, normally cornstarch, with the Latex proteins attached, can easily become airborne and may then be inhaled, thus worsening the allergic reaction.

Now rubber accelerators are new to me.

Rubber accelerators are chemicals used to speed up the manufacturing process of rubber. These chemical ingredients are present in almost every elastomeric or unsupported glove made. In particular, the accelerators thiurams, carbamates and benzothiazoles can cause allergic contact dermatitis (Type IV allergic reaction - delayed hypersensitivity).

This type of dermatitis is an itchy skin condition caused by an allergic reaction to a material - for example, natural rubber and nitrile gloves - after chronic skin exposure. This allergic reaction arises a few hours after exposure and only affects the area the glove material touched.

If you truly have latex allergies, the only solution is therefore to avoid exposure, and use powder-free and accelerator-free nitrile gloves instead of natural rubber gloves. For gardening purposes, there are cute Nitrile Garden and Work Gloves at Amazon, available in fun colors!