By Latex I gather you mean a very thin pair that will still allow good finger / hand feel and movement. There ain't any. Xylol or Xylene or solvents like that will just permeate through such thin material.
You will need a thicker viton glove and even then it will eventually get trough.
From: Occupational Safety and Health Guideline for Xylene
"The following materials have been recommended for use against permeation by xylene and may provide protection for periods greater than 8 hours: polyvinyl alcohol and Viton. Materials that may withstand permeation for more than 4 but fewer than 8 hours are Teflon and polyethylene/ethylene vinyl alcohol ..."
You won't get thin "Latex like" gloves for this stuff. I last used xylene/xylol decades ago and one should think seriously as to why you need to use it. What ever you want to dissolve, try another solvent for it if you can.
Try meths first and even petrol, you yanks call it gas ,
or turps or kero. For that you can use prob normal household wash-up gloves. xylenes are quite dangerous, bloody good solvent though!