Just some thoughts;
I used to play guitar in a 6 - 8 piece worship team with a drummer that played an early design Roland V-Drums set. The sound module claimed something like an un-noticable 8 ms latency. I guess that's the time from hitting a trigger to processing the sound through the headphone jack or other output. But, when he monitored only through floor monitors via aux sends from Front Of House "analog" mixer he always seemed late and everybody would follow whatever the actual latency was in the monitors. Then he would correct his trigger strikes by feel of stick on drum head to the band which would then sound late again and in no time everybody was way out of timing. Even though the his floor monitor was right next to his throne this still happened. I connected a KC500 keyboard amp from the sound module and put it right behind him facing forward. Somehow this solved the problem. Don't know why. I suspect he is naturally a late drummer and the keyboard amp just had a better feel than monitors. So, I wonder if some of us have our own human latency that adds to it all. Hopefully we can adjust that to stay on better timing. Kind of like learning to shake a maraca. You have to end your swing/snap a little early so the beads inside strike at the right time. Same might go for those sensitive to computer latency. Practice to get use to the feel, if that even works.

Does the noise in your head bother me ?