In case it matters, just moved from a Mac Mini 2014 to the M1 2020 and the delay is now (almost) gone. It's still there but you have to be really focused to notice it.