A USB interface device can be MIDI, Audio or both.
The MIDISport is MIDI.
In your case what is likely happening is the MIDISport is getting the MIDI signal to the computer plenty fast. The latency occurs when the computer tries to take that MIDI and turn it into sound.
This is the process where latency normally occurs; incoming MIDI needs to be turned into audio by whatever softsynth you are using inside the computer.

ASIO drivers were designed to handle this exact situation.
Audio Stream Input/Output.
^ That's the ASIO acronym; it helps create audio (sound) faster.

Your MIDI device is fine, it is the audio side that is slowing things down, most likely.
The aforementioned Audio Interface devices will likely help.
Every system is different and some play nicer with one device than another .. but I have used all the ones I mentioned above without issue.
The key is getting that MIDI to turn into Audio and get it back to the speakers/monitors/headphones as quickly as possible. Natively Windows is not geared toward this.
By design Windows (MME, WDM, etc) 'buffers' audio, meaning it needs a moment to do the work, look ahead and get the audio stream going.
ASIO is much faster.

I do not work here, but the benefits are still awesome
Make your sound your own!