The Tempo Map should get used on all tracks, unless you opt out of it during certain conversions.
I know of no settings that would cause this.
Like I said, it usually just works here.
Sometimes something weird like bit/rate mismatch can cause issues, but your issue sounds nothing like that.

I do not work here, but the benefits are still awesome
Make your sound your own!