"I though you wanted to keep the Audio and just toggle between them "

Ideally yes. But that might require deep re-programming logic. Unless the data in midi has info to fully reconstruct RT from scratch. Your idea of muting RT and replacing with MIDI is interesting, but again, I think it will require deep code & logic change as RT tracks would need a place holder for custom and/or default synth when switched to MIDI and wise versa.

What Charlie showed works for me as alternative, I just think these functions could be better integrated. Simple toggle at Track Settings or such, without extra menus/pop up questions.