Scott,
You've nailed exactly what I'm working on right now. And it's really not at all so simple. It takes quite a while to learn the 4 chord types in all keys, and to move between the 3 inversions for the most common progressions ( say even only the I, IV, V and the II, V, I) of the major scale - in a timely (e.g. fast enough to jam and noodle while sounding musical). The motor control and memorization takes far longer than the 'conceptual' understanding of how and where to move.
Sometimes I think we musicians overemphasize learning things in ALL 12 keys (no doubt a good goal and one to gradually work towards) and I could see why - we want to be ready to play any song we come across.
But I think we often understate the importance of learning how to do many different things in one single key. How many musicians are playing the modes in all 12 keys from 2 to 2, 3 to 3, etc, without knowing how to switch them within the same Key Signature - and realizing how different they sound as we compare 'apples with apples', so to speak.