David, I may not be as groovy as you, but I pretty much agree. I usually won't even bother starting a song if I don't have a bit of lyric (with melody and a way to sing it rhythmically), but music most often drives the lyric rather than the other way round and neither gets too far ahead of the other. I've written other ways, so I know they too are possible from my own experience. But writing to a groove is absolutely my preference.
If it's not EASIER to write like that, it's definitely more fun. Rock on.