Joe,
Why start with books if you don't want to read music?
Because it is simply more efficient.
Those who bypass learning to read the dots very seldom ever get back around to going back and filling in what they have missed.
Here we may not be talking becoming an accomplished sight reader of the kind who can sit down in front of rather complicated music charts and play them through the first time and have the chart sound pretty much as it was designed to sound, but rather someone who has the basic ability to know at least which note is being depicted on the two clefs of the Grand Staff and the basic Rhythm Value indicators.
The ear player, the improviser, the working and performing musician, and even the part time hobbiest will only find benefit through doing it the right way the first time around.
--Mac