...My wife euphemistically told me then that it was not one of my best...
Don't worry, I haven't written a song yet that my wife calls "good"
I think the song works as it is. You create a mood that fits your lyrics perfectly, and your vocals are very good.
As for the structure, the last 3 lines of each verse sound like a pre-chorus to me, so I was actually expecting a chorus, but like I said, it works as it is now.