I find that to get the most from the real tracks, I have to pick and choose what sections I want. In other words, I don't just take the whole track and use it "as is". I use volume envelopes in my DAW to edit out the things I don't want to use.

I also find that occasionally, I hear a part I like but it's in the wrong place in the track. I use audio edit to cut it out and move it to where I want it. It takes a bit of work but it's well worth the effort.

I do understand what you say about some tracks simply don't work because they are too busy or don't fit the groove. I've deleted more tracks from projects than I wish to recall, for that reason. Somehow, it always tends to work out. I look for a different track that does work, or I simply do without. No one knows in the end but me.

You can find my music at:
www.herbhartley.com
Add nothing that adds nothing to the music.
You can make excuses or you can make progress but not both.

The magic you are looking for is in the work you are avoiding.