I've had a quick play with this option. I tried setting the limit to 1000, I have about 600 songs in the folder.

It now adds new songs automatically, but it does add a noticeable delay when opening the song picker. This varies, it's worse the first time, then gets quicker, probably because stuff is in the disk cache. For my case it's on the verge of unacceptable, so I'll see how it goes.

I think a new option to do the rebuild on save rather than every time the song picker is opened would be better. Save already takes longer and only happens occasionally, so the extra delay would be more acceptable.

I haven't tried deleting songs to see if they disappear. I would guess this is what causes the problem you are seeing. The auto rebuild is much quicker than the full rebuild done by the button in the picker, so there must be some optimisations that cause the problem you are seeing. Maybe something like only updating songs that are newer than the last time the list was rebuilt and ignoring deleted files. Just a guess.