Hi

With buffer sizes the smaller the buffer the less the latency.
So IMO the smaller the better within the confines of not getting stutter and drop outs.
If it stutters, increase it a bit at a time say 50 or so depending who many steps you are prepared to try till it stutters no more , your latency will increase with each step.
As soon as you reach the right compromise stop. At this point I general add a tad more just in case the audio I am currently using in the future is a bit more complex than the test song.
In other words give your self a bit of margin for that odd extra complex audio function.
Just my thoughts
Mike

BIAB2021 UltraPlus,AsusN55S1Tbssd, W10/64,Akai EIEpro
Yamaha CVP405,SquireStrat, CoolsoftVMidSynth
Novatation Impulse61 Ctr kbd, Cwalk blab Kontakt

http://mikesmusic.byethost16.com/