The sync problem with the RC-50 is a known issue. It's definitely an inherent glitch in the looper.
I am with Scott. Convert all the loops to audio and load them into the looper and control them that way. You can output the click separately to monitors.

Note: To put the click in sync with your track you have to adjust the master tempo NOT just the phrase tempo.
BIAB has an awesome soundtrack function that helps with the output of the audio so that it stays consistent with your click. Aligning the tracks to the click this way does away with the need to record a count in. You can set the RC-50 to do that for you.

Good luck.

FC