It sounds to me like you are recording mono source into a stereo track.

In the window where you select the inputs, be sure to select either side....right or left as opposed to "stereo" input. In my rig, I have several options and one of them is "stereo".

ALWAYS, always select only the ONE SIDE.... in my case, just for no particular reason, I always ALWAYS choose the LEFT Saffire audio input. (Saffire being my interface's name)

When I forget that step .....and BTW.... it defaults to the stereo input in my rig....and I hit record.... I see a stereo wave getting recorded and on playback I can not pan to the opposite side.... because there is no audio information in that opposite side.

Ensure that you record mono source to a track set up for ONE SIDE MONO and things should work perfectly. The mono track will be naturally recorded dead center and will be pannable to either the right or left side of the sound field.

Doing what you describe in the OP..... be sure that again, you have the tracks (plural) set up as mono..... one right, one left in the Tascam. Send the signals into the Tascam and ensure that they are going to different tracks. You will end up with 2 mono tracks that you can pan as needed. I can do this with my 2 channel Focusrite Saffire interface..... for example, one channel is voice, and the other is acoustic guitar..... I can place each into a mono track independent of each other at the same time. Afterward, I can pan either as needed to either side. The key is to ensure everything in the chain is mono, from the signal to the final track.

Should you mistakenly record into a stereo track in the DAW..... most DAW's have the ability to create mono tracks from stereo tracks. A simple conversion gets you there.

You can find my music at:
www.herbhartley.com
Add nothing that adds nothing to the music.
You can make excuses or you can make progress but not both.

The magic you are looking for is in the work you are avoiding.