I just tried to duplicate this and I could not.

As I understand it, you want it to go up a perfect fifth, and it goes down a perfect fourth.

Since it worked for me, I think we need to see the song you are using, or have you list the exact steps. BIAB has many ways to transpose. I don't know if you are transposing all parts, just the melody, etc. And where are you doing it? In the space between key signature and tempo? In Edit, Transpose? In Preferences, Notation? All these are possible places to transpose.

Once we know that, one more question: is there anything unusual about the range of your song such that BIAB might be using some intelligence to keep it from going into too many ledger lines (doubtful, but not to be ruled out yet)?

And what version of BIAB do you have, and what build?

BIAB 2025 Win Audiophile. Software: Studio One 7 Pro, Swam horns, Acoustica-7, Notion 6, Song Master Pro, Win 11 Home. Hardware: Intel i9, 32 Gb; Roland Integra-7, Presonus 192 & Faderport 8, Royer 121, Adam Sub8 & Neumann 120 monitors.