Yes, B -> E sounds good, even if you're in the key of A major.
As you suggest, it's the power of the V -> I progression. If you have any doubts, play it as B7 -> E, and see if your ears hear it as functionally the same. If so, it's a V -> I progression in action.
Depending on context, you'll probably just hear it as a temporary modulation.