Don't discard the headphone amp suggestion.
Many do what they do very well.
Plus most have lots of room around the dial; no problem adjusting.

I have a couple of these (both over 10 years old and still work like new) -
https://www.sweetwater.com/store/detail/HA8000

Well worth it, especially if multiple headphones are ever needed. That said, one of the ones I use has one set of headphones plugged in most of the time .. still glad I have it. Easy control, ability to add more headphones if needed, and does what a headphone amp is supposed to do.
If you have a CD due in a month, and you are not using trusted monitors on your computer while you mix, you got a lot of extra pressure.
If you have to rely on headphones, a good headphone amp may be well worth it.

I do not work here, but the benefits are still awesome
Make your sound your own!