Your assessment of it Notes is how I view things too.
Music as far as I am concerned should be for pleasure for both the player and the listener. If you are wanting to improve your standard, it comes as and when you try to apply it over time and yes you might push yourself to do so, though only at your own level and capabilities, not at someone else's pressure.
As you said, some things do leave themselves open to be a competition, running, throwing a javelin, jumping over a high bar, in fact anything that can be truly measured in an impartial way. Music to my mind can never be measured in that sense.
A player of say a trombone who now and then doesn't quite get a note correct and it is either sharp or slightly flat, against a player who nails every note bang on centre is a bit obvious as to who should win a prize. However at the top of brass banding where the adjudicator is sat in a tent and can't see the band or bands playing and can only hear them, I do feel there is certainly a human element of preference for the way a piece is played, rather than listening for skill of playing or even truly following the correct guidelines for passing judgement. It does all become a bit haphazard or chaotic and more dependant on the personality of the adjudicator, rather than anything that can be measured where everyone can agree the result is correct.