Judging Again
I was at an aerobatic club meeting recently and observed some of the discussion about judging. There was criticism of the standard of judging
and it seemed to me that criticism was directed at the Unlimited judges
of which I was one. We didn't use the Fairplay System where the judges
would expect an analysis of the scoring and a ranking of the judges - a
pity as us judges have no knowledge of the basis of that criticism. The
first that some may know about it may be when the proposed judging
committee comprising "senior pilots" decides that they are no longer
wanted at the contest. The Fairplay System would tell all where we stood
although I wonder how it deals with a majority of the judges not detecting errors worthy of a hard zero. (I admit to missing the odd thing while judging at the nationals - no-one says that judging Unlimited is easy)
Those of us who didn't compete and just participated as judges will probably respond in the following manner if this idea gets off the ground:- "I need to know a long way in advance of the contest whether you want me to judge or not. If not, then I won't bother with the judges refresher course and I will make other plans for that weekend. Please don't wait until the last minute as I would've already organised something else to do.
Good luck in finding judges for your contest."
I didn't participate in that discussion as I apparently got into enough
trouble some years ago with the comments in the posts below. That was when the aerobatic club instructed their webmaster to delete the link to my website - reminds me that I asked on several occasions what the reason for that was and still no answer despite the webmaster agreeing to provide the response.
--- In ozaeros@yahoogroups.com, ozaeros@yahoogroups.com wrote:
JUDGING - Are judges overworked or under-work?
http://www.aerobaticsoz.asn.au/0600news5.htm
"But it's hard to ignore the facts. At the last 2 nationals the largest variance in scoring has been in the Advanced Grade. Here the judges are arguably some of the most experienced judges in Australia - our Unlimited Pilots."
"I discussed all this with John Gaillard and he gave me some food for
thought with the following ideas:
Advanced and Unlimited pilots work only as assistant judges
The chief judge is chief for all grades and has no other administrative duties
Judges are on the line for a minimum of half a day
The only way your judging improves is with experience ie. volume If you're like me, you're thinking "but in the real world."
Well it's just something to think about - I'd be very pleased to hear from anyone with ideas on the subject. What do you think?"
__________________________________________________________
What this editor thinks is:
Whats the actual variance? Is there a difference in ranking?
I wouldn't want an advanced/unlimited pilot as assistant - my requirements for assistant are quite simple - enough knowledge to correctly fill in the score sheet and include my comments in the correct place; strength to hold the umbrella in a strong wind, remembers to bring sunscreen and water; doesn't talk when I'm judging.
And, more of what I've thunk on this subject is copied below. I was severely criticised for my comments two years ago so .....
Pilkington wrote back in 1999:
Just got my December AAC Magazine (well ahead of the IAC - only just
got October's Sport Aerobatics). I agree with Mr Magic's views on pilots
criticising individual judges. I was bashed up recently for my criticism
of some judges in general (I promise to only do that out of season). I
wonder if anyone south of the Murray has done a judge's refresher course
yet. I must admit that I got stuck halfway through it. I strongly
disagree with Mr Magic's remark "The absolute scores are irrelevant. It
doesn't matter if a judge scores you lower than every other judge. They
have probably scored every other pilot low as well. It doesn't matter as
long as they are consistent. It has been my experience in all the years
I have been flying that the best pilot in the competition has placed
first - without exception ...."
Firstly, how is "the best pilot" defined if not by the scores from judges who are correctly applying the defined judging criteria? If there's inconsistency between the judges' absolute scores there's a good chance that some of them are not applying the correct criteria but simply plucking a number from the air. Consistency is essential as well - the judging criteria are not that watertight that there won't be variations between judges - whether they apply the criteria for round loops harsher than others or whether the sunlight flashed off the canopy at the wrong time or, like me, just get it wrong (hopefully not too often). These judging criteria are not easy to remember, occupying 24 pages in my copy of the AAC Regulations. Its takes a lot of effort for a judge to remain up to speed. As I said in my earlier postings (eg #557) - some of our contests have very close scores. I mentioned a contest where the top four pilots had scores within 5%. The winning score was by 5 points in 4000. Who knows if the best pilot really won? We must accept that the system has limited accuracy and accept that the best pilot won. I don't want to know how the individual judges ranked the pilots for that very reason. On the other hand we must ensure that the judges are reasonable.
_____________________________
Editor: David Pilkington
and it seemed to me that criticism was directed at the Unlimited judges
of which I was one. We didn't use the Fairplay System where the judges
would expect an analysis of the scoring and a ranking of the judges - a
pity as us judges have no knowledge of the basis of that criticism. The
first that some may know about it may be when the proposed judging
committee comprising "senior pilots" decides that they are no longer
wanted at the contest. The Fairplay System would tell all where we stood
although I wonder how it deals with a majority of the judges not detecting errors worthy of a hard zero. (I admit to missing the odd thing while judging at the nationals - no-one says that judging Unlimited is easy)
Those of us who didn't compete and just participated as judges will probably respond in the following manner if this idea gets off the ground:- "I need to know a long way in advance of the contest whether you want me to judge or not. If not, then I won't bother with the judges refresher course and I will make other plans for that weekend. Please don't wait until the last minute as I would've already organised something else to do.
Good luck in finding judges for your contest."
I didn't participate in that discussion as I apparently got into enough
trouble some years ago with the comments in the posts below. That was when the aerobatic club instructed their webmaster to delete the link to my website - reminds me that I asked on several occasions what the reason for that was and still no answer despite the webmaster agreeing to provide the response.
--- In ozaeros@yahoogroups.com, ozaeros@yahoogroups.com wrote:
JUDGING - Are judges overworked or under-work?
http://www.aerobaticsoz.asn.au/0600news5.htm
"But it's hard to ignore the facts. At the last 2 nationals the largest variance in scoring has been in the Advanced Grade. Here the judges are arguably some of the most experienced judges in Australia - our Unlimited Pilots."
"I discussed all this with John Gaillard and he gave me some food for
thought with the following ideas:
Advanced and Unlimited pilots work only as assistant judges
The chief judge is chief for all grades and has no other administrative duties
Judges are on the line for a minimum of half a day
The only way your judging improves is with experience ie. volume If you're like me, you're thinking "but in the real world."
Well it's just something to think about - I'd be very pleased to hear from anyone with ideas on the subject. What do you think?"
__________________________________________________________
What this editor thinks is:
Whats the actual variance? Is there a difference in ranking?
I wouldn't want an advanced/unlimited pilot as assistant - my requirements for assistant are quite simple - enough knowledge to correctly fill in the score sheet and include my comments in the correct place; strength to hold the umbrella in a strong wind, remembers to bring sunscreen and water; doesn't talk when I'm judging.
And, more of what I've thunk on this subject is copied below. I was severely criticised for my comments two years ago so .....
Pilkington wrote back in 1999:
Just got my December AAC Magazine (well ahead of the IAC - only just
got October's Sport Aerobatics). I agree with Mr Magic's views on pilots
criticising individual judges. I was bashed up recently for my criticism
of some judges in general (I promise to only do that out of season). I
wonder if anyone south of the Murray has done a judge's refresher course
yet. I must admit that I got stuck halfway through it. I strongly
disagree with Mr Magic's remark "The absolute scores are irrelevant. It
doesn't matter if a judge scores you lower than every other judge. They
have probably scored every other pilot low as well. It doesn't matter as
long as they are consistent. It has been my experience in all the years
I have been flying that the best pilot in the competition has placed
first - without exception ...."
Firstly, how is "the best pilot" defined if not by the scores from judges who are correctly applying the defined judging criteria? If there's inconsistency between the judges' absolute scores there's a good chance that some of them are not applying the correct criteria but simply plucking a number from the air. Consistency is essential as well - the judging criteria are not that watertight that there won't be variations between judges - whether they apply the criteria for round loops harsher than others or whether the sunlight flashed off the canopy at the wrong time or, like me, just get it wrong (hopefully not too often). These judging criteria are not easy to remember, occupying 24 pages in my copy of the AAC Regulations. Its takes a lot of effort for a judge to remain up to speed. As I said in my earlier postings (eg #557) - some of our contests have very close scores. I mentioned a contest where the top four pilots had scores within 5%. The winning score was by 5 points in 4000. Who knows if the best pilot really won? We must accept that the system has limited accuracy and accept that the best pilot won. I don't want to know how the individual judges ranked the pilots for that very reason. On the other hand we must ensure that the judges are reasonable.
_____________________________
Editor: David Pilkington
Comments
Post a Comment