Statistical study: draw in relation to placing

Discussion in 'The Rehearsal Room' started by DublinBass, Mar 17, 2008.

  1. DublinBass

    DublinBass Supporting Member

    I think as one of the other threads mentioned there should be ways to hold adjudicators accountable. Having not even been within 3000 miles of the LSC areas, I have to say I think some of the complaints made against Mr. Horsfield are valid.

    Here is my explination....

    All else being equal there should be no correlation between draw order and placing (this would equate to a correlation coefficient of 0.00). the first band draw finished first, all the way through the last band finishing last there would be a correlation coefficient of 1.00. If the first band drawn finished last, all the way through the last band draw finishing first there would be a correlation coefficient of -1.00.

    -0.15 to 0.15 is usually considered neglible, as you get to -0.3 to -0.5 and 0.3 to 0.5 you get a significant correlation.

    Here is are the correlation coeffients for the LSC areas
    Championship (Horsfield) -0.50
    1st Section (Horsfield) -0.36
    Horsfield overall -0.36

    2nd Section (Pritchard-Jones) 0.02
    3rd Section (Pritchard-Jones) 0.23
    Pritchard -Jones overall 0.18

    4th Section (Newton) 0.01

    Obviously there are many other variables in effect, but statistics show that this weekend, when Horsfield was adjudicating, there was a definite trend towards bands draw later placing higher.
  2. andywooler

    andywooler Supporting Member

    how about an alternative view; there was a definite trend towards the bands drawing later playing ,much better than the earlier drawn bands?
    I mentioned the draw stats earlier out of interest rather than an adjudicator criticism - I would suggest that
    is a somewhat unreasonable statement to make. Statistically interesting but certainly not enough in isolation to condemn the judge!
  3. DublinBass

    DublinBass Supporting Member

    That's one of the problems with statistics...there always will be those wierd cases.

    BTW, I noticed Horsfield also judged the Welsh areas. Here are the correlations for those sections
    Second 0.16
    Fourth 0.64
    Overall Welsh 0.31
    Overall LSC & Welsh -0.06

    Showing that over 4 sections there wasn't a significant correlation, lending more credence to your theory that bands that played later played better.

    I'm just trying to suggestion that when we have statistical tools that can be used over the course of a year to determine potential issues with adjudicators why don't we use them.

    As a teacher, I am often judged on statistics (my students' standardised test scores). Why can't we see if adjudicators show trends that they shouldn't, or if they match with other adjudicators better.

    However, as you pointed out and is verified, doing stats for one contest is a bit risky, there really needs to be a good sized pool of data to draw from.
  4. andywooler

    andywooler Supporting Member

    As I was reading this, I also had something work related from Cisco arriving which contains a very valid point: "without data, you are just another opinion".
    An interesting Masters study for someone on this maybe?
  5. MoominDave

    MoominDave Well-Known Member

    We could take a step in the direction of the next order of accuracy by correlating the results with the ranking table as it stood going into this contest. We see the following comparison for the top section(ranking table position followed by result):

    1: 2
    2: 1
    3: 10
    4: 4
    5: 12
    6=: 3, 8, 9
    9: 6
    10: 5
    11: 7
    12: 11

    Correlation coefficient: 0.39 - some correlation between existing placings and this year's results, but rather a weak one.

    For comparison, the equivalent number for last year's contest in the same section was 0.50 - certainly significantly more correlated, though not as much as I had expected.
    It would be interesting to see the variation in the correlation from one year's results to the next over long periods - and to see whether particular adjudicators' results showed lower correlations to the status quo than others!
  6. andywooler

    andywooler Supporting Member

    and doing that over a significant period of time could be an interesting comparison.
  7. Owen S

    Owen S Member

    Hmm. I haven't calculated it myself, but I'm pretty sure that with only 12 samples, you'd have to have a much bigger difference in correlation coefficient than 0.11 to be statistically significant.
  8. MoominDave

    MoominDave Well-Known Member

    Mmm, yes, almost certainly. Sorry, I used the word 'significantly' loosely.

    Ironically, it's exactly the same issue as exists with band rankings.
  9. andywooler

    andywooler Supporting Member

    My post earlier made it clear that the comments around placings vs draw had nothing to do with adjudicator bashing but are purely an observation. Indeed, this thread is so far refreshingly light on that sort of thing!
    Perhaps we should move the statistical related posts to a new thread looking at different ways of measuring results trends. Personally, I think it's worth looking at in more depth.
  10. PeterBale

    PeterBale Moderator Staff Member

    Good suggestion, Andy, and I think I've moved all the key ones over here - now let the discussions continue ;)
  11. Will the Sec

    Will the Sec Active Member

    "Correlation coefficient"? Beats me. I'm still waiting for a good explanation for "What is Radio?" :rolleyes:
  12. andywooler

    andywooler Supporting Member

    A good point Will - could you explain how we calculate this BBCBass? If we could spread the workload, it might be possible to amass enough data to come to a conclusion!
    Last edited: Mar 18, 2008
  13. DublinBass

    DublinBass Supporting Member

    Okay, here is the short of it.

    If you type two columns (array1 & array2)next to each other into microsoft excel (they must be the same length), there is a function that looks something like this


    Excel will calulcate the correlation coefficient for you.

    Less than -0.3 shows a negative correlation
    Greater than 0.3 shows a positive correlation

    - - - -
    If you want to see if there is a significant difference between two arrays you would use a t=test


    The 1 is for one-tail, you usually have an idea that there is a difference if you are testing things, 3 is for unequal variance (as usually they are)

    If you get a result of 0.05 or lower, there is a significant difference between the two arrays

    1) These tests do not show causality, but rather relationships
    2) You always need a decent sized-sample of data to draw valid conclusions
  14. David Mann

    David Mann Member

    Don't all these statistical tests assume a comparison with otherwise random events? The assumption is that adjudicator X tends to give statistically higher placings to bands later in the draw and that that is the only cause of variation. I believe that bands suffer from early draws not only as a function of adjudication but also a loss or gain of confidence due to the draw they receive.
  15. ploughboy

    ploughboy Active Member

    I think you'll find he's called Mr. Horsfield or David, And I didn't find it very poilte refering to him by just his last name, makes him sound like the accused! Which I hope he is not.
  16. DublinBass

    DublinBass Supporting Member

    That is exactly the point I was trying to make with one of my two disclaimers. Statistical tests can not show causality, there are many other variables.

    However, if enough data is collected to show that on the whole (with all adjudicators) there is no correlation between draw order and placement, there would be little support for the claim that draw order has any effect on outcome (be it confidence or any other reason). Of course, this has not been done yet here (but I do remember some article in the bandsman about that).

    *Sorry about not addressing Mr. Horsfield correctly. I appreciate that all people should be treated with proper respect (which I accidently did not).
  17. nigeb12

    nigeb12 Member

    Could also be deemed as flattery. Many great musicians are referred to by their surname. David is a fairly common name (no offence). Ludwig, Mr Mozart or simple Elgar ;)
  18. Anno Draconis

    Anno Draconis Well-Known Member

    And there's the problem; I'm not an economist or a statistician, but how can you draw conclusions about an adjudication based on how the placings correlate to the draw? Because of the "hoodoo" surrounding the number 1 draw, bands drawn earlier in a contest are often at a slight psychological disadvantage which manifests itself in a slightly less secure performance. So maybe some correlation should be expected.

    If you're saying that we should expect the results to be more "random", there's also the problem that people percieve random to be the opposite of sequential - it isn't. A random number draw is just as likely statistically to produce the numbers 1,2,3,4,5 in sequence as it is any other number sequence and it would still be random.

    Also, unless you're factoring in the quality of the bands playing, how much use is it? For instance, if the last four bands on stage at the Yorkshire area were Brighouse, YBS, Grimethorpe and Dyke, you'd expect the main placings to come from the last four draw numbers. Staistically this looks like a "skewed" result, but in real life, it isn't.
  19. MoominDave

    MoominDave Well-Known Member

    You need a bigger sample - take many contests from the same adjudicator together. Chance dictates that the random skew won't persist. And if it does persist - well, then we have a conclusion.
  20. Anno Draconis

    Anno Draconis Well-Known Member

    OK I suppose, but that still doesn't take account of the psychological factors. It's often easier (imho) to coax a good performance from a band playing last to a hall filling up with players in anticipation of the results, than from a band playing first at 10.00 in a cold hall to three men and a dog. That's why we all hate the "dreaded number 1 draw". So I'd expect more "late" performances to be in the frame on a regular basis because of psychological factors affecting the performance, rather than adjudicatorial (is that a word? ;) ) failings.

Share This Page