The Role of Measurement Error

A case study of how SCIENCE is IGNORED by Academic Institutions in determining grades. A grade is a measurement. A measurement is uncertain. WE IGNORE THAT.

For a lot of students, as well as the general public, there is a wholesale failure to understand and apply the concept that every measurement has an error. A large measure of scientific literacy in any society is there understanding that measurement errors exist and need to be known before policy is to be made.

There is simply no such thing as a perfect measurement or a perfect detector. All dectetors/measurements have random noise associated with them.

The effect of random noise is that no two measurements are ever exactly the same. Now if the noise is sufficiently small, the average person will not notice this effect in there every day life. Without such notice, the individual labors under the illusion that measurements/science are perfect Nothing could be further from the truth.

To begin with, we will apply the concepts of errors to something that you care about - your exam scores!

On any exam, your score reflects two things:

• Your ability in applying the material learned in this course
• Errors associated with how the exam was graded. These errors come in two forms; one is much worse than the other:

  1. Random error (can be corrected for - see below)
  2. Systematic Error (extremely serious if you don't know it exists)

  These Errors works like the following:

  

  What we measure is X but what we are interested in is the distribution of the true variable, T. To measure T, however, we have to know what the random error, e_r and systematic error e_s is.

  Without knowledge of e_r and e_s , T can never be accurately measured. This potentially is a huge problem.

  What is e_r

  

  Random errors increase the variability around the mean (in fact this process in nature is what drives genetic evolution).

  Random errors are associated with apparatus or method used in obtaining the data. All data sampling is subject to random error, period. There is no way to avoid it.

  Note that in most detectors random error is a fixed quantity so that the accuracy of the measurement depends upon the amplitude of the measured phenomena.

  For now, here is a simple example.

  Suppose I have a thermometer with a fixed random error of +/- 2 degrees. That is the accuracy of this device. Well, if its really 100 degrees outside, then this device will measure the tempearture to an accuracy of 2% (2 out of 100). However, if its 20 degrees outside, the device will only measure the temperature to an accuracy of 10% (2 out of 20).

  
  
  
  What is e_s ? (Systematic Error; often called calibration error).

  

  Systematic errors mean that different methods of measurement are being applied to the same variable. This means that the position of the mean is strongly effect. For example, suppose there are two patrolmen on the freeway both with identical radar guns. Except that one of them systematically reads 5 mph higher than the other due to a "calibration" error back at the station. Which policeman do you want to speed by?

  Systematic error is also known as BIAS and it plagues most scientific and social measurements. This is true in a lot of social measurements that are done without an adequate control sample and forms the basis for a lot of professional hostility between the natural sciences and the social sciences.

  Reliability of Measurements:

  In addition to the errors just discussed, there is another important issue with respect to the reliability of the measurement. Often times reliability is limited by instrumental precision.

  The example of your GPA as an UNreliable measurement:

  • Let's say your GPA is 3.72.
  • To qualify for the "There's an App for That" scholarship your GPA must be at least 3.75.
  • So you take that literally and don't apply.
  • Is 3.72 really 3.72 or is it 3.72 +/- some amount. Is 3.75 different than 3.72? Are you more qualified for this scholarship if your GPA is 3.75 vs 3.72? In other words, does having a 3.72 automatically disqualify you?

  • So how realiable is your GPA: let's see what it might depend on:
    • One term you take 1 class and get an A: your GPA is 4.0 - wow
    • The next term you take 1 class and get a C: Now your GPA is 3.0 - damn, shouldn't have taken that class, should have taken one in Military Science instead.

      clearly your GPA will depend on N - the number of classes you take.

    • Student X takes 3 classes one term and gets all B's because the professor only gives A's, B's and C's. Their GPA is 3.0

    • Student Y takes 3 clases on term and gets 1 B and 2 B+ - because that professor is less anal. Student Y has a Gpa for that term of (3+3.3+3.3)/3 = 9.6/3 = 3.2.

      Student X and Y are identical; the different GPAs are a consequence of grading precision and intrinsic ability or at least you can't tell the difference between the two.

  A tenent of statistics is that any average value has an uncertainty of σ/√N
  where σ is the standard deviation (basically the amount of random error) around and average and N is the number of measurements.

  For most UO students, the value of σ is around 0.4 and N is about 45 (the number of classes you need to take for your degree).

  So at the end of your UO experience, your GPA is determined to an accuraty of 0.4/√45 = .07

  But 1/2 way through your UO experience the accuracy is 0.4/√22 = .09

  So how is it that we report GPA to an accuracy of .01?

  There is no difference between 3.75 and 3.72 but your still penalized!
  
  
  Astronomical measurements are systematically plagued by potential bias that must be sleuthed out or understood to correctly record the phenomena and arrive at a reasonable physical model. Historically, the best example of this bias is the repeated reconciliation of (biased) observations with the geocentric cosmology in order to keep the earth at the center of the Universe.