Noisy Data

In general, most environmental data has a considerable amount of random noise it. The origin of this random noise is

1. the phenomena itself has a lot of intrinsic variability - this is especially true with climate data

2. the measurements are often difficult and imprecise

The example below shows a time series where the intrinsic variability completely swamps out any potential trend. To identify possible trends it is necessary to smooth the data.

A variety of techniques exist for this.


(blue line = smoothed trend)

Any data that has significant flucutations as a function of time may dominate the underlying wave form (trend with time). You therefore want to dampen out the fluctuations to reveal the underlying wave form.

The most common technique available is "smoothing". These techniques serve to surpress the noise and to extract real trends and patterns.

Note that smoothing and averaging are not the same:

• Averaging ends up with fewer actually data points
• Smoothing preserves the number of data points but lowers the amplitude of the noise by coupling the data points.

Techniques:

• Boxcar smoothing or sliding average: this works best when there is some underlying waveform to the data (long period systematics vs. short period fluctuations) (Note that the Excel function of "moving average" is, in fact equivalent to boxcar smoothing).

• Box Averaging: Replace N data points by the average of those data points. So suppose you have 100 data points and you average them in groups of 3. You will then up with 33 data points.

  
  
  In general you will use the sliding average approach. The length of the smoothing window (number of years in the above example) is bounded by too choices:

  
  
  
  • Too short of window does not reduce the noise very much to see tne underlying trend.
    
    
  • Too long of window will smooth every thing a way eventually reaching a straight line
  
  
  General rule of thumb say if you had a 1000 years worth of data - the minimum smoothing length to use is about 5% and the maximum is about 15%. Most moving average algorithms (like the one in excel) you want to pick a smoothing length which is an odd number.