Noisy Data

In general, most environmental data has a considerable amount of random noise it. The origin of this random noise is

1. the phenomena itself has a lot of intrinsic variability - this is especially true with climate data

2. the measurements are often difficult and imprecise

The example below shows a time series where the intrinsic variability completely swamps out any potential trend. To identify possible trends it is necessary to smooth the data.

A variety of techniques exist for this.


(blue line = smoothed trend)

Any data that has significant flucutations as a function of time may dominate the underlying wave form (trend with time). You therefore want to dampen out the fluctuations to reveal the underlying wave form.

The most common technique available is "smoothing". These techniques serve to surpress the noise and to extract real trends and patterns.

Note that smoothing and averaging are not the same:

• Averaging ends up with fewer actually data points
• Smoothing preserves the number of data points but lowers the amplitude of the noise by coupling the data points.

Techniques:

• Boxcar smoothing or sliding average: this works best when there is some underlying waveform to the data (long period systematics vs. short period fluctuations) (Note that the Excel function of "moving average" is, in fact equivalent to boxcar smoothing).

• Box Averaging: Replace N data points by the average of those data points. So suppose you have 100 data points and you average them in groups of 3. You will then up with 33 data points.

  Climate example

• Exponential Smoothing:
  
  
  Here the smoothed statistic, s_t is the weighed average of the previous data value (x_t-i) and the previous smoothed statistic, s_t-i; α is the smoothing factor (a value between 0 and 1).
  
  This method is often applied to data which has lots of short term fluctuations in it. Financial or stock market data is often treated this way but climate time series also lend itself to this smoothing approach.

  The functional form of this smoothing is governed by one parameter, the smoothing parameter, and it has a value between 0 and 1. Values close to one have less of a smoothing effect and give greater weight to recent changes in the data, while closer to zero have a greater smoothing effect and are less responsive to recent changes. There is no formally correct procedure for choosing the smoothing factor -its a scientific judgement call.

  Example excel spreadsheet (cell K1 is the only thing to change and is the smoothing parameter)

• Guassian Kernel Smoothing: This is very well suited to time series analysis.
  The 'kernel' for smoothing, defines the shape of the function that is used to take the average of the neighbouring points. A Gaussian kernel is a kernel with the shape of a Gaussian (normal distribution) curve. Here is a standard Gaussian, with a mean of 0 and σ (=population standard deviation) of 1.
  
  The basic process of smoothing is very simple. We proceed through the data point by point. For each data point we generate a new value that is some function of the original value at that point and the surrounding data points
  
  
  
  
  For an example, let's say we want to replace the 14th value in our data set with a smoothed value.

  We have chosen our kernel to have standard deviation of 2.5 units on the X-axis. With the kernel centered x=14 and scaled so that the total area of the Kernal = 1 (this is required) our smoothing function now looks like this.
  
  
  
  

  The discrete version of the function is shown at the left. The table below summarizes the x,y values of the function as well as the actual data.

  X	Y	Y-data
  12	0.117	1.065
  13	0.198	0.389
  14	0.235	0.349
  15	0.198	-0.656
  16	0.117	-0.195

  Note: we used a gaussian kernel with σ = 2.5. The example used here just uses the first 5 points (eg. ~ +/- 1 σ) which represents 86.5 % of the total area and so contains most of the weight in the smoothing (but you really do want to use all of the kernel to get the best results)
  
  
  
  
  We want to produce a new value for x = 14 based on this weighted smoothing function. That means we convolve our function with the data. That sounds onerous, but its really quite simple and works like this.

  The new Y-data point at X=14 is then:

  0.117*1.065 + 0.198*0.389 + 0.235*0.349 + 0.198*-0.656 + 0.117*-.195

  = .125 + .077 + .082 - .130 - .023 = .141 (compared to the actual data value of .349).

  We then apply the same procedure to point 15 etc and end up with a convolved time series distribution that looks like this, where larger scale patterns are now more evident

  

  This particular method is very good for image or pattern recognition and can be important for analyzing climate data: