Statistical treatment of data

 

The Statistical Treatment of Indeterminate Errors

Statistics is the mathematical science that deals with chance variations. We must emphasize at the outset that statistics only reveals information that is already present in a data set. That is, no new information is created by statistics. Statistical treatment of a data set does, however, allow us to make objective judgements concerning the validity of results that are otherwise difficult to make.

 

Population Mean () and the Sample Mean ()

When N is small

When N

When N is small, sample mean does not represent population mean because a small sample of data does not exactly represent its population.

In the absence of determinate error, the population mean is the true value of a measured quantity.

 

Sample Standard Deviation (s) and the Population Standard Deviation (s )

When N is small

When N

(this is true standard deviation)

When N , , and s s . When N is greater than about 20, s and  s can be assumed to be identical for all practical purposes.

Uncertainty in the calculated value of s decreases as N increases as shown inn the following figure:

Therefore do as many repetitions as possible as the time, equipment and sample resources allow.

 

Properties of Normal Error Curve

Error Curves can be drawn in two ways. One is the frequency of occurrence Vs (x - ) (part a in the following figure) and the frequency of occurrence versus which turns out to be 1s, 2s, 3s

The normal error curve, which is a Gaussian curve, has several general properties.

  1. The mean occurs at the central point of maximum frequency
  2. There is a symmetrical distribution of positive and negative deviations about the maximum.
  3. There is an exponential decrease in frequency as the magnitude of the deviations increases. Thus, small indeterminate uncertainties are observed much more often than very large ones.

It can be shown that 68.3% of the area beneath any normal curve lies within one standard deviation (1s ) of the mean , meaning 68.3% of the data making up the population lie within these bounds. Furthermore,

95.5 % m  2s

99.7 % m  3s

So one can say that the chances are 68.3 in 100 that the indeterminate uncertainty of any single measurement in a normal distribution is no more than 1s. Similarly, the chances are 95.5 in 100 that the error is less than 2s, and so forth.

 

The Uses of Statistics

Experimentalists use statistical calculations to sharpen their judgement concerning the effects of indeterminate errors. The most common applications are:

  1. Defining the interval around the mean of a set within which the population mean can be expected to be found with a given probability.
  2. Determining the number of replicate measurements required to assure (at a given probability) that an experimental mean falls within a predetermined interval around the population mean.
  3. Deciding whether an outlying value in a set of replicate results should be retained or rejected in calculating the mean for the set.
  4. Estimating the probability that there is a difference in precision between two sets of data. obtained by different workers or by different methods
  5. Defining and estimation of detection limits.
  6. Treating calibration data.

Confidence Limits

The exact value of the mean for a population of data can never be determined exactly because such a determination requires an infinite number of measurements. Statistical theory allows us to set limits around an experimentally determined mean, , however, and the true mean lies within these limits with a given degree of probability. These limits are called confidence limits, and the interval they define is known as the confidence interval. In other words, confidence limits define an interval around that probably contains .

Confidence Interval when s is close to s

Consider the following figure:

So for the first plot, the confidence level is 50% and the confidence interval is 0.67s meaning that the probability of existence of the true value in between (- 0.67s) to (- 0.67s) is 50%. So, the confidence interval is a function of confidence level which in turn is a function of z:

So, the general expression for the confidence limits (CL) of a single measurement can easily be shown to be:

CL for = x zs

Values of z is fixed for differet confidence levels and listed in the following table:

CONFIDENCE LEVELS FOR

VARIOUS VALUES OF z

Confidence Levels, % z

50

68

80

90

95

96

99

99.7

99.9

0.67

1.00

1.29

1.64

1.96

2.00

2.58

3.00

3.29

Example: Calculate the 95% confidence limits of the measurement (11.24 cm) for a set of measurements where s s = 0.07 cm.

95% CL for m = 11.24 1.96 x 0.07 = 11.24 0.14

For the confidence limits of the mean (average) value, use the following equation:

CL for

Example: How many replicate measurements of specimen in the previous example should be done to decrease the 95% confidence interval to 0.01s

Using above equation,

Confidence interval = = 0.01 =

N is, then, calculated as 4.

Confidence Limits When s is unknown:

If s is known but s is unknown, use the equation:

CL for

Use the following table for the values of t (t is often called student's t)

VALUES OF t FOR VARIOUS LEVELS OF PROBABILITY
Number of Observations Factor for Confidence Interval
80% 90% 95% 99% 99.9%

1

3.08 6.31 12.7 63.7 637

2

1.89 2.92 4.30 9.92 31.6

3

1.64 2.35 3.18 5.84 12.9

4

1.53 2.13 2.78 4.60 8.60

5

1.48 2.02 2.57 4.03 6.86

6

1.44 1.94 2.45 3.71 5.96

7

1.42 1.90 2.36 3.50 5.40

8

1.40 1.86 2.31 3.36 5.04

9

1.38 1.83 2.26 3.25 4.78

10

1.37 1.81 2.23 3.17 4.59

11

1.36 1.80 2.20 3.11 4.44

12

1.36 1.78 2.18 3.06 4.32

13

1.35 1.77 2.16 3.01 4.22

14

1.34 1.76 2.14 2.98 4.14

1.29 1.64 1.96 2.58 3.29

Please note that as N , t z.

Rejection of outliers

When a set of data contains an outlying result (possibly out of a gross error) that appears to differ excessively from the average, a decision must be made whether to retain or reject the result. It is not easy and there is no universal rule that can be invoked to settle the question of retention or rejection.

Statistical Tests.

All tests assume that the distribution of the population data is normal, or gaussian. Unfortunately, this condition cannot be proven, or disproved for samples consisting of many fewer than 50 results. Therefore, statistical rules should be used with extreme caution when applied to samples containing only a few data. Thus, statistical tests for rejection should be used only as aids to common sense when small samples are involved.

The Q Test

It is the simplest test.

In this test, the absolute value of the difference between the questionable result xq and its nearest neighbour xi is divided by the spread (range) of the entire set to give the quantity Qexp:

Qexp = xq - xn / w

This ratio is then compared with rejection values Qcrit found in the following table:

CRITICAL VALUES FOR REJECTION QUOTIENT Q
Qcrit (Reject if Qexp > Qcrit)
Number of

Observations

90%

Confidence

95%

Confidence

99%

Confidence

3

0.941

0.970

0.994

   

4

0.765

0.829

0.926

   

5

0.642

0.710

0.821

   

6

0.560

0.625

0.740

   

7

0.507

0.568

0.680

   

8

0.468

0.526

0.634

   

9

0.437

0.493

0.598

   

10

0.412

0.466

0.568

   

If Qexp is greater than Qcrit, the questionable result can be rejected with the indicated degree of confidence.

Example:The measurement of the density of a certain mineral yielded 3.456, 3.451, 3.475, and 3.452 g/cm3. 3.477 g/cm3 appears to be anomalous; should be retained or rejected at the 90% confidence level?

Qexp = 3.475 - 3.456 / (3.475 - 3.451) = 0.792

At 90% confidence level, with 4 measurements Qcrit = 0.765 which is smaller than Qexp= 0.792. Therefore the questionable data 3.475 can be rejected at 90% confidence level. But it can be retained at 95% confidence level since Qexp (0.765)< Qcrit (0.829)

The Tn Test

In the Amerikan Society for Testing Materials (ASTM) Tn serves as the rejection criterion, where

Here, xq is the questionable result, and and s are the mean and standard deviations of the entire set including the questionable result. Rejection is indicated if the calculated Tn is greater than the critical values found in the following table:

CRITICAL VALUES FOR REJECTION QUOTIENT Tn
Number of

Observations

Tnc

95% Confidence 97.5% Confid. 95% Confidence

3

1.15

1.15

1.15

   

4

1.46

1.48

1.49

   

5

1.67

1.71

1.75

   

6

1.82

1.89

1.94

   

7

1.94

2.02

2.10

   

8

2.03

2.13

2.22

   

9

2.11

2.21

2.52

   

10

2.18

2.29

2.41

   

Example: The analysis of a calcite sample yielded CaO percentages of 55.95, 56.00, 56.04, 56.08, and 56.23. Should the last data retained or rejected at 95% confidence level?

The standard deviation s can be calculated as s = 0.107. The mean value, = 56.06.

Thus = 1.59

Since Tn (1.59) is smaller than Tnc (1.67), 56.23 is retained as an acceptable data.

 

Recommendations for the Treatment of Outliers

Recommendations for the treatment of a small set of results that contains a suspect value follow:

  1. Reexamine carefully all data relating to the outlying result to see if a gross error could have effected its value. This recommendation demands properly kept laboratory notebook containing carefull notations of all observations.
  2. If possible, estimate the precision that can be reasonably expected from the procedure to be sure that the outline result actually is questionable.
  3.  

  4. Repeat the analysis if sufficient sample, time and other resourses are available. Agreement between the newly acquired data and those data of the original set that appear to be valid will lend weight to the notion that the outlying result should be rejected. Furthermore, if retention is still indicated, the questionable result will have a relatively small effect on the mean of the larger set of data.
  5. If more data cannot be secured, apply the Q test or the Tn test to the existing set to see if the doubtful result should be retained or rejected on the statistical grounds.
  6.  

  7. If the statistical tests indicates retention, consider reporting the median of the set rather than the main. The median has the great virtue of allowing inclusion of all data in a set without undue influence from an outline value. In addition, the medion of the normally distrubeted set containing 3 measurements provides a better estimate of the correct value than the mean of the set after the outline value has been discarded.

The blind application of statistical tests to retain or reject a suspect measurement in a small set of data is likely to be much more fruitful than an arbitrary decision. The application of good judgement based on broad experience with an analytical method is usually a sounder approach. In the end, the only valid reason for rejecting a result from a small set of data is the sure knowledge that a mistake was made in the measurement process. Without this knowledge, a cautious approach to rejection of an outlier is wise.