Continuous improvement
Regression analysis: a process example
Work through linear regression using NIST calibration data. Check the fitted equation, residuals and why a high R squared cannot approve a model.
Regression estimates a relationship between a response and one or more explanatory variables. A straight-line least-squares fit chooses an intercept and slope to minimise the sum of squared residuals.
A residual is the observed response minus the fitted response. It keeps the model's errors visible after the line summarises the relationship.
NIST's regression guide explains the method. The worked example below uses NIST's published calibration data.
Fit the 36 NIST observations
The Norris dataset contains 36 x/y pairs from ozone-monitor calibration. Recalculating ordinary least squares from those rows reproduces NIST's certified results to the displayed precision.
Fitted relationship in the source variable scale
Fitted y = −0.262323 + 1.002117 × x
Residual = observed y − fitted y
Residual standard deviation = 0.884796
R² = 0.999993746
The source does not assign physical units in the data file, so the chart retains x and y. It also supplies no production acceptance tolerance.
NIST Norris calibration dataset / 36 observed pairs
Inspect the fit and its residuals
The fitted values reproduce the published certified statistics. Select a point to inspect its error.
Observation 1: x = 0.2, y = 0.1. Fitted y = -0.0619; residual = 0.1619.
Fitted y = -0.262323 + 1.002117 × x. R² = 0.999993746. Residual standard deviation = 0.884796. No uncertainty band or acceptance tolerance is assumed.
Inspect all 36 source pairs and fitted values
| Observation | x | y | Fitted y | Residual |
|---|---|---|---|---|
| 1 | 0.2 | 0.1 | -0.0619 | 0.1619 |
| 2 | 337.4 | 338.8 | 337.851891 | 0.948109 |
| 3 | 118.2 | 118.1 | 118.187885 | -0.087885 |
| 4 | 884.6 | 888 | 886.210214 | 1.789786 |
| 5 | 10.1 | 9.2 | 9.859057 | -0.659057 |
| 6 | 226.5 | 228.1 | 226.717136 | 1.382864 |
| 7 | 666.3 | 668.5 | 667.448113 | 1.051887 |
| 8 | 996.3 | 998.5 | 998.146663 | 0.353337 |
| 9 | 448.6 | 449.1 | 449.287281 | -0.187281 |
| 10 | 777 | 778.9 | 778.382445 | 0.517555 |
| 11 | 558.2 | 559.2 | 559.119285 | 0.080715 |
| 12 | 0.4 | 0.3 | 0.138524 | 0.161476 |
| 13 | 0.6 | 0.1 | 0.338947 | -0.238947 |
| 14 | 775.5 | 778.1 | 776.879269 | 1.220731 |
| 15 | 666.9 | 668.8 | 668.049383 | 0.750617 |
| 16 | 338 | 339.3 | 338.453161 | 0.846839 |
| 17 | 447.5 | 448.9 | 448.184953 | 0.715047 |
| 18 | 11.6 | 10.8 | 11.362232 | -0.562232 |
| 19 | 556 | 557.7 | 556.914628 | 0.785372 |
| 20 | 228.1 | 228.3 | 228.320523 | -0.020523 |
| 21 | 995.8 | 998 | 997.645604 | 0.354396 |
| 22 | 887.6 | 888.8 | 889.216565 | -0.416565 |
| 23 | 120.2 | 119.6 | 120.192118 | -0.592118 |
| 24 | 0.3 | 0.3 | 0.038312 | 0.261688 |
| 25 | 0.3 | 0.6 | 0.038312 | 0.561688 |
| 26 | 556.8 | 557.6 | 557.716321 | -0.116321 |
| 27 | 339.1 | 339.3 | 339.55549 | -0.25549 |
| 28 | 887.2 | 888 | 888.815718 | -0.815718 |
| 29 | 999 | 998.5 | 1,000.852378 | -2.352378 |
| 30 | 779 | 778.9 | 780.386678 | -1.486678 |
| 31 | 11.1 | 10.2 | 10.861174 | -0.661174 |
| 32 | 118.3 | 117.6 | 118.288096 | -0.688096 |
| 33 | 229.2 | 228.9 | 229.422852 | -0.522852 |
| 34 | 669.1 | 668.4 | 670.25404 | -1.85404 |
| 35 | 448.9 | 449.2 | 449.587917 | -0.387917 |
| 36 | 0.5 | 0.2 | 0.238735 | -0.038735 |
NIST source observations and certified statistics
At x = 500, the fitted response is approximately 500.796. That is an estimate from the equation within the observed x range of 0.2 to 999.0, not an additional measured row.
Read the errors after reading the fit
The R² value is close to one because the line accounts for nearly all the response variation across this broad range. It does not say that the remaining errors are small enough for a particular use.
The recalculated residuals range from about −2.352 to 1.790. Whether those differences matter requires an operating requirement that the dataset does not provide.
NIST's validation guidance explicitly warns that a high R² is insufficient. Inspect residuals for curvature, changing spread and other departures from the selected model's assumptions. Use the actual acquisition order for time-related checks; a row number alone does not establish a timestamp.
Keep the model within its evidence
Extrapolating beyond the observed range adds assumptions about behaviour the data have not shown. Even an estimate within the range needs a suitable uncertainty calculation before it supports a consequential decision.
The figure shows fitted values and residuals. Confidence and prediction intervals need their own interpretation and assumptions.
The scatter-diagram guide examines the same pairs before fitting. The confidence-interval Note explains why uncertainty belongs to a defined estimate.
Separate estimation from a process change
An observational fit can help estimate a response or plan another study. It does not establish that adjusting x will cause the predicted change in y.
Use the design-of-experiments example when the task is to study controlled changes to settings.
For a model review, keep the equation, residuals, observed range and intended decision together. State the requirement first, then decide whether the remaining prediction error is acceptable for that use.