PD-QC theory step 1:  Define performance standards; select candidate method(s); perform validation studies.

Expert theories generally agree on requirements for method selection and validation, including the application of performance standards.

Research and select an analytical process that will produce results at a rate of x to y per day, at a cost of x to y each, with accuracy of x % and precision of y %

Define performance standards for each analyte: if a sample has a true value of x units, reported results must be within y units or z % of that value

Other critical method factors or characteristics, such as cost/test, specimen types, specimen volumes, time required for analysis, rate of analysis, equipment required, personnel required, efficiency, safety, etc., must be considered during the selection of the analytical method. [Westgard ] [1]

For about 40 years, there has been a steady stream of publications concerned with the generation and application of quality specifications. There appeared to be a real conflict about how to set quality specifications, but a decisive recent advance was that a consensus was reached in 1999 on global strategies to set quality specifications in laboratory medicine. The hierarchy is shown below:

1. Assessment of the effect of analytical performance on specific clinical decision-making quality specifications in specific clinical situations
2. Assessment of the effect of analytical performance on general clinical decision-making
3. Professional recommendations
4. Quality specifications laid down by regulation or by external quality assessment scheme (EQAS) organizers
5. Published data on the state of the art [Fraser] [2]

PD-QC theory step 2: Method performance (e.g. accuracy, precision, total error) meets performance standards?

The experts have recommended application of performance standards (TEa limits) to method validation and quality control since the 1960s.

Ensure performance standards are met before reporting any patient results

One of the first recommendations for establishing quality standards was published by Tonks in 1963; these standards were presented in the form of allowable total errors. [3]

In 1970, Cotlove et al utilized within-subject biological variation to derive standards for allowable SDs. [4]

A few years later in 1974, Westgard et al formulated criteria, based on the medical usefulness of test results, which could be used to decide whether an analytical method has acceptable precision and accuracy. [5]

PD-QC theory step 3:  Design/modify and implement daily QC processes to detect significant change

Expert theories have recommended choosing QC rules by comparing method accuracy and precision to defined performance standards for decades.

Select QC samples.

Select QC samples with analyte levels that monitor clinical decision points

Verify that QC samples react the same way as patient samples (i.e. changes in accuracy or precision of patient results are reflected by proportional changes in accuracy or precision in QC sample results)

Select a QC strategy consisting of frequency of testing, QC rules and processes to create and examine QC charts. Choose a QC strategy that will always detect changes that would cause any results to fail to meet the defined performance standard.

Calculate the mean as an indicator of accuracy

Calculate the SD or CV as an indicator of imprecision

Assign the calculated mean and SD on the QC chart

Calculate the critical systematic error (SEc)

Select appropriate control strategy based on SEc

The laboratory must establish the number, type and frequency of testing control materials, using the performance specifications established by the laboratory as specified in §493.1253(b)(3). [CLIA regulations] [6]

We rely on controls to behave the same way as patient samples and detect errors in the analytical system. The ideal control material will:

1. Mimic the matrix and viscosity of patient specimens
2. Be both physically and chemically as sensitive to changes in the analytical system as patient samples
3. Contain all the analytes the laboratory measures in a given department
4. Contain concentrations of analytes at or near medical decision limits [Plaut] [7]

"The appropriateness of an optimal QC strategy seems very much to depend on the quality required, as well as the expected instability of the analytical method (e.g., type, magnitude and frequency of errors)." [Wandrup ] [8]

"The act of establishing your own means and SDs brings in the performance specifications observed in your own laboratory." [Carey] [9]

“A laboratory metric that has been used to select and design QC procedures is the critical systematic error, SEc, which can be calculated from the tolerance or quality requirement defined for the test and the imprecision and inaccuracy observed for the method […] The implication is that the error detection capability of the QC procedure should complement the performance capability of the process. With high process capability, the errors that cause defective results will be large and more easily detected by statistical quality control. As process capability decreases, the errors that must be detected get smaller, which requires better detection capabilities for the statistical QC procedure.” [Westgard] [10]

PD-QC theory step 4: QC flags or chart examination indicate change in accuracy or precision of daily QC?

The experts agree that a well-designed QC chart and strategy will identify change in an analytical system. (Such change must be evaluated before patient results are reported.)

Test QC samples. Plot all results on QC charts. Apply rules. Examine charts.

If there are no QC flags, report patient results

If none of the rules […] are violated, accept the run and report patient results. If any one of the rules [...] is violated, the run is out-of-control; do not report patient test results. [Westgard] [11]

PD-QC theory step 5: Compare to performance standards; validate data; improve or replace method if necessary

The very premise of quality control states that if change in method performance is unacceptable (no longer meets performance standards) you should stop reporting patient results.

If QC flags indicate that the accuracy or precision of the method has changed, compare the mean and SD of the current data population to your performance standards

If the changed analytical process still produces results within allowable limits of the correct/true value, adjust the QC process and carry on

If your method no longer meets performance standards, then stop reporting results while you make sure the numbers are correct and take corrective action if indicated

Results obtained for the performance characteristics (mean and SD) should be compared objectively to well-documented quality specifications, e.g., published data on the state of the art, performance required by regulatory bodies such as CLIA `88, or recommendations documented by expert professional groups. In addition, quality specifications can be derived from analysis of performance on clinical decision-making. [Fraser] [12]

"To evaluate laboratory performance, the analytical imprecision and bias (obtained from the internal quality control protocol) is compared against the quality specifications (standards) for these two components of analytical error [accuracy and precision]. The analytical procedures that deviate from the standards have to be reviewed by laboratory professionals, and processes for improving performance implemented." [Ricos] [13]

These are the 15 steps involved in performance-driven quality control:

Score a value of "5.0" if your practice is similar to this example…

Enter your ra-ting here

When selecting an analytical process, compare claims of accuracy and precision.  These claims form an important component of the selection process.

We do not include accuracy and precision on our criteria list.

Accuracy and precision are the most important items on our criteria list.

Define performance standards for each analyte that state "if a sample has a true value of x units, reported results must be within y units or z % of that value."

Never. We do not have performance standards or TEa limits defined for any analyte. “This is a good lab. We are good people. Our methods are all good.”

Ensure performance standards are met before reporting any results.

Never.

Always.

Select QC samples with analyte levels that monitor clinical decision points.

No. We just accept what the manufacturer provides.

Yes. If necessary, we purchase separate controls or make samples.

Verify that QC samples react the same way as patient samples (i.e. changes in accuracy or precision of patient results are reflected by proportional changes in accuracy or precision in QC sample results).

No. We never verify changes in QC with patient samples, or monitor patient values. We don’t check for this when purchasing new controls.

Yes. We always verify changes in QC with patient samples, and monitor patient values or moving averages. We check new QC samples for this before purchasing.

Periodically calculate the mean of a defined set of QC points as an indicator of accuracy.

No. We use a running mean. We don't examine or assess mean values on a regular basis.

Yes. We calculate, examine and assess mean values on a regular basis.

Periodically calculate the SD or CV of a defined set of QC points as an indicator of imprecision.

No. We use a running SD. We don't examine or assess SDs or CVs on a regular basis.

Yes. We calculate and examine and assess SDs or CVs on a regular basis.

Assign the current calculated mean and SD on the QC chart.

No. We assign a mean from history or the package insert or whatever. and/or…

“The SD assigned on the chart is not the actual method SD; it comes from PT limits, or package inserts or we just multiply it a few times so we don’t get false QC flags.”

Yes. We always assign the current calculated mean and SD on the QC chart.We rely on our QC strategy to alert us to change. When a shift in the mean occurs, we update the mean on the QC chart (after making sure the system still meets performance standards).

Periodically calculate the margin for error or critical systematic error (SEc) for each QC sample.

Never.

Regularly.

Select appropriate control strategies (frequency of testing, QC rules, and processes to create and examine QC charts) for each QC sample on each analyte based on margin for error (SEc). Choose a QC strategy that will detect changes that would cause results to fail to meet the performance standards defined for each QC sample.

No. We use whatever QC software comes with our instrument or LIS or QC samples. We never compare QC results to performance standards.

Or … “Someone (who cannot be questioned) decided to use a 1-2s or 1-3s rule for all controls on all tests.”

Yes. We proactively select QC strategies and implement performance-driven quality control. We regularly compare QC results to performance standards and adjust the QC process if the method performance changes (as noted by a change in mean or SD of a new data set.)

Plot all results on QC charts.

Never. We don’t plot results.

Always. All values.

Apply rules, examine charts, and report patient results only if there are no QC flags.

No. We report results and then someone examines the QC later.

Or, we report all results – QC flags don’t make us stop reporting. The doctors need the results.

Yes. We always check QC before reporting patients.We never report results on runs with QC rejects until the cause of the flag has been determined and we are sure the method still meets performance standards.

If QC flags indicate that the accuracy or precision of the method has changed, compare the mean and SD of the current data population to performance standards.

 If we start getting a new mean, then that must be what the control should be now.

Or ..“Change is OK if the supervisor says so."Or ..“If you can explain the change, it’s OK."Or ..“If the change is “not too big”, it’s OK.”

 If we start getting a new mean or SD, then we calculate Total Error and SEc to make sure the method is within allowable error.  

If the changed analytical process still produces results within allowable limits of the correct/true value, adjust the QC process and carry on.

Sort of. We change the values on the chart whenever we start getting a new mean or SD. Then the chart looks better.

Or .. No. Whenever there is a change we call for technical support. “All change is bad. It must be eliminated."

If the method is within TEa, we change the mean or SD on the QC chart and, if advisable, adjust the QC rules and process.

We realize change can be for the better – change is not always bad.

If the method no longer meets performance standards, then stop reporting results while you make sure the numbers are correct and take corrective action if indicated.

No. We never stop reporting results. The doctors need the results.

Yes. We never release results that may be wrong, and therefore lead the clinician to the wrong decision and subsequent action, thus harming the patient.