Verification or validation, that is the question

ISO 15189:2012 requirements 5.5.1.2 and 5.5.1.3

The latest revision of ISO 15189 standard in 2012 is now universally considered the most important document for accreditation of medical laboratories (1), and 60 countries have made it mandatory for clinical laboratories so far (2). The standard focuses on patient care as the main objective of laboratory services, which can be achieved by verifying management organization, responsibilities and technical quality, and by adopting the philosophy of continuous improvement (3). The technical needs focus on mitigating laboratory errors through evaluation of the total testing process. Regarding the analytical phase, this standard entails verification and validation of examination procedures (EPs) (section 5.5.1.2 and 5.5.1.3). According to International Vocabulary of Metrology 3 (VIM 3), the definition of verification is “provision of objective evidence that a given item fulfils specified requirements” (4). In particular, the ISO 15189 states that “the independent verification by the laboratory shall confirm, through obtaining objective evidence (in the form of performance characteristics) that the performance claims for the EP have been met. The performance claims for the EP confirmed during the verification process shall be those relevant to the intended use of examination results.” (1). Validation is defined as “verification, where the specified requirements are adequate for the intended use” (4). In particular, ISO 15189 attests that “the validation shall be as extensive as is necessary and confirm, through the provision of objective evidence (in the form of performance characteristics), that the specific requirements for the intended use of examination have been fulfilled.” (1).

When used without modification, the validated EP shall be verified, whilst non-standard methods, home brew methods, validated methods which have been modified or are being used outside their intended scope shall be validated. The main objective of validation of an EP is to demonstrate its fitness-for-purpose (5).

The pragmatic approach to verify and validate the examination procedures (EPs)

In order to fulfil the above requirements, three different approaches were used for:

• Verification of EPs already in use in laboratory;
• Verification of newly introduced EPs;
• Validation of EPs.

Verification of EPs already in use in laboratory

The developed operative flow for EPs already in use for over two years is based on consideration that the accreditation process is developed in a laboratory where the quality system has already been successfully implemented (6,7). Therefore, the operative procedures refer to a laboratory in which the total testing process is under control, users’ satisfaction is monitored and an improvement process is ongoing. Therefore, if the EP provides results satisfying clinical needs and, subsequently, the purpose of the test, the verification can be limited to parameters focused on the accuracy of test results. This approach is described in detail elsewhere (8), and summarized in Table 1.

Verification of newly introduced EPs

Following the same approach as for verification of EPs already in use, different procedures were identified for quantitative, qualitative and semi-quantitative newly-introduced EPs.

Verification of newly introduced quantitative EPs

For quantitative EPs, the document CLSI EP15-A3 (9) was analysed for establishing the most suitable workflow. As for quantitative EPs already used in the laboratory, no less than imprecision and trueness need to be verified in terms of CV% and bias%, respectively.

Imprecision verification

As suggested by CLSI EP15-A3 (9), the imprecision verification study consisted of three parts, as summarized in Figure 1.

Figure 1 Operative flow for imprecision verification of newly introduced examination procedures. S_WL, within-laboratory imprecision; S_R, laboratory repeatability; σ_WL, manufacturer total imprecision; σ_R, manufacturer intra-assay imprecision; UVLs, upper limits of verification; UVL_WL, upper limit of verification for total imprecision; UVL_R, upper limit of verification for intra-assay imprecision; CV%_ref, clinically allowable limit of variability.

(I) Repeated measurements over 5 days of at least two patients’ samples. For this study, patient samples, pool of patient samples or commercially available quality controls can be used, in this hierarchical order of priority. The samples should have different concentrations and in particular, the concentrations should be close to those validated by the manufacturer. The proposed basic design (3×5) needs five days, three replicates per run, one run per day, for a total of 15 results per sample. In relation to the measurand stability, the length of the design can be reduced to a 3×4 scheme, but not less than a 3×3 design should be used. The presence of outliers needs to be investigated using the Grubbs’ test. If more than one outlier is observed throughout the entire study, the verification of imprecision study should then be repeated.

(II) Calculations of within-laboratory imprecision (S_WL) and laboratory repeatability (intra-assay-S_R).

(III) Assessment of uniformity with claims and acceptability of test results. For each level, the S_WL should be less than the total imprecision claimed by the manufacturer (σ_WL).When S_WL is higher than σ_WL, the upper verification limit (UVL_WL) needs to be calculated, as described in the CLSI EP15-A3 (9), by considering the degrees of freedom and the UVL factor. If the S_WL is higher than the respective UVL_WL, a clinically allowable limit of variability (CV%_ref) needs to be defined. This limit can be identified following the hierarchical structure established in the 1999 Stockholm Consensus Conference (10) and revised in the 1^st strategic conference held in Milan (11), according to: (i) clinical recommendations; (ii) biological variation; (iii) state-of-the-art, as we proposed for verification of EPs already used in the laboratory (8). If S_WL is higher than CV%_ref, the S_R has to be compared with the intra-assay imprecision claimed by manufacturer (σ_R). If S_R is higher than σ_R, the UVL_R needs to calculated, as previously discussed. If the S_R is higher than the respective UVL_R, the verification process fails.

Trueness verification

Trueness is estimated by analyzing materials with known concentration, comparing results with target values, and establishing the verification interval, as described in the CLSI EP15-A3 document (9). The materials with known concentration can be: (I) materials whose concentrations can be adjusted to the desired levels, with negligible imprecision, e.g., by spiking a therapeutic drug into patient sample pools known to be analyte-free; (II) certified reference standards; (III) survey materials from PT/EQA programs; (IV) materials used in inter-laboratory quality control programs; (V) materials intended for routine internal quality control with formerly assigned target values. The mean value obtained in these studies should be included in the verification interval. If the verification interval did not include the mean value, the bias should be calculated (Bias%_LAB). If the manufacturer provides a declared bias, the bias obtained by the laboratory should be less than or equal to that claimed by manufacturer. If the manufacturer did not declare this value, a specific allowable bias (Bias%_ref) has to be identified, according to criteria reported for imprecision. If the Bias%_LAB is higher than the Bias%_ref, then the verification fails. When materials at known concentrations are unavailable, another performance characteristic should be verified according to performances claimed by the manufacturer, so that at least two performance characteristics are verified.

Verification of newly introduced qualitative EPs

For qualitative EPs, the document CLSI EP12 (12) is used as a guideline. For this type of procedures, the diagnostic accuracy is evaluated, in terms of sensitivity and specificity, using not less than ten true negative samples and not less than ten true positive patient samples. To verify this parameter, the manufacturer needs to declare the diagnostic accuracy in terms of diagnostic sensitivity, diagnostic specificity, number of true positive and true negative samples. The 95% confidence intervals (95% CI) are calculated with the Wilson’s method, as we proposed for verification of EPs already used in laboratory (8). The diagnostic sensitivity and specificity obtained by the local laboratory should be included in the calculated 95% CI derived from manufacturer’s data.

Verification of newly introduced semi-quantitative EPs

For semi-quantitative EPs, imprecision and diagnostic accuracy are assessed, as described above for quantitative and qualitative EPs, respectively.

Validation of EPs

The validation process consists of six parts:

(I) Analysis of available scientific documentation pertinent to specific EP. In particular, clinical guidelines, recommendations and peer-reviewed texts or journals should be considered. In our experience, documents are selected for validation of mass spectrometry-based methods (13,14), for chromatographic techniques (15,16), for ligand-binding assays (15), for cell-based fluorescence assay (17), and for multiplex nucleic acid assays (18).

• Evaluation of the intended use of EP.
• Identification of performance characteristics for specific EP. The choice of the performance characteristics is strictly dependent on the necessity to confirm that specific requirements for intended use are fulfilled, as mandated by the ISO 15189 (1).
• Definition of experimental procedure: the type of the tests to be performed, thus including number of samples and replicates, along with the time need for testing should be determined.
• Identification of acceptability criterion for evaluating results according to the appropriateness of intended use. The acceptability of results should be evaluated according to reliable scientific documents, previously identified.

A validation plan should be planned, summarizing all these points. The template of a validation plan is briefly shown in Figure 2. All results obtained during the validation process should be recorded, along with instrumental output. In fact, the ISO 15189 states that “the laboratory shall document the procedure used for the validation and record the results obtained” (1).

Figure 2 Template of a validation plan.

(VI) Production of a validation certificate, i.e., the final step of the validation process. The validation certificate is a tool aimed to summarize the validation process and define the approval for use in clinical practice. The validation certificate should report type of EP, intended use of the test, reference documents, results obtained during the evaluation of performance characteristics and approval for use. In particular, the personnel who carried out the validation and who is responsible for specific EP has the duty of the fitness for purpose of validating EP. The Laboratory Director has the responsibility for adopting and using the EP in clinical practice. The template of a validation certificate is shown in Figure 3.

Figure 3 Template of a validation certificate.

Conclusions

The International Standard ISO 15189 is widely recognized as the guideline for medical laboratories accreditation. Despite the first edition was published in 2007, this guideline has not been extensively implemented so far by medical laboratories at an international level, with large variations between different Countries and Regions. For example, there are only six accredited laboratories in Italy (19). The verification and validation of the EPs are novel requirements compared to the more widespread ISO 9001 certification. The development of operating procedures based on a practice approach should help medical laboratories implementing the accreditation process, thus considering the balance between technological possibilities, risks and costs. The proposed verification and validation procedures have been recognized to comply with the ISO 15189 requirements during the accreditation audit. Therefore they should represent a valuable model to be followed by other medical laboratories.

Acknowledgments

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned by the editorial office, Journal of Laboratory and Precision Medicine for the series “International Conference on Laboratory Medicine”. The article has undergone external peer review.

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/jlpm.2017.07.11). The series “International Conference on Laboratory Medicine” was commissioned by the editorial office without any funding or sponsorship. Mario Plebani served as an unpaid Guest Editor of the series. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. International Organization for Standardization (ISO). Medical laboratories-Particular requirements for quality and competence. ISO 15189:2012.
2. Farmer T. Toward a culture shift in laboratory quality: application of the full ISO 15189 standard. MLO Med Lab Obs 2015;47:38-9. [PubMed]
3. Plebani M, Sciacovelli L, Chiozza ML, et al. Once upon a time: a tale of ISO 15189 accreditation. Clin Chem Lab Med 2015;53:1127-9. [Crossref] [PubMed]
4. International vocabulary of metrology-Basic and general concepts and associated terms (VIM) JCGM 200:2012. VIM 3rd edition, Geneva, 2012.
5. Stöckl D, D'Hondt H, Thienpont LM. Method validation across the disciplines--critical investigation of major validation criteria and associated experimental protocols. J Chromatogr B Analyt Technol Biomed Life Sci 2009;877:2180-90. [Crossref] [PubMed]
6. Clinical Pathology Accreditation (UK). Standards for Medical Laboratory. Version 2.02 (March 2009).
7. International Organization for Standardization (ISO). Quality management system. ISO 9001:2008.
8. Antonelli G, Padoan A, Aita A, et al. Verification of examination procedures in clinical laboratory for imprecision, trueness and diagnostic accuracy according to ISO 15189:2012: a pragmatic approach. Clin Chem Lab Med 2017; [Epub ahead of print]. [PubMed]
9. Clinical and Laboratory Standards Institute (CLSI). User Verification of Precision and Estimation of Bias; Approved Guideline-Third Edition. CLSI EP15, 2014.
10. Fraser CG. The 1999 Stockholm Consensus Conference on quality specifications in laboratory medicine. Clin Chem Lab Med 2015;53:837-40. [Crossref] [PubMed]
11. Defining analytical performance specifications: Consensus Statement from the 1st Strategic Conference of the Sandberg S, Fraser CG, Horvath AR, et al. European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med 2015;53:833-5.
12. Clinical and Laboratory Standards Institute (CLSI). User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline-Second Edition. CLSI EP12, 2008.
13. Clinical and Laboratory Standards Institute (CLSI). Mass spectrometry in the clinical laboratory: general principles and guidance; proposed guideline. CLSI C50-P, 2007.
14. Clinical and Laboratory Standards Institute (CLSI). Liquid chromatography-mass spectrometry methods; approved guideline. CLSI C62-A, 2014.
15. Food and Drug Administration. Bioanalytical method validation. FDA, 2013.
16. European Medicines Agency. Guideline on bioanalytical method validation. EMA, 2012.
17. Wood B, Jevremovic D, Béné MC, et al. Validation of cell-based fluorescence assays: practice guidelines from the ICSH and ICCS - part V - assay performance criteria. Cytometry B Clin Cytom 2013;84:315-23. [Crossref] [PubMed]
18. Clinical and Laboratory Standards Institute (CLSI). Verification and validation of multiplex nucleic acid assays; approved guideline. CLSI MM17-A, 2008.
19. Accredia [Internet]: accredited Italian medical laboratories. Available online: http://www.accredia.it/accredia_labsearch.jsp?ID_LINK=293&area=7