Pilot Laboratory Assessment and Peer Review Mechanism for Pathology Key Performance Indicators (2007)
Description
This project arose from recommendation 5.1 of the Corrs Chambers Westgarth report titled The Evaluation of Australian Pathology Laboratory Accreditation Arrangements for the Australian Government Department of Health and Ageing 2002 (Corrs Chambers Westgarth Lawyers) which states:
that the DHA and the HIC seek the cooperation of the RCPA QAP to establish explicit external quality assurance performance criteria, initially in chemical pathology and gynaecological cytology, and a mechanism for the RCPA QAP to identify relatively poorly performing laboratories.
The first stage of this project developed Key Performance Indicators (KPIs) for Chemical Pathology and Cytopathology that could potentially be used to identify an early warning alert system of poor laboratory performance. It was identified that a further step was required in the development of KPIs to assess if they were a valid tool for this purpose. This resulted in the project’s second stage as represented by this project.
Royal College of Pathologists of Australasia Quality Assurance Programs Pty Ltd (RCPA QAP)
Aims and Objectives
establishment of Peer Review Committees (PRCs) for Chemical Pathology and gynaecological Cytopathology
pilot extension of the KPI assessment and review process as developed in the initial project
address the issue of the National Pathology Accreditation Advisory Council (NPAAC) producing a standard or other document to provide legislative backing to the process
funding option for the KPI assessment and review process in the long term to be investigated
the RCPA QAP needs to obtain legal advice regarding the feasibility of a waiver with respect to liability for laboratories that sign up for participation
education of laboratory participants on the KPI assessment and review process
potential further refinement of the KPIs is recognised as being a logical development of the program and beneficial to the improvement of diagnostic capabilities.
These aims and objectives were achieved by this project.
Outcome
This project demonstrated the need for an appropriate professional review of the proposed mechanisms for the development of an effective early warning system to detect potentially poorly performing laboratories.
Findings
The results of the pilot project to evaluate the use of KPIs to identify poorly performing laboratories has meant the Steering Committee could not recommend the use of KPIs as a formal monitoring process for poorly performing laboratories as part of NPAAC.
The Chemical Pathology KPIs have been identified as a useful internal quality tool for laboratories to use.
The power of the KPI methodology is very dependent on the amount of data that can be analysed, and as a consequence, the KPIs have been more useful in Chemical Pathology than in Cytopathology.
The information to hand suggests that KPIs and the PRC process can identify laboratories with poor External Quality Assessment (EQA) performance, but this does not necessarily equate to poor ratings at an on-site assessment.
The lowest KPI laboratories were not distinguished by more adverse assessment outcomes when compared with the highest KPI group.
The methodologies used suggested the current KPIs were not a sensitive test of overall laboratory performance when compared to assessment findings.
The correlation of the KPIs and National Association of Testing Authorities (NATA) assessments did not support their use as an ‘early warning system.’
The KPIs as calculated for this project were based solely on quality assurance program (QAP) performance. Their name infers this correlates with actual laboratory performance, however, this does not appear to be the case.
Recommendations
The frequency of the KPI review for Chemical Pathology should be every six months.
Cytopathology does not have sufficient data for a six-month review, so it should have an overlapping 12 month review every six months.
KPIs are not a valid tool to use for identifying poorly performing Gynaecological Cytopathology Laboratories, and as a consequence the Steering Committee recommends that they NOT be implemented in the existing format.
The KPIs for Chemical Pathology appeared to be of some use in identifying poorly performing laboratories and are well accepted by laboratories as a useful internal quality tool. However, due to the poor correlation between laboratories known to be poorly performing and the KPI, the Steering Committee could NOT recommend the introduction of KPIs in Chemical Pathology as a formal tool enshrined in NPAAC Standards to help identify poorly performing laboratories. However, because of the usefulness of the KPI in Chemistry to evaluate performance based on quality assurance (QA) data, and potentially useful in ranking laboratories overall performance, the RCPA QAP will continue producing the KPIs for Chemical Pathology and this will be included within the services it offers to its participants.
The Steering Committee considered it appropriate to continue a Peer Review Mechanism for Chemistry on a trial basis.
The Steering Committee considered developing KPIs in Blood Transfusions on a pilot basis as useful.
QAP data in Cytology expressed as KPIs has not proven useful as a predictor of “poorly performing” laboratories, although a large number of major errors (e.g. greater than three) in a QAP may still be a useful trigger for review.
Examination of the relationship of Performance Measures to NATA Assessment findings should be the subject of a further study.
The role and membership of the PRC and its Terms of Reference need to be explored further.
Legal advice to the RCPA QAP regarding feasibility of a waiver with respect to liability for laboratories that signed up for participation included the recommendation that identifying information never be released to the PRC in general, but only to NATA. They also advised seeking a legal agreement with NATA to ensure that NATA never discloses the identifying information to a third party, and that their actions be limited to an accelerated review of the laboratory in question.
Ongoing funding for any further projects needs to be considered.
Potential further refinement of the KPIs was recognised as being a logical development of the program and beneficial to the improvement of diagnostic capabilities.
The work undertaken at Pacific Laboratory Medicine Services (PaLMs) regarding KPIs for quantitative pathology work could form the basis for further analysis of their findings to provide the scientific community with a clear understanding of which tests need to be concentrated on in terms of further development and quality improvement.
Key Project Learning
There were delays completing the project related to KPI software and the workforce crisis in pathology meaning the time pathologists could give to this project on a voluntary basis was restricted. It also sometimes took long periods of time to get the Committee together to review data which also delayed the project.
Follow on Initiatives and Projects
Performance Monitoring of External Quality Assurance (Current)
NATA File Audit – Risk Analysis of Assessment Non Conformances Identified in Pathology Laboratory Assessment Accreditation (Current).
