Quang et al.1 critically appraised systematic reviews of health economic evaluations using the 27 items recommended by the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) Statement.3 The authors state that PRISMA is widely known as an instrument to qualitatively appraise a systematic review of health economic evaluations. Based on the number of items fulfilled, they categorized the quality of each article on a scale with a range of “Very bad” (< 10 points) to “Excellent” (≥ 25 points). The respective threshold values for the 5 categories of quality were defined by the authors themselves.
The methods applied by Quang et al.1 seem to be questionable for the following reasons in particular:
Using the PRISMA checklist to assess the quality of systematic reviews
Although its value for a critical appraisal of the reporting of a systematic review is undoubted, “PRISMA is not intended to be a quality assessment tool and it should not be used as such” as stated in the PRISMA Explanation and Elaboration Document.4
Using the PRISMA checklist in health economics
PRISMA was primarily developed to support the reporting of systematic reviews and meta-analyses of randomized trials.3 Even though several of its criteria are also relevant for systematic reviews of health economic evaluations, others are not applicable, e.g. registration of systematic reviews of health economic evaluations is usually not possible with PROSPERO, as they do not meet the eligibility criterion of dealing with clinical outcomes. A meta-analytical approach for data synthesis is not feasible in most systematic reviews of health economic evaluations,5 as the results of health economic evaluations are, in most cases, not generalizable.6 Furthermore, a standard for the assessment of risk of bias across health economic studies is not established yet. Accordingly, Quang et al.1 found that just one or no articles at all fulfilled 5 out of the 27 items of the PRISMA checklist. A use of PRISMA for the assessment of reporting or the quality of a systematic review of health economic evaluations (without modifications) is inappropriate.
Using summary scores for the categorization of quality
Quang et al.1 derived the score for categorization of quality by counting the number of items fulfilled in each systematic review of health economic evaluations. This suggests that all the items on the PRISMA checklist are equally important for such a categorization. This can hardly be justified, particularly when taking into account that several of the items are not applicable to systematic reviews of health economic evaluations. As it is difficult to justify assigned weights and for reasons of transparency, the use of summary scores to distinguish between high and low quality is generally not recommended.7,8 The fact that the authors did not justify the arbitrarily set thresholds for the different categories of quality makes the evaluation even more questionable.
In conclusion, the assessment of quality of systematic reviews of health economic evaluations is challenging. To date, no validated tool exists which is able to consider the particularities of health economic evaluations. We strongly question the use of a summary score based on the items of the PRISMA checklist. With AMSTAR (a measurement tool to assess the methodological quality of systematic reviews) a validated tool is available for the assessment of the methodological quality of systematic reviews.9
Instead of using a summary score for quality assessment, we suggest a separate assessment and discussion of all single items.