INTRODUCTION

Systematic reviews, both Cochrane and non-Cochrane, are of increasing importance in guiding future medical research in addition to changing clinical practice and informing health policy. A comprehensive range of reviews has been conducted in dentistry, including a number of important reviews on fundamental orthodontic procedures and treatments.1 It is therefore critical that systematic reviews are conducted according to scientifically rigorous methodology2,3 and are clear and unbiased, which will facilitate their assessment and the translation of their findings into clinical practice. Reporting guidelines have been developed with the ultimate aim of improving the quality and content of systematic reviews, leading to more transparent results and estimates of treatment effects.

An 18-item checklist, the QUOROM Statement (Quality of Reporting of Meta-analyses), was developed in 1996 to improve the reporting of meta-analyses in health care research.4 This has since been superseded by the publication of the “Preferred Reporting Items for Systematic Reviews and Meta-analyses” (PRISMA) statement,5,6 comprising 27 specific items encompassing all aspects of the report. The key requirements of PRISMA include the accessibility of a protocol for the review, reporting of at least one complete electronic search, assessment of risk of bias in and across included studies, description of selective outcome reporting, reporting of limitations of the review and future research implications, and comment on sources of funding. Editors of orthodontic journals have been relatively efficient in implementing and endorsing reporting guidelines7,8 and most recently the PRISMA guidelines.8

While reporting guidelines are integral to helping improve the quality of published manuscripts, it is often difficult to be certain of the scientific quality and rigor of the underlying research. Checklists have also been developed in an attempt to assess the quality of systematic reviews, as distinct from the reporting of the review process.9 Previous studies have been undertaken on orthodontic systematic reviews to investigate the quality of the research using AMSTAR criteria.10,11 It is unclear, however, whether reporting of systematic reviews is necessarily likely to reflect improved quality of the review process. The aims of this study were to assess the reporting quality of Cochrane and non-Cochrane systematic reviews published in leading orthodontic journals and the Cochrane Database of Systematic Reviews (CDSR) with reference to PRISMA guidelines. A secondary aim was to assess whether reporting quality (PRISMA score) is synonymous with the quality of the review process itself (AMSTAR score).

MATERIALS AND METHODS

A comprehensive literature search was undertaken to identify systematic reviews by searching five major orthodontic journals (American Journal of Orthodontics and Dentofacial Orthopedics [AJODO], Angle Orthodontist, European Journal of Orthodontics [EJO], Journal of Orthodontics, and Orthodontics and Craniofacial Research; February 2002 to July 2011) and the Cochrane Library from January 2000 to July 2011. The search process is outlined elsewhere.10 Reviews were considered eligible for inclusion if the terms systematic review or meta-analysis were in the title or abstract or if it was apparent in the text that a systematic review had been undertaken. Narrative reviews, surveys, historical reviews, and case reports with extensive literature reviews were excluded. Two authors screened potentially relevant articles independently, with disagreement resolved by discussion with a third author (N.P.); full-text articles were obtained for each potentially relevant study.

The quality of the reporting of the included reviews was assessed according to the level of compliance with the PRISMA guidelines. These guidelines incorporate 27 items pertaining to each section of the review, including title, abstract, introduction, methods, results, discussion, conclusion, and funding. Each criterion was assigned one of three grades: “no description,” “inadequate,” or “adequate.” Overall, each article was assigned a cumulative grade based on fulfilment of these 27 criteria. An overall percentage score was assigned to each review based on the sum of the applicable items; thus, if all 27 items were applicable, the total possible score was 81 (or 100%). Individual items were not applicable to several of the included reviews, and in these instances, these criteria were omitted and the denominator amended accordingly. Discrepancies between the authors in the grading of individual criteria were resolved by joint discussion. AMSTAR scores for the included studies have been reported in a prior study10; in this study, the relationship between overall reporting (PRISMA) and quality (AMSTAR) scores for each review was to be assessed.

Initially descriptive statistics on the characteristics of the included systematic reviews were undertaken. Modified PRISMA scores for each review were calculated and converted into a percentage. Modified PRISMA scores were approximately normally distributed; thus, parametric testing was undertaken including both simple and multiple linear regression analyses. Associations of reporting quality, with overall PRISMA score as the dependent variable, and independent predictors including journal of publication, time since publication, authorship country of origin, and number of authors were analyzed. The association between reporting of systematic reviews and the quality of the review process was also investigated using linear regression analysis. The level of statistical significance was prespecified at .05. Statistical analyses were performed with STATA version 12.0 software (Stata Corporation, College Station, Tex).

RESULTS

The distribution of the publications in each journal has previously been highlighted.10 Most reviews were published in AJODO (31.2%), in Angle Orthodontist (29.4%), and on the CDSR (23.9%). In terms of adherence to individual criteria, the overall performance was assessed as good for reporting of the abstract, explanation of rationale for the review, listing of information sources and criteria for study selection, analysis of risk of bias (or quality) of primary studies, presentation of results from individual studies, summary of the evidence, and reporting of both limitations and conclusions. However, other important steps in the review process, including registration of review protocol or reporting of funding sources, were undertaken only in Cochrane reviews. Similarly, reporting of risk of bias across studies, definition of summary measures, and explanation of methods analysis was considered adequate in only a minority of reviews (Table 1).

Table 1.  Distribution of Scores for Individual Items on the 27-Item Modified PRISMA Checklist

View Fullscreen

The overall fulfilment of PRISMA criteria for articles published in individual journals, published in different geographical locations, comprising greater numbers of authors, and based on time since publication is provided in Table 2. The mean overall PRISMA score was 64.1% (95% confidence interval [CI], 62%–65%). Univariable analysis indicated that journal of publication and number of authors were significantly associated with the PRISMA score. Improved PRISMA scores were noted in reviews published in the CDSR (72.7%). In general, the reporting quality of reviews in each of the five orthodontic journals was not dissimilar (57% to 62.1%). Reporting appeared to be slightly better in AJODO (62.1%) and Angle Orthodontist (61.9%) than was the case for the three other journals. An increased number of authors were associated with an average 1.7% PRISMA score improvement with each additional coinvestigator; a mean reporting score in excess of 70% was reported when more than seven authors were involved. A weak trend toward improvement was noted with more recently published reviews, although the mean differences were minor.

Table 2.  PRISMA Score Based on the Journal of Publication, Years Since Publication, Country of Authorship, and Number of Authors^a

View Fullscreen

Adherence to PRISMA guidelines related to journal of publication, continent of publication, number of authors, and time since publication were assessed using univariate and multivariate analyses, with reviews published in the CDSR and those published in Europe as the reference group for journal of publication and continent of publication, respectively (see Tables 3 and 4). Multivariable analysis demonstrated that the summary PRISMA score was related to journal of publication when both journal of publication and number of authors were considered simultaneously. The quality of reporting was considerably better in Cochrane systematic reviews (P < .001) than in non-Cochrane reviews published in all five of the specialty journals, with systematic reviews published in EJO having a 13.7% lower PRISMA score after accounting for the possible confounding effect of the number of authors. Similarly, both univariable (1.72; 95% CI, 0.96, 2.47; P < .001) and multivariable (0.79; 95% CI, 0.1, 1.48; P < .05) analysis suggested that adherence to PRISMA guidelines was likely to be improved as the number of authors increased. However, no statistically significant relationships were found between PRISMA scores and either continent of publication or length of time since publication. The association between AMSTAR score and modified PRISMA score was found to be highly statistically significant. A mean increase of 1 unit in AMSTAR score corresponds to an almost 3% (2.98%; 95% CI, 2.6, 3.36; P < .001) increase in modified PRISMA score.

Table 3.  Mean PRISMA Score for Journal of Publication, Years Since Publication, Authorship Country, and Number of Authors^a

View Fullscreen

Table 4.  Univariable and Multivariable Linear Regression–Derived Coefficients (β), 95% Confidence Intervals (CI), and P Values for Modified PRISMA Score as a Dependent Variable for Journal of Publication, Country of Authorship, Number of Authors, and Number of Years Since Publication^a

View Fullscreen

DISCUSSION

In a previous study, we assessed the quality of systematic reviews in orthodontics using AMSTAR criteria. This study differs in that it focuses primarily on reporting characteristics rather than review quality. Previous studies evaluating systematic reviews in medicine have examined the reporting compliance based on QUORUM criteria.12,13 The relatively recent introduction of PRISMA complicated the present review as it was developed in 2009 and endorsement in orthodontic journals typically lagged its initial publication by a further year. Nevertheless, it is hoped that the present study will provide a benchmark against which the reporting of further dental and orthodontic systematic reviews can be compared.

Previous research has estimated the ratio of non-Cochrane to Cochrane systematic reviews at 4∶1 based on a MEDLINE search.13 The relative proportion of either review type is similar in orthodontics, with the proportion of Cochrane reviews being slightly higher, although both Cochrane systematic reviews of orthodontic treatment and Cochrane systematic reviews considered relevant to orthodontics were included in the present study. Previous research has highlighted that the reporting quality of Cochrane reviews exceeded that of non-Cochrane reviews in both medical14,15 and dental16 research. This article is the first to substantiate these findings in orthodontic research. The mean percentage score for Cochrane reviews was more than 10% higher than for any of the specialty journals. This is not surprising as the Cochrane Collaboration sets stringent guidance on performance of systematic reviews that incorporates PRISMA guidelines. Furthermore, electronic publications such as Cochrane reviews are not constrained by limits on word counts, thereby permitting more complete reporting of the conduct of the review process; this may be a further reason for the enhanced reporting standard of these reviews.

Clearly, Cochrane reviews were generally reported to a very high standard and fulfilled most criteria in the majority of reviews that were included in this study. However, a systematic omission was noted in the failure to highlight the nature of the review in the title. It could, however, be argued that given the publication of these papers in the CDSR, the inclusion of the term systematic review in the title could be considered superfluous. It is therefore Cochrane policy to not include the words systematic review or meta-analysis in the review title. This was the only criterion in which Cochrane reviews were less compliant with PRISMA guidelines than non-Cochrane reviews.

Significant methodological weaknesses were identified in many of the non-Cochrane reviews, which are likely to have compromised interpretation and conclusions. In particular, there was no evidence of any prospective registration of the protocol in any of the non-Cochrane reviews. Registration of all systematic reviews has recently been advocated in electronic databases17 (eg, PROSPERO, Centre for Reviews and Dissemination, University of York, UK) to include a priori statements outlining review methodology that would more likely reduce the propensity for biased post hoc changes. While publication of protocols in up to 46% of reviews has been reported in the medical literature,13 this practice has obviously not filtered through into non-Cochrane orthodontic reviews. In addition, none of the non-Cochrane reviews disclosed any sources of funding, as opposed to Cochrane reviews in which an acknowledgment of any sources of funding and declarations of conflicts of interest are mandatory. Clear delineation of objectives and eligibility criteria using the PICOS acronym has been advocated in the PRISMA guidelines. Distinct and unambiguous criteria are instrumental in developing a relevant search strategy and in study selection. Ideally, these eligibility criteria should also refer to inclusion of studies with specific outcomes to permit assessment of the possibility of reporting bias due to selective reporting of results.18 Consequently, it is important that the reporting of eligibility criteria be improved in non-Cochrane orthodontic reviews, as 36% were found to be inadequate in the present study, with likely implications for the review process, selection of studies, results, and conclusions.

The search strategy was generally found to be satisfactory, although several of the reviews (17%) failed to provide full details of at least one electronic search. Similar shortcomings were identified in a previous assessment of orthodontic systematic reviews,2 with problems identified including restriction of searches to a single database (typically MEDLINE), omission of names and dates of database searches, involvement of only a single reviewer, and confinement of searches to English-language databases. The central problem related to a competent and comprehensive search is the possibility of omitting relevant articles, resulting in erroneous and possibly biased conclusions.2 Failure to report detailed searches also reduces transparency, preventing verification of the results of the review process. An important reason for failure to undertake comprehensive searches among dental review authors may relate to a perceived lack of return for additional effort.3 In view of the pivotal role played by systematic reviews in the provision of health care, journal editors and reviewers should not countenance inadequate search methods; improved reporting of search strategies in orthodontic systematic reviews should now be considered obligatory.

While little temporal change in the reporting quality was noticed in this study, it may be considered premature to expect the introduction and wider acceptance of PRISMA to have resulted in a significant difference in overall reporting quality in orthodontics given that these guidelines were officially adopted by AJODO in only 2010 and other orthodontic journals including EJO have yet to endorse them fully. The precursor of PRISMA was the QUOROM Statement published in 1999 providing guidance for meta-analysis of randomized controlled trials4; improved reporting of these studies has since been confirmed in medical research. A further problem stems from the suitability of PRISMA guidelines for analysis of reporting of all systematic review subtypes. In particular, PRISMA was initially intended as a template to guide reporting of reviews of the effects of interventions; however, it has also been used to guide diagnostic systematic reviews despite its limitations in this respect. It is likely that reporting guidelines for other types of systematic reviews including those of diagnostic studies and economic analyses will emerge, particularly as other review types are given increasing priority by the Cochrane Collaboration.19

Many orthodontic journals now stipulate strict adherence to PRISMA reporting guidelines, with the result that the reporting of reviews is often quality assured with submissions accompanied by completed PRISMA checklists highlighting compliance with individual criteria. While conduct and reporting of clinical trials are quite distinct, the reporting and underlying methodological quality of systematic reviews are likely to be interrelated. This was confirmed in the present study, which identified a significant association between the PRISMA and AMSTAR scores. This positive association between AMSTAR scores and PRISMA reporting is reassuring and indicates that adherence to PRISMA guidelines is most likely to result in both the enhanced reporting and improved methodological quality of orthodontic reviews.

CONCLUSIONS

• Reporting of orthodontic systematic reviews in leading orthodontic journals was found to be deficient in certain areas, particularly with respect to prospective registration of review protocols, reporting of sources of funding, assessment of reports of risk of bias across studies, definition of summary measures, and detailed explanation of the methods of analysis and eligibility criteria.

• The quality of reporting assessed using PRISMA guidelines was significantly better in orthodontic systematic reviews published in the CDSR.