Study Type

As shown in Table 2, almost 60% of articles drew conclusions regarding changes in occlusion attributable to treatment interventions. The smallest category was biology studies, with 5% of the articles. AJO-DO had the largest proportion of articles describing population characteristics. Angle offered proportionally more reports on treatment effects. EJO presented more science regarding the biological characteristics of orthodontics (χ² = 24.83, df = 8, P = .002, φ = 266).

Table 2.  Domain of Generalization in Articles Published in 2020 in Three Orthodontic Journals^a

View Fullscreen

Table 3 summarizes the results concerning the unit of analysis. More than three-quarters of articles generalized to the individual human, usually a patient. When tissues, animals, mathematical models, or phantoms were used, this was labeled “nonpersons,” and these accounted for 145 of the studies (41%). Secondary analyses comprised about 10%. These were SRs and MAs where previously published articles were the unit of analysis. EJO reported fewer studies with patients as the unit of analysis and more secondary work (χ² = 17.47, df = 4, P = .001, φ = .223).

Table 3.  Subjects, Units of Analysis^a

View Fullscreen

Table 4 displays studies by research design. Retrospective studies, at 32%, and studies using latent structure or regression designed to explain observed outcomes, were used in 28% of publications. Prospective and descriptive studies of naturally occurring populations each accounted for about 20%. AJO-DO was less likely to publish research with prospective designs and Angle less likely to publish observational studies. EJO was less apt to publish observational or retrospective research. It did not publish any case studies in 2020 (χ² = 36.49, df = 6, P < .001, φ = .323).

Table 4.  Research Design^a

View Fullscreen

As shown in Figure 1, RCTs to test interventions to improve patient occlusion accounted for 9% of the published articles. Prospective designs were well represented (about one-third of studies) but used about twice as often for research with animals, modeling, tissue, or phantoms (χ² = 20.96, df = 1, P < .001, φ = .060).

View Figure

• Download as Powerpoint
• Download Figure

Study Analysis

Integrity of the data sets and statistical tests are shown in Table 5. Power tests and tests for parametric properties of data appeared in fewer than half of the studies where these would have been appropriate. The effect of covariables (such as age and sex of patients) on tested outcomes was considered in few cases and tested in only 7%. One-third of the studies had multiple dependent variables. This was especially true for cohort studies at 55%. Almost no studies reported make appropriate corrections. The Bonferroni adjustment is simple, but conservative. Multiple analysis of variance (MANOVA) is preferred when it can be performed.

Table 5.  Design Features and Statistics^a

View Fullscreen

Parametric and nonparametric tests for differences between two means were used in almost half of the reported work. Other common tests were chi-square, correlation coefficient, one-way ANOVA, and regression analysis. Less common were tests for complex experimental designs such as repeated-measures ANOVA, N-factorial ANOVA, and latent structure methods.

Study Reporting

Use of graphic and tabular presentations of information in the materials and methods section and results section of articles are displayed in Table 6. In almost half of the studies, diagrams were used to help readers understand landmark and measurement locations. In about a quarter each, flow charts and pictures of equipment were presented. Consistent with the underreporting of covariables, data on characteristics of the sample other than independent and dependent variables were presented in fewer than 10% of cases. Tables showing average and standard deviation scores for various classifications of the outcomes were common in every applicable case (80% overall). These were supplemented with bar graphs, scatterplots, and sometimes box-and-whisker diagrams where tests were performed on differences between group averages or associations among variables. The second most common way of reporting outcomes (about one-third of the cases) was by using photographs. The expected use of data reporting associated with type of project was observed. For example, forest plots always appeared for MAs, bias tables were very common for SRs, and before and after photos and patient chart information always accompanied case reports.

Table 6.  Reporting

View Fullscreen

Several demographics features of the literature were also recorded. A total of 1025 author names appeared on the 350 publications. These were categorized in five geographic regions. Almost one-third (31%) of authors were from Europe, followed by Asia (21%), the United States and Canada (18%), Latin America (16%, almost entirely from Brazil), and the Middle East (14%, mostly Turkey and a few from India). EJO featured more articles from European authors (χ² = 24.32, df = 4, P < .001, φ = .069).

All three publications used a two-column, 8.5 × 11 format so it was possible to compare allocation of text to sections, measured in inches. The averages were 10.1 column inches for the introduction; 45.5 for materials and methods; 48.7 for results; and 28.3 for discussion, limitations, and conclusions. Graphics were measured in inches in their appropriate sections. The typical article had 33.4 references. The shortest discussion sections (averaging 29.0 inches) were found in EJO (χ² = 31.24, P < .001), and the fewest references (averaging 28.0) were found in Angle (χ² = 19.30, df = 2, P < .001, φ = 055).

Strengthening of Standards

Study standards are beginning to appear regularly, such as CONSORT for reporting RCTs; STROBE for observational studies; COREQ for qualitative studies; PRISMA for reporting SRs; and Grading of Recommendations Assessment, Development, and Evaluation and AMSTAR for critiquing them. But these were not as widely evident as were forest plots for MAs and before-and-after images for case presentations.

Literature About the Literature

The findings in this summary were generally consistent with the articles exploring individual features in the literature. Baumgartner et al.¹⁹ also studied AJO-DO, Angle, and EJO, retrospectively, prior to 2012 and found that AJO-DO reported cases but Angle and EJO carried more “research” articles. Bearn and Alharbi³ reported a different order for publishing research articles in the period 2008 through 2012, with Angle being the least scholarly. Pandis and colleagues⁹ presented an overview article in the Journal of Dental Research characterizing “publication waste” as “insufficient information to make clear decisions.” They found adequate sample size calculations in no more than 36% of studied articles, adequate randomization in no more than 68%, and management of covariables in fewer than 43% of studies. Koletsi et al.¹³ reported that sample sizes were inadequate in 71% of the studies they reviewed. Seehra et al.¹⁴ found adequate sample size calculations in only 52% of the articles. In the current study, power tests were performed in 43% of the cases, and management of covariables was observed in 17%. Seehra et al.¹⁴ also reported that 52% of studies of treatment outcomes used only single-site data, 23% were prospective, 19% retrospective, and 53% used cohort designs. In the current article, these numbers were 83%, 63%, 27%, and 37%. Donatelli et al.²⁰ reported that 47% of the articles they reviewed in five orthodontic journals reported consistency of measurement, with intraclass correlation coefficient (ICC) being the most common method. In this research, 52% of articles reported measurement consistency. The distribution of methods was comparable. Satisfaction with the quality of RCTs and summary reporting in orthodontics generally tended to group in the mediocre range, with slightly less than half of articles meeting reviewers' criteria.^5–9 A recent trend called “RCT spin” has been identified where authors claim in the materials and methods section to have used high levels of research design that cannot be confirmed in the results section. Flint and Harrison⁵ placed the proportion of “spin RCTs” at 42%. Harrison⁷ put it at 55%. The review article by Koletsi et al.²¹ in the Journal of Evidence-Based Practice reported that only 40% of claimed RCTs they reviewed met the criteria. In the current study, articles were classified as RCT if the authors labeled them as such, but studies claiming to randomly assign subjects to treatment conditions were very rare. Chen et al.⁴ reviewed five leading orthodontic journals in 2019 and reported that the top level of evidence, using the GRADE system, was achieved in only 1% of articles analyzed.

Opening the Question

Recent attention in the secondary literature focused on strengthening the RCT model to enhance protection of claimed treatment benefits from bias. Claims for differences between groups were the most common study reported in the literature sampled. However, these were usually retrospective and performed at a single location, perhaps with a single operator or common protocol for patient selection. Analysis of variance tests (including the t-test and its nonparametric versions) were the most popular form of statistical analysis, and often, multiple such tests were applied to a single study. Although calls for improving this model are appropriate, the CONSORT standard is actually silent on this approach. Instead, it is urged that measures of effect be reported to quantify the anticipated impact of intervention on outcomes of interest.

More than a quarter (28%) of the 2020 articles studied used a regression approach intended to measure effect and to identify which of several candidate sources of variance mattered most to producing the observed outcomes. More regular use of regression approaches would have the double advantage of addressing concerns over covariables and increasing the prominence of measures of effect, such as proportion of explained variance. A full enumeration of the methods of analysis suggested a trend toward more elaborate analysis, including latent stature approaches that were not even possible a few decades ago.

The orthodontic research literature is diverse and representative of the range of inquiry now being used to build stronger practices. Although conducted primarily by residents under the tutelage of university faculty²² and responsive to those dedicated to evidence-based dentistry, it should retain its dynamic nature. It has become truly international^17,18 and represents a diversity of topics and approaches. It is easy enough to find examples of design and methodology that could be improved or at least standardized, but that, in itself, is not sufficient. What is needed is scholarship that deepens our understanding of theory and promotes better practice. This involves both high standards and a large pipeline. The true guardians and promoters of quality are the reviewers and editors who identify articles that address the most important topics and expect rigor that will reduce the potential for plausible unexplained causes.