INTRODUCTION

During the last decades, academic publishing moved radically online, allowing rapid dissemination and monitoring of scholarly work through the web.¹ The advent of the digital age of scientific journals has fundamentally altered how scientific impact is measured and evaluated.² Impact of an article may be influenced by various factors including the significance of the findings, article availability, type, title, subject, length, and authors.³ Traditional citation counting measures, upon which researchers, publishers, and funding bodies have long exclusively relied to gauge research quality and impact, may fall short in capturing the today’s prolific digital publishing activity.⁴ Just as importantly, citations have been often criticized due to delays in inclusion in bibliographic databases that may take up to several years.⁵

Newly emerged article usage statistics (ie, views and downloads) facilitate tracking of attention around scholarly work immediately after publication.⁶ Besides direct availability, easy data recording and measurement of article use from a broader perspective have been described as advantages of usage metrics compared to citation metrics.⁷ Notwithstanding the abundance of usage indicators based on download data, including, for example, download impact factor,⁸ usage impact factor,⁹ usage immediacy index,⁹ download immediacy index,¹⁰ and usage half-life,⁹ raw article view and download counts are the ones routinely used in scholarly communication.

For the same reason, a third group of article-level metrics, labeled as “altmetrics”¹¹ or “social media metrics”¹² has been recently introduced to record the diffusion of articles through blogs, podcasts, social network platforms, and news channels and complement academic impact analysis. Altmetrics are collected by altmetrics aggregators or providers, namely platforms covering a multitude of electronic sources. Altmetric (Altmetric LLP, London, UK) currently appears to be the most prevalent aggregator in altmetric research and digital orthodontic publishing, used in more than half of the published bibliometric/scientometric studies,¹³ and by most of the orthodontic journals providing altmetrics.¹⁴

Given the obvious interest of academia and other stakeholders in scientific impact indicators, several studies have investigated the relationship between usage metrics, citations and, to a lesser extent, altmetrics. Clear links of moderate-to-high strength between the number of downloads and citations, in association or correlation, have been demonstrated in medical specialty^5,7,15–19 as well as multidisciplinary open-access journals.²⁰ No correlation was found between Altmetric scores and downloads in plastic and reconstructive surgery and rheumatology articles.^2,19

Evidence on article usage metrics in orthodontic literature has been lacking to date. Thus, the aims of the present study were to explore the association between article usage metrics, citations and altmetrics, and to compare download counts of articles published in peer-reviewed orthodontic journals in relation to article type, topic, title, and authorship.

MATERIALS AND METHODS

Journal Citation Reports 2022 (JCR 2022; Clarivate, Philadelphia, PA, USA, and London, UK) was screened for orthodontic journals providing homogeneous reporting of article usage metrics (ie, page views, downloads, and total views), citations and Altmetric Attention Score (AAS) at the article metrics-section on the official journal website. Out of 15 journals dedicated to orthodontics listed in JCR 2022, the European Journal of Orthodontics (EJO) and The Angle Orthodontist (AO) met the inclusion criteria (Figure 1, Table 1). As AAS of scientific articles tends to peak during the first 2 years after publication,¹ and article metrics were available for both journals since January 2021, all EJO and AO issues published in that year were reviewed. Free access was provided by the publishers for all issues published in 2021. Short-type articles such as cases reports, editorials/commentaries, and letters to the editor were excluded due to the relatively lower citation frequencies.^21,22 Two researchers (CL, KD) simultaneously retrieved from the journal webpages (https://academic.oup.com/ejo; https://meridian.allenpress.com/angle-orthodontist) the full content of 2021 issues and collected on a consensus basis the following data: article type (ie, original research article, systematic review/meta-analysis), subject (ie, biomaterials, craniofacial growth/oral biology, diagnosis, new technologies, OHRQOL: oral health-related quality of life, practice management, side effects, study design, treatment, and other), title type, number of substantive words in the title, number of authors, article citations, AAS, X (formerly Twitter) posts, and Mendeley reads. Mendeley and X have been identified as the most active Altmetric resources in Cochrane systematic reviews in dentistry²³ and, therefore, were examined for the purposes of the study. Noninformative words (ie, articles, prepositions, conjunctions, pronouns, and auxiliary verbs) were excluded from counting.^24,25 Abbreviations and acronyms were counted as words. Article titles were classified into the following main types: (i) Declarative title: A title that summarizes the main conclusion(s); (ii) Descriptive or neutral titles: A title that describes the topic of the article without revealing the main result or conclusion; (iii) Interrogative or question titles: A title that indicates the topic of the article or the main objective in the form of a question.³ Web of Science platform (WoS, Clarivate, Philadelphia, PA, USA and London, UK; https://clarivate.com/) was accessed through an institutional account to determine article citations, number of publications, and h-index of the last authors.

View Figure

• Download as Powerpoint
• Download Figure

Table 1. Article Usage Metrics and Altmetric Attention Score (AAS) Cited by the Orthodontic Journals Listed in JCR 2022

View Fullscreen

RESULTS

A total of 200 articles were published by AO and EJO in 2021. After excluding (guest) editorials/commentaries, letters to the editor/author responses, and case reports, 161 original research articles and 20 systematic reviews/meta-analyses were eligible for the study.

Descriptive statistics of article usage metrics, WoS citations, AASs, and numbers of X posts and Mendeley readers are displayed in Table 2. The data did not follow a normal distribution, as indicated by the Shapiro-Wilk test and the Kolmogorov-Smirnov test, both of which showed P values < .001 for all variables (Supplementary Table 1). Therefore, a skewed distribution was assumed. The median number of views and downloads of the included articles was 1296 (range: 355–10,233) and 793 (range: 167–3629), respectively, and received a median of 5.1 citations (range: 0–35). Regarding social media metrics, the analyzed articles scored a median AAS of 0 (range: 0–8) and attracted a median of 0 (range: 0–51) mentions on X and 9.1 (range: 0–135) Mendeley readers. The highest numbers of downloads and total views, ie, 3629 and 13,682, were recorded for a systematic review and meta-analysis on the efficacy of skeletally anchored rapid maxillary expansion in late adolescents and adults, which was published in EJO.²⁷ Page views, downloads, and total views were significantly correlated with WoS citations (rho > 0.345; P < .001; Table 3). No significant correlations were found between usage metrics and AAS, X posts, and Mendeley readers (Table 3).

Table 2 Means, Standard Deviations (SD), Medians, Interquartile Range (IQR), and Ranges of Article Usage Metrics, Web of Science Citations, Altmetric Attention Score, X posts, and Mendeley Readership. Both Mean (SD) and Median (Range) Values Are Presented to Provide a More Comprehensive Understanding of the Data’s Central Tendency and Variability)*

View Fullscreen

Table 3 Correlation Results of Usage Metrics With Web of Science Citations, AAS, X Posts and Mendeley Readers^a

View Fullscreen

The distribution of articles per article type and topic, title and authorship groups, is presented in Table 3. The most popular topics were orthodontic treatment aspects, investigated in 35 out of 181 articles, followed by new technologies (14.36%) and side effects (11.60%). With respect to article titles, descriptive was the prevailing type appearing in nine out of 10 articles, while most of the titles (79) were composed of 11–14 words. Broad author collaborations with more than 10 authors were confirmed in more than 43% of the cases. In terms of scientific productivity and impact of the last authors, the majority coauthored more than 100 publications while an h-index higher than 30 was assigned to 38 authors.

Table 4 summarizes the number of downloads within article groups and the comparison among groups. No significant differences in downloads were observed among articles as classified by article type, title type, number of article words, and number of authors (P > .05). There was a significant difference in download counts within subject groups (P < .001) with articles dealing with new technologies, being the most frequently downloaded (Figure 2; Supplementary Table 2). Significant differences were observed when comparing articles in relation to the number of publications (P = .005) and h-index of the last authors (P = .010). In particular, articles signed by authors with 1–40 publications or an h-index ranging between 0 and 30 were downloaded significantly more often than the rest of the articles (Figure 2; Supplementary Table 2).

View Figure

• Download as Powerpoint
• Download Figure

Table 4. Total Numbers, Percentages, Medians, Interquartile Range (IQR) of Article Downloads and Comparisons Of Groups.

View Fullscreen

DISCUSSION

Seeing that research impact assessment is fundamental in the continuous development and success of the scholarly communication,²⁰ comparative studies on traditional and alternative bibliometric indicators may be deemed beneficial. This is the first orthodontic investigation on usage metrics of electronic journal articles, and one of the few available in biomedical sciences.^2,19

According to the results, page views, downloads, and total views of EJO and AO articles were significantly positively correlated with citations. This was in agreement with the majority of the literature examining the association between downloads and citations.^{2,7,15–18,28} Lack of correlation was evident between AAS, Mendeley readers, X mentions, and downloads. Apparently, lay social media users may be less keen on accessing the full text of an article before commenting on or sharing online. Likewise, Ruan et al.² confirmed no association between downloads and Mendeley reader numbers. In contrast, the volume of X mentions has been reported to be statistically correlated with downloads from arXiv, an open-access repository of electronic preprints.²⁹ However, this analysis was limited to a cohort of the 70 most discussed articles on X and, thus, the results should be approached with caution.

As scholars mainly depend on the article title while screening literature for relevant information,³⁰ this study attempted to decipher trends behind download counts. Contrary to evidence pointing to significantly more downloads of articles with question-type titles compared to descriptive and declarative titles,³ no such differences were observed. The theoretical assumption that an informative and longer title may lead to more full-text downloads³ was not confirmed in this study either. Regardless of previous claims,^31,32 a stronger publication record and impact of the authors or broader collaborations were not reflected in article download numbers. Subject appeared to be the only article feature on which significant download differences were seen and, therefore, may be seriously considered in publishing and editorial policies. Seemingly, early and middle-career scholars may more intensively pursue engagement in research innovations to boost their profile than established or more experienced peers.

When interpreting usage metrics results, a distinction has to be made in the information-seeking behavior between authors and readers.⁹ Generally speaking, downloads reflect broader interest expressed not only by researchers but also by practitioners seeking clinical tips, not involved in scientific research and, therefore, unlikely to publish scholarly material themselves.^18,20 In practical terms, even scientists may read/download a plethora of articles before deciding what fits the frame of their research and needs to be referenced.³¹ Aside from scientific writing and clinical practice, downloads may be driven by educational tasks and training activities of faculty, students or information specialists in academic departments or libraries, respectively. Although article downloads cannot be equated with article reads,¹⁷ and every article download does not necessarily result in citation, it can be assumed that the number of downloads basically represents the reader community’s awareness of an article.³¹

This study had some limitations that need to be acknowledged. The exclusion of 13 out of 15 orthodontic journals listed in JCR 2022 due to incomplete reporting of article usage and social media metrics restricted data collection and, thus, results should be read with caution. However, unlike other studies focused merely on a potentially biased sample of highly cited, downloaded, or viewed articles,^19,33,34 statistical analysis was carried out on all articles published in the journals to strengthen the results. The directionality issue,⁵ in other words whether downloads generated citations or vice versa, was not addressed in the current study. As researchers commonly download an article looking for details probably redirected by a specific citation,⁶ no causal or directional claims can be justified. Though the minimum 2-year lag extending from the article publication to the analysis was considered sufficient to measure the reach of the articles in academia,^18,33 there may be opposing voices. It has also been postulated that authors appear to have an exponentially decaying memory in citation behavior with newly published articles being more likely to get cited than older articles.² Additionally, the selection of the specific publication year/time lapse between publication and citation allowed inclusion of a second orthodontic journal and, to neutralize the growth with age effect in download and citation figures, journal articles published within a given year were investigated.¹⁷

Given that downloads and citations may be viewed in some respects as complementary measures, perfect correlation should not be expected. A substantial correlation though, together with the faster accumulation of downloads compared to citations, may suggest the use of article download and view rates as early predictors of future citations.¹⁷ The combined use of download data and citation records by academic departments and granting committees may provide a more comprehensive picture of the impact of researchers’ or applicants’ work. The advantage of immediacy of downloads over citations may be especially valuable for tenure evaluation panels and granting agencies in case of younger evaluatees.²⁸ Expanding on the results, coupling download rates with popular subjects may also help librarians evaluate reader demands and subsequently renew or cancel journal subscriptions. Publication of articles in trending research areas is essential for journals to establish their presence in the competing world of scholarly publishing and, therefore, should be systematically pursued by editors and publishers.²⁸

In view of claimed download variations by journal impact factor and discipline,²⁸ it would be useful to study usage metrics in more journals, including other specialty and general dentistry journals. At the time this study was conducted, complete reporting of page views, downloads, and total views by orthodontic journals covered by JCR was scarce. Along with this, most of the journals displayed unclearly defined metrics varying from “accesses” and “captures” to “views” and “total views and downloads.” Given that certain publisher platforms enable simultaneous download of both PDF (portable document format) and HTML (hypertext markup language) article copies counting two downloads for a single usage,²⁸ direct communication with editorial offices may be recommended for future studies to ensure accurate recording of article usage metrics. Publishers and journals should collaborate to develop mutually agreed article usage metrics, measuring the popularity of articles among readers and, potentially, their scientific significance.

CONCLUSIONS

• The subject investigated in the article, h-index, and number of publications of the last author significantly affected the number of article downloads.

• Article page views, downloads, and total views obtained from journal webpages were positively correlated with citations and, therefore, may serve as an early proxy of upcoming traditional citations.

• Use of common article usage metrics by journals and further research should be encouraged to substantiate the potential value of usage metrics in predicting the scientific impact of articles.

SUPPLEMENTAL DATA

Supplemental Tables 1 and 2 are available online.