INTRODUCTION

A scientific paper starts with an idea or with the formulation of a hypothesis and culminates with publication in a specialized journal. At first sight, it seems an absurdly trivial matter to think about one single letter in this complex, protracted process underlying a study. At the end of this process, ie, in the printed text, one missing or one extra letter will have virtually no influence on the study. But what about the initial stages of the study, which are characterized by a full-scale comprehensive search of the literature? The initial triviality is very quickly put into perspective with an awareness that dealings with literature databases permitting a freetext search are subject to the laws of the chaos theory. One of these laws, also referred to as the butterfly effect, states: “Small variations in initial conditions result in huge, dynamic transformations in concluding events.”1

A comprehensive, unbiased search is one of the key differences between a systematic review and a traditional review.2 A lack of or a failure to find clinical controlled trials may have dire consequences on the quality of a systematic review. Therefore, search strategies2 have been devised and filters3 designed to optimize the electronic paper-locating or -selecting process. The Cochrane Oral Health Group4 revises and updates its highly sensitive search strategy once a quarter based on 96 search queries (approximately 570 terms) with 188 OR functions and 17 AND functions to identify all relevant articles within the scope of this group. Orthodontics as part of this group is represented by the key word “orthodontic” with the truncation sign “*.” If, then, all articles of orthodontic relevance are to be found, the specific key words are combined with “orthodontic*,” but “orthodontic*” entails the implicit exclusion of the word “orthodontist.” To determine whether this exclusion affects other orthodontically important terms and, thus, may have a dramatic impact, a PubMed (US National Library of Medicine) search was carried out for the term “orthodontic*,” and one letter before “*” was deleted before each new search query (Table 1).

TABLE 1. Effect of Truncating the Search Query “orthodontic*” Alone and As a Component in the Search Strategy of Three Systematic Reviews Selected at Random From PubMed and The Cochrane Database

View Fullscreen

Overall, omitting the letter “c” alone led to 186 papers of orthodontic relevance and 30 papers of minor orthodontic relevance not being localized. Of these, 37 missing papers were published in the American Journal of Orthodontics and Dentofacial Orthopedics alone. An additional 196 papers of orthodontic relevance were eliminated through omission of the letters “t,” “i,” and “c” alone. This clearly dramatic impact raised the question of whether consideration had previously been given to this aspect or whether such errors have already been reflected in existing studies.

MATERIALS AND METHODS

Because systematic reviews, being literature-sensitive investigations, are exceptionally susceptible, three orthodontically oriented systematic reviews were selected at random from the 24 localized in PubMed and The Cochrane Database of Systematic Reviews (The Cochrane Library). All three reviews reported their exact search strategy and used the key word “orthodontic*” or “orthodontics” in combined form. The key words from the reviews were collected in their logical sequence as a search query and entered without limits in PubMed.

The search query was then sequentially changed in that “orthodontic*” was truncated step by step by the letters “c,” “i,” and “t.” To ascertain the effects of the omission of individual letters, the difference between the search queries was formed with the boolean operator NOT. The articles found in the difference were classified by both authors jointly with respect to the title and abstract as orthodontically nonrelevant, relevant, or only moderately relevant. Because it was only the impact of the different search query on authentic examples that was of interest, the selected systematic reviews were neither named nor assessed with respect to the results of the present study.

RESULTS

Table 1 shows the effect of the truncation in conjunction with different authentic search instructions. Depending on the type of logic operation, the number of articles not detected is substantial. A variation in just one letter resulted in 152 nonlocalized papers on average, 81% of them of orthodontic relevance. Yet, the opposite effect, the elimination of papers (−418), was also observed, confirming the underlying nonlinear pattern.

“Orthodon*” proved to be the most robust truncation, whereas the word fragment “orthodo*” yielded no more orthodontic articles.

DISCUSSION

The results show clearly that apparently trivial changes in the initial conditions of a search strategy may have a major impact on the concluding event. Chaotic systems are characterized by the fact that the next respective measuring point cannot be predicted. An initial indication of systems developing into chaos is the branching of a previously regular measuring curve at one measuring point. This kind of bifurcation seems to occur in the three search strategies selected at random for the purpose of the present study, as indicated by the sequence of the number of nonlocalized articles: 273, 69, 141, −418.

The constant improvement in user guidance and sophisticated search functions (automatic term mapping, filters, etc.), as provided for example by PubMed and other state- of-the-art retrieval software, creates the impression that these systems have cognitive skills. However, the user should bear in mind that nothing more is involved than the comparison of a specific sequence of characters with other sequences of characters in a database.

Our demonstration using authentic search statements showed clearly that there is a marked, but nonpredictable, number of nonlocalized articles despite meticulous care being taken in the studies in question and an awareness of systematic errors, especially that of publication bias.2 The more complex the search query, the more likely the system is to reach a threshold of chaos. This phenomenon is not confined to “orthodontic*” but can be observed with any key word, so that the chaotic system is given feedback by the consequently increasing combination potential. Because such systems are indeterminable, the influence of one letter on the study as a whole remains unpredictable. The worst case scenario for a systematic review would be the failure to locate an existing systematic review on the same subject—a very real risk in chaotic systems.

CONCLUSIONS

Search queries for orthodontic articles should invariably be provided with the robust truncation “orthodon*.” As with the mathematical simulation of chaotic systems, the outcome of the search should be verified by iteration, ie, the key words used should be truncated step by step and reentered into the actual search function. Although this manual procedure is time-intensive, it does offer maximum security.