INTRODUCTION

Craniofacial growth may play an important role in the success of orthodontic treatment.^1,2 The reliable prediction of patient mandibular and maxillary development could help in understanding the best therapeutic decision regarding treatment timing, appliance choice, and the possible need for surgery.³ As most orthodontic patients are growing individuals, orthodontists have to consider their craniofacial growth path for successful treatment planning.⁴ However, individuals with the same chronologic age may have different growth patterns regarding onset, duration, speed, direction, and amount of residual growth, as shown in several studies.^5–9

Many indicators have been suggested to evaluate the timing of mandibular growth peak and skeletal maturation,^10–12 and the most used methods for this assessment are based on radiographic analysis. Specifically, numerous authors investigated the relationship between mandibular growth and skeletal maturation estimated by means of hand-wrist bone analysis (HWBA) or the cervical vertebrae maturation method (CVMM).^3,13–19 Since the CVMM is advocated as a timing tool for orthopedic treatment, its validity and reproducibility should be assessed by procedures without any methodologic shortcomings.²⁰ Different investigations have studied the association between the CVMM and skeletal maturity and the CVMM and mandibular growth spurt, testing the validity.^21–31 Several studies found a good correlation between CVMM and HWBA, suggesting the possibility to use the CVMM instead of the HWBA to reduce the radiation dose.^21–26 However, a recent systematic review of the CVMM was not able to establish the validity of this method due to the lack of moderate/high quality papers on this topic.³² Several studies criticized the validity of the CVMM, showing that it is unable to predict the start of the peak in mandibular growth.^27–29 Furthermore, it has been shown that the effective radiation dose for a lateral cephalogram without a thyroid shield is 1.5 fold higher than the effective dose for a lateral cephalogram with a thyroid shield plus a hand-wrist radiograph.³³

On the other hand, the reproducibility of the CVMM, which can affect its clinical usefulness, is strongly debated. Very high levels of interobserver and intraobserver reproducibility (over 90%) were reported by some examinations.^18,22,24 However, these studies have some limitations, such as the analysis of traced vertebrae on the lateral cephalograms instead of actual radiographs, the presence of the authors among the group of judges performing the analysis, the use of small sample sizes, and the use of an improper statistical method.²⁰

Some authors have succeeded in the investigation of the reproducibility overcoming the previous methodologic weakness but without exploring the influence of clinical experience on the reproducibility of the CVMM.^20,30,31,34 Indeed, there is little information about the impact of judges' clinical experience on the CVMM, even though it should not be underestimated.^20,35 Therefore, the aim of this study was to evaluate the interobserver and intraobserver reproducibility of the CVMM among three judge panels with different levels of orthodontic experience (OE). The null hypothesis was that the orthodontic clinical experience did not have any influence on the reproducibility of the CVMM.

MATERIALS AND METHODS

The study was approved by the Local Ethical Committee of the University of Naples Federico II. Fifty good quality individual lateral cephalograms of patients attending the School of Orthodontics of the University of Naples Federico II, with complete visualization of cervical vertebrae 1 through 4, were selected for our sample. The cephalograms were randomly chosen from the school's electronic database, by means of the random function of a scientific calculator (EL-506VB, Sharp Corp, Osaka, Japan) and were equally divided by sex (mean age 12.4 years ± 3.2; 25 female and 25 male). Afterwards, the original lateral cephalograms were searched and scanned at 600 dpi (Perfection V750 Pro, Seiko Epson Corp, Suwa, Japan) for presentation as high-resolution images in TIFF format to maintain the radiographic quality.

To avoid any additional information that might influence the observer during the evaluation of the CVMM (as stage of dentition), the lateral cephalograms were cut to include only cervical vertebrae from C1 to C4.

The judges were divided into three groups according to the level of clinical experience: the junior group (JU) formed by 10 recent graduates in dentistry, with less than 1 year of orthodontic experience (mean age 25.1 years ± 1.2; OE 0.4 years ± 0.5; 6 female and 4 male); the postgraduate group (PG), comprising 10 postgraduate students in orthodontics with clinical experience ranging between 2 and 4 years (mean age 29.7 years ± 1.4; OE 2.6 years ± 0.5; 6 female and 4 male); and the specialist group (SP) including 10 specialists in orthodontics with more than 7 years of orthodontic experience (mean age 41.8 years ± 10.3; OE 19.1 years ± 10.4; 3 female and 7 male). None of the clinicians recruited for the assessment participated in the study design. All of the judges belonged to the School of Orthodontics of the University of Naples Federico II as tutor/professor (SP), postgraduate student (PG), or voluntary frequenter (JU).

Each observer was invited to perform two sessions of evaluation of cervical stage on the lateral cephalograms according to the method suggested by Baccetti et al.³ Between the two sessions, a 3-week time interval was respected (T₁  =  initial; T₂  =  3 weeks). The cephalograms were presented in a high-resolution file, randomly ordered for the two sessions. Before the first session, all participants attended a lecture on the CVMM by one investigator (Dr D'Antò). Moreover, at the start of each session the observers also received a copy of the paper by Baccetti et al.,³ and beside each cephalogram a schematic representation of the CVMM was shown as well.

RESULTS

The 30 participants performed a total of 3000 evaluations at two different time intervals. The Kendall's W coefficient for each group varied from 0.61–0.87. The interobserver agreement was the highest for the JU group in both time intervals and showed an almost perfect agreement (T₁ W  =  0.87; T₂ W  =  0.86). On the other hand, the SP group achieved the lowest Kendall's W values presenting a substantial agreement (T₁ W  =  0.61; T₂ W  =  0.78). The total interobserver agreement for the 30 participants at the two time intervals varied from W  =  0.70 at T₁ (substantial agreement) to W  =  0.81 at T₂ (almost perfect agreement) (Table 1).

Table 1. Kendall's W Coefficient for Interobserver Agreement at Two Time Intervals (T₁ and T₂)

View Fullscreen

A total of 21,750 comparisons between the evaluations for each couple of judges, for each session, was analyzed, to assess the Inter-PPA. The Inter-PPA was 42.3% (N  =  9204/21,750) at T₁ and 46.3% (N  =  10,082/21,750) at T₂. The percentage of one stage apart disagreement was 40.1% (N  =  8721/21,750) at T₁ and 41.4% (N  =  9011/21,750) at T₂ (Table 2).

Table 2. Percentage of Interobserver Perfect Agreement (Inter-PPA) With Cervical Stage Differences

View Fullscreen

Table 2. Extended

View Fullscreen

The intraobserver agreement for each observer ranged from κ  =  0.24 fair agreement to κ  =  0.81 almost perfect agreement. Among the 30 participants, however, only three showed a fair agreement, while 18 showed a substantial agreement, two showed an almost perfect agreement, and seven a moderate agreement (Table 3).

Table 3. Weighted Cohen's κ Coefficient for Intraobserver Agreement and Percentage of Agreement by Observers^a

View Fullscreen

The group showing the best result was the JU, where the κ coefficient ranged from moderate agreement to almost perfect agreement. Moreover, the weighted Cohen's κ coefficient for the whole group was the highest (κ  =  0.78 substantial agreement). In the PG results there was one fair agreement but also one almost perfect agreement, while the SP had the worst performance. The latter two groups, PG and SP, showed a similar Cohen's κ coefficient for the whole group (κ  =  0.64 substantial agreement; Tables 3 and 4). For each observer the comparison between the evaluation at T₁ and T₂ of the 50 assessments was analyzed to evaluate the Intra-PPA. The total Intra-PPA was 54.7% (N  =  821/1500), while the percentage of intraobserver one stage apart disagreement was 34.3% (N  =  515/1500). The JU showed the highest Intra-PPA of 57.8% (N  =  289/500), and 34% (170/500) of their disagreements were of just one cervical stage (Table 5).

Table 4. Weighted Cohen's κ Coefficient for Intraobserver Agreement by Orthodontic Experience Groups

View Fullscreen

Table 5. Percentage of Intraobserver Perfect Agreement (Intra-PPA) With Cervical Stage Differences

View Fullscreen

The Top 10 group was formed by six observers of the JU, only one of the PG, and three of the SP (Table 3). This group achieved the best results in the Inter-PPA, Intra-PPA, and Kendall's W (Tables 1, 2, and 5).

DISCUSSION

The aim of this study was to analyze the influence of the OE on the reproducibility of the CVMM using three judge panels with different levels of clinical experience. The SP was the group with the highest clinical experience (OE  =  19.1), but this group achieved the lowest values of W, κ, Inter-PPA, and Intra-PPA. On the contrary, the group with less than 1 year of clinical experience (JU), showed the highest values for all of the parameters investigated. These results suggest that the OE does not improve the reproducibility of the CVMM, and they could be explained by the different familiarity with the CVMM. Indeed, during their last year of undergraduate courses at the University of Naples Federico II, the JU were educated on the CVMM and they were extensively trained on this method. On the other hand, all of the components of the SP group did not use the CVMM in their daily clinical practice. This might lead to confirmation that to correctly use the CVMM, a more in-depth training may be critical.

The interobserver agreement for the overall group of observers was 0.81 at T₂, higher than that reported by Gabriel et al.²⁰ (0.74). The main differences between the two works are the number of participants, the sample size, and the division of judge panels according to the level of clinical experience. Although the results are similar for the weighted Cohen's κ and for the Intra-PPA for all observers, in this study the JU and the PG groups reached a level of intraobserver and interobserver agreement higher than the results showed by Gabriel et al.²⁰

These findings highlight that, probably, the level of practice and knowledge of the CVMM might be an important factor for its reproducibility, and that a simple use of a handout is likely not sufficient to obtain a good level of knowledge and familiarity with the CVMM.

In Zhao et al.,³¹ the CVMM was explained to the observers by means of training sessions, and it showed values of W and κ coefficients similar to our best performing sample (JU). Hence, to correctly use the CVMM there might be a need for multiple training sessions to understand how to assess the cervical stage and to acquire a consistent method of evaluation. Interestingly, the SP and PG groups showed an increase of the Kendall's W between T₁ and T₂ probably due to a training effect.

In a very recent study,³⁴ the CVMM was tested for accuracy and reproducibility, comparing the cephalometric evaluations of the concavities and shape of the cervical vertebrae with the visual evaluation of CVMM. The authors found the visual method accurate and reproducible; moreover, they stated that in their group the reproducibility was high, independently of the level of the experience of the observers, which is similar to our results.

In the current investigation, albeit the CVMM showed a good reproducibility, the Inter-PPA and Intra-PPA were low. In fact, in the JU group, the Intra-PPA was 57.8% as opposed to a high κ coefficient of 0.78. This means that in almost 4 of 10 cases a clinician could have evaluated a cervical stage differently and might have changed his treatment plan. A similar situation was observed for the other two groups as well. Moreover, the Inter-PPA was similar to the that found in the studies by Gabriel et al.²⁰ and Zhao et al.,³¹ and it exceeded the 40% with a peak of 51.9% for JU at T₁. This means that a clinician has almost a 50% of chance to disagree on cervical staging assessed by others. It has to be taken into account that the value of the weighted Cohen's κ coefficient increases with the number of categories of the assessed method; therefore, this can be a possible explanation of the difference between the high κ coefficient and the low perfect agreement.³⁷ Finally, it is interesting to note that in almost all three judge panels, on average, the 42% of the cervical staging were judged with only one stage apart of difference for both interobserver and intraobserver analysis. This might be another factor that affects the κ coefficient; in fact, having most disagreements in just one category increases the weighted Cohen's κ. Therefore, even if the weighted Cohen's κ and the Kendall's W coefficients were sufficiently high, the level of perfect agreement in the intraobserver and interobserver analysis seems to be not adequate enough to support the reproducibility of the CVMM as a method to evaluate skeletal maturation. Also analyzing the top 10 group, even if the achieved results were better than the results of the other groups, there was still a low level of Inter-PPA and Intra-PPA that could affect the clinical decision, and so the usability of the CVMM.

Having a powerful means to predict the residual potential growth in young patients, without any added biological cost, is clinically helpful, and the CVMM could assume an important role in orthodontics, especially during clinical diagnoses and treatment planning decisions.³ However, the level of reproducibility of this method may affect the clinical decision in the orthopedic orthodontic treatment.

One limitation of this work is that it cannot provide information on the validity of the CVMM due to the lack of a longitudinal sample analysis. Moreover, without using a gold standard observation there was not the possibility to assess the accuracy of the CVMM.

CONCLUSIONS

• The main finding of this research was the lack of influence of orthodontic clinical experience on the interobserver and intraobserver reproducibility of the CVMM. The group with the lowest level of OE had the best performance. Hence, high level of orthodontic experience does not increase the reproducibility of the CVMM.

• The Inter-PPA (36.1%–56.1%) and the Intra-PPA (52.6%–66.0%) of the CVMM were too low to suggest the exclusive use of the CVMM in the assessment of skeletal growth.