INTRODUCTION

Since facial profile changes can accompany orthodontic treatment, photograph-cephalogram image integration has commonly been used in various clinical orthodontic situations, such as in planning treatment,^1–3 predicting treatment outcomes,^4–7 and predicting facial growth.^8,9 Authors of these studies typically analyze and predict soft tissue changes from lateral cephalometric radiographs; however, to present these changes more realistically, visualization of the changes through photographs is necessary.

Authors of all previous prediction studies used manual integration results to match the soft tissue appearance in the digital photograph with that in the radiograph. Such manual image integration is a laborious procedure that heavily depends on clinician skill in aligning the photograph and cephalogram using soft tissue curves and landmarks. However, interest is growing in applying artificial intelligence (AI) models in clinical orthodontic practice. For example, to reduce manpower burden and increase objectivity, various approaches have been used to improve the accuracy of automatic identification of cephalometric landmarks, and authors of a recent study showed that the performance of AI models was comparable with that of human experts.^10–13 Currently, although some commercial cephalometric programs can provide orthodontic clinicians with a kind of automated photograph-cephalogram image overlay function, it is difficult to find an explicitly reported method to implement such automated photograph-cephalogram image integration.

The purpose of this study was to develop an automated method for combining digital photography and lateral cephalography through the application of AI models. To evaluate the clinical applicability of this automatic integration method, its image integration accuracy was compared with that of a traditional manual integration method.

MATERIALS AND METHODS

All training, test, and validation photographs and cephalograms were collected from the picture-aided communication system server (Infinitt Healthcare Co Ltd, Seoul, Korea) at Seoul National University Dental Hospital, Seoul, Korea. The institutional review boards at Seoul National University Dental Hospital (ERI 19007) and Seoul National University School of Dentistry (S-D20200015) approved the research protocol.

Training Two Different AI Models

To develop an automated method for combining a photograph and cephalogram, two different AI models were created: one for detecting soft tissue profiles on photographs and the other for identifying soft tissue curves on cephalograms.

A total of 985 digital photographs were collected to create an AI model capable of automatically identifying soft tissue landmarks on profile photographs. To train the AI model, a total of 18 soft tissue landmarks, from glabella to the cervical point (Table 1), were manually defined on these photographs by an examiner (JHM). Subsequently, this labeling information was used to train a residual neural network (ResNet), which is a deep-learning method frequently used for image identification problems.¹⁴

Table 1. A Total of 32 Soft Tissue Landmarks Was Used to Train the AI to Detect Landmarks in Lateral Cephalometric Images. Eighteen Soft Tissue Landmarks on Digital Photographs, Marked With a Symbol ■, Were Used to Train the AI. To Validate the Integration Methods, Eight Landmarks Marked With Were Selected and Used for Comparisons Between Automated and Manual Integration Methods

View Fullscreen

Then 2500 lateral cephalograms were collected, and a total of 32 soft tissue landmarks, from glabella to the soft tissue terminal point (Table 1, first column), were manually detected on these cephalograms by another examiner (SJL) to create an AI model that could identify soft tissue landmarks on lateral cephalograms. This AI model was based on the you-only-look-once version 3 (YOLO-v3) algorithm, which is a deep-learning method developed for real-time object detection.^15,16 YOLO-v3 was reported to demonstrate higher accuracy than other machine-learning methods in automated cephalometric analyses and the detection of multiple cephalometric landmarks.^11–13,15

Automated Integration Method

Figure 1 demonstrates the automated integration method. Initially, the two AI models automatically identified landmarks in digital photographs and lateral cephalometric images. Once the 18 landmarks which are common to both modalities were identified, the digital photograph was rotated, scaled, and shifted to minimize the squared sum of distances between the corresponding soft tissue landmarks identified by the two different AI models.

View Figure

• Download as Powerpoint
• Download Figure

In mathematical terms, the automated integration by the AI was equivalent to finding the transformation condition T that minimizes the root mean square (RMS) argument where and represent the position of the ith soft tissue landmark automatically identified in the digital photograph and lateral cephalometric radiograph, respectively, n is the total number of soft tissue landmarks identified in the digital photograph, and ‖ ‖ stands for the Euclidean distance measure calculated in millimeters (Figure 1).

Manual Integration Method

The manual procedure for combining photographs and cephalograms is a common practice used by orthodontic clinicians. In contrast with computer-based methods that rely on multiple soft tissue landmarks, human clinicians can easily draw and connect a smooth outline of soft tissue without much difficulty.

To facilitate the manual integration procedure, custom-made software was developed in the Python programming language (Python Software Foundation, Wilmington, Del) to move, rotate, and resize the soft tissue outline as needed. Figure 2 illustrates the manual integration of soft tissue profiles obtained from the lateral cephalometric radiograph and digital photograph. Three guidelines were used: the soft tissue outline from glabella to the cervical point, based on landmarks identified by the AI model; the Frankfort horizontal plane; and the sella-nasion plane. The manual image integrations were performed by a different examiner (MGK).

View Figure

• Download as Powerpoint
• Download Figure

Validation via Comparisons Between Automated and Manual Integration Methods

To validate the automated image integration method according to the common validation protocol,^17,18 new data not used during the training or learning procedures were collected from 100 randomly selected additional subjects who had undergone both lateral cephalometric radiographs and digital profile photographs for diagnostic purposes. Individuals whose images were used in the training of either of the two AI models were excluded from the validation dataset.

For each of these 100 validation subjects, eight soft tissue landmarks (glabella, soft tissue nasion, pronasale, subnasale, upper lip, lower lip, soft tissue point B, and soft tissue pogonion) were manually identified on both lateral cephalometric images and digital photographs by a different human examiner (SJC). Custom-made digitizing software developed in Python was used to record x and y coordinates of each landmark relative to the Cartesian coordinate system. For each manual integration, the coordinates of the soft tissue landmarks on the digital photograph were transformed according to the transformation implemented by the AI model when it transformed the digital photograph to match the soft tissue profile of the lateral cephalometric image (Figure 3). Then RMS values in millimeters were measured between the coordinates of the soft tissue landmarks obtained from the lateral cephalometric image and the coordinates of the transformed soft tissue landmarks on the digital photograph, both involving eight landmarks that were manually identified. These values were used to compare the accuracy between the two image integration methods.

View Figure

• Download as Powerpoint
• Download Figure

Paired t-tests were conducted to compare the accuracy of the measured RMS for each landmark and to compare the pooled RMS values. Statistical significance was set at P < .05. All statistical analyses were carried out using R language.¹⁹

RESULTS

The paired t-test results for each soft tissue landmark showed statistically significant differences in RMS values between the automated and manual image integration methods for the upper lip (0.37 mm, P < .01) and soft tissue B point (0.52 mm, P < .01). The other six landmarks did not show a statistically significant difference.

The pooled mean RMS values measured from the automated and manual soft tissue integration methods were 2.97 ± 0.95 mm and 2.92 ± 1.04 mm, respectively, and did not show a statistically significant difference (Table 2).

Table 2. To Compare Accuracy Between the Automated and Manual Image Integration Methods, Eight Soft Tissue Landmarks Were Selected on the Digital Photographs and Lateral Cephalometric Radiographs for the 100 Validation Subjects. For Each Integration Method, the Root Mean Square (RMS) Values in Millimeters Were Calculated Between the Position of a Given Soft Tissue Landmark in the Lateral Cephalometric Image and Its Transformed Position in the Digital Photograph, and Paired t-Tests Were Conducted

View Fullscreen

DISCUSSION

The purpose of this study was to develop and evaluate an automated method for combining digital photographs with lateral cephalograms. To develop this automated integration method, 18 soft tissue landmarks on digital photographs and lateral cephalometric images were automatically identified by two different AI models, and the images were then transformed to minimize the distances between these soft tissue landmarks in the two image types. We are the first to focus on developing an automated photograph-cephalogram image integration method and to evaluate it in comparison with the traditional manual process. The results of the present study showed that applying multiple AI models and the least squares concept might be as reliable as manual superimposition procedures.

The accuracy of the automatic identification of soft tissue landmarks could affect the accuracy of image integration in the later step. When developing an accurate AI model, not only the characteristics of the target variables but also the AI data size should be considered.²⁰ However, no solid guidance exists to determine appropriate sample sizes when developing an AI model.²¹ Moon et al. (2020)¹ suggested that at least 2300 cephalograms would be necessary to develop an automatic landmark identification AI model that would be as accurate as human examiners. Accordingly, in the present study, we collected 2500 cephalograms, manually identified cephalometric landmarks, and developed an AI model using the manually labeled data. Additionally, to estimate the number of digital photographs required, a preliminary study was conducted using 200 digital photographs as learning data. The results of this preliminary study demonstrated a failure rate of approximately 10% in the detection of 18 soft tissue landmarks in the test data. According to the sample size estimation guidance by Moon et al.,¹¹ 985 photographs would be needed to increase the accuracy of the AI model to an acceptable level that was characterized by a human interexaminer difference of 1.5 mm.^11,17

Considering that combining photography and cephalometry is often necessary with particular care in several clinical areas of orthodontics, it was unexpected that only a few studies on this procedure could be found in a literature review.^2,3 The study of Dvortsin et al. (2011)² appears to have been the first study to focus on the issue of photograph-cephalogram image integration. The authors suggested a method for reorientating the lateral cephalogram to the natural head position (NHP) according to a standardized photograph taken at NHP.² However, this method was too heavily dependent upon the NHP photograph, and it might not be feasible to obtain this in clinical orthodontic practice. Unlike cephalometric radiography, for which a tripod head stabilizer or cephalostat is used, photography does not generally involve a head holder that can fix head orientation in a repeatable manner. Consequently, profile photographs may have elongated and/or foreshortened interlandmark distances in addition to random intersubject variability. Therefore, a more sophisticated method for photograph-cephalogram image integration or a standardized method for overlaying two images might be necessary to develop an automated process.

Manual image integration is commonly performed by overlaying two images according to their profile outlines. Like manual procedures, the photograph-cephalogram image integration method proposed by Wang et al. (2018)³ used profile outlines. Their method appeared to provide an automatic solution for image integration using a hierarchical contour detection algorithm to achieve image congruence of soft tissue outlines of the forehead on the lateral cephalogram and photograph.³ However, this method seemed to be a bit theoretical. In practice, its application would still require corrections to control for curve deviation. Even more seriously, image integration accuracy could be vulnerable to the heavy reliance on a short outline span limited to the forehead region.

The image integration method used in the present study depended on multiple soft tissue landmarks that were automatically detected on both photographs and cephalograms. The idea of depending on multiple landmarks (MLs) and seeking a least sums of squares solution was inspired by studies on automated cephalometric superimposition.^12,22 Authors of these cephalometric image superimposition studies used MLs located on the cranial base to align two cephalographic images so that the distance between each landmark was minimized. This ML superimposition method is known to have better accuracy than the conventional sella-nasion-line superimposition method. The ML superimposition produced results like those of Bjork’s superimposition method, especially in the evaluation of growth changes in growing children.^12,22 While the ML superimposition method overlayed two cephalometric images automatically, in the present study, we aligned and overlaid images from two different modalities: digital photography and cephalography. However, the basic idea of using MLs and pursuing the least squares solutions was the same.

To facilitate three-dimensional (3D) visualization during orthodontic diagnosis, several methods for overlaying 3D tomographic images and digital dental models have been introduced.^23–25 While these 3D image integration methods are likely to gain popularity in the near future, it is also true that 3D images are not routinely obtained during a patient’s first visit, with planar two-dimensional photographs and lateral cephalograms being more commonly used in clinical orthodontic practice.²⁶ It may be some time before 3D image integration tools become routinely accessible.

A limitation of this study was that the method was dependent upon the landmark-based approach, and a relatively large number of 18 landmarks needed to be found on both photographs and cephalograms. However, these limitations might no longer be a significant barrier for today’s practice environment since most contemporary commercial cephalometric software programs provide automatic landmark detection functions. Therefore, the automatic image integration method of the present study might be compatible with the current clinical environment. In addition, since the photograph-cephalogram overlay method of the present study uses 18 soft tissue landmarks, deviations or aberrations in some of the landmarks would not cause a significant impact on the pooled mean RMS values or the quality of the whole image integration. Even the statistically significant RMS values shown in Table 2 might not be clinically significant if each landmark is positioned within the profile curves. Applying the simple and intuitive least squares concept might be another advantageous feature of this method. The simple least squares idea using multiple soft tissue landmarks that forms the focus of this study could be easily implemented in most cephalometric programs currently available for use in clinical orthodontic practice.

CONCLUSIONS

• The automated photograph-cephalogram image integration method using AI models seemed as reliable as manual superimposition procedures.

• The method may be particularly compatible with contemporary cephalometric programs providing automatic landmark identification functions.