INTRODUCTION

Orthodontists, like other health care professionals, rely on patient cooperation for successful treatment outcomes, and an important factor in this success is patient satisfaction with the delivery of their care. Although there are professional measures of treatment success, such as the Peer Assessment Rating,1 at present there are no accepted measures of patient satisfaction, with either the process or outcome of treatment within the United Kingdom. Measures have been developed in Continental Europe and the United States, but because of differences in funding and patients' attitudes, these may not be valid within the United Kingdom.2–5

Assessments of the quality of health care can be based on patient, parental, and professional perspectives, or a combination of the three. Although there is evidence that adult6 and adolescent7 patients can provide reliable judgments on their health care experience, there is some concern that parental reports, although valid, should not substitute for child reports. This is because such reports may be affected by parental anxiety, previous experience, and levels of education.8

A number of factors may influence patient satisfaction, including physical comfort, emotional support, patients' expectations, and respect for patient preference.6910 Previous measures of patient satisfaction with dental treatment have been broadly based on clinicians' perceptions, which may lack validity and/or reliability.11 It is now recognized that to be of value, patient satisfaction measures should be based on the issues that are important to patients. Most studies have examined issues that concern parents, rather than patients, and have used measures developed by professionals without patient input. Only four studies351213 have focused on patients' opinions. Although some have examined satisfaction with both process and outcome,12 others have examined each separately.3514 The only qualitative study previously undertaken to examine satisfaction with treatment process showed that all patients expressed some dissatisfaction, even though they were satisfied with their outcomes. Thus, treatment process and outcome may not be correlated, and there may be important differences between patients', parents', and orthodontists' perceptions of process.12

In the development of a questionnaire to measure patient perceptions and satisfaction with the process of care, there are three stages.15 The items for investigation are first generated through qualitative research centered on focus groups. Second, a questionnaire is designed using these items and then evaluated to ensure it does not contain ambiguous or leading questions. Finally, the questionnaire is tested for readability, reliability, validity, and ease of administration.16 Although reliability refers to the reproducibility, consistency, and homogeneity and the degree to which it is free from random error,1216 validity refers to the degree that a measure is an accurate reflection of what it is designed to measure.17

A patient-centered questionnaire to assess satisfaction with the process of orthodontic healthcare delivery in the United Kingdom had previously been developed using focus groups18 and had good face and content validity. However, it was not tested for readability, ease of administration, reliability, criterion validity (how well the criteria match objective measures), or construct validity (the degree to which a test measures a theoretical construct for which there are no predefined measures). The investigation reported here aimed to determine these properties.

MATERIALS AND METHODS

Ethical committee approval was obtained from the Plymouth (local research ethics committee [LREC] 1960), Solihull (LREC Q15/03), and United Bristol Healthcare Trust (LREC E5604) LRECs. In an earlier qualitative study,18 issues of importance to adolescent orthodontic patients were identified. Subsequently, a questionnaire was developed to examine how children and adolescents perceive the delivery of orthodontic care. The sections of the questionnaire included the following:

• Age, gender, type of appliance, and length of treatment (sections A and B)

• Reasons for treatment (section C)

• Visiting the orthodontist (section D)

• Having treatment (section E)

• Information on braces before treatment (section F)

• Wearing a brace (section G)

• Problems with wearing a brace (section H)

RESULTS

Forty-nine patients undergoing active orthodontic treatment agreed to take part in the initial part of this study (T1), 20 males and 29 females (mean age = 13.9 years; age range = 9 to 17 years). Forty patients were wearing fixed appliances and nine removable appliances. Thirty-one patients responded to the same questionnaire posted to them 12 to 14 weeks later (T2), giving a response rate of 63.3%.

Test-retest Reliability Results

The Kappa values are summarized in Table 1 and show excellent or moderate agreement for all sections, except section C. The specific questions where reliability was moderate were as follows:

• C1 Whose idea was it for you to see the orthodontist about a brace?

• C2 What would you like your brace to do for you?

Table 1.  Median (and Range) Value of Kappa for Items Within Each Section of the Questionnaire

View Fullscreen

Readability Results

The questionnaire had a Flesch Reading Score of 79.8, which is equivalent to a reading age of 10 years. When converted to a Flesch-Kincaid Grade Level score (4.8), it showed good levels of readability, being scored at a US grade 5, or a 10-year-old reading ability (Table 2). The average reading age in the United Kingdom is 12 years, and the questionnaire, therefore, should have been readable for the majority of the study population, which ranged in age from 9 to 17 years (mean age = 13.9 years).

Table 2.  Reading Ease and Grade-level Scores for Questionnaire Format

View Fullscreen

Criterion Validity Results

Seventeen patients took part in this section of the study. Each interview took between 5 and 10 minutes. Table 3 shows the extent of agreement between the responses of the telephone interview and the questionnaire (T1). Agreement was observed to be poor with 50% of the items tested. The median value of kappa for the criterion question was only 0.27, but the range was 0.06 to 0.77.

Table 3.  Value of Kappa Between T1 and the Telephone Interview in Criterion Validity Study

View Fullscreen

Construct Validity Results

Levels of agreement are shown in Table 4. Of the items used to assess construct validity, one item was in the moderate range of agreement and five were in the excellent range of agreement.

Table 4.  Value of Kappa or Spearman's Correlation between T1 and Medical Records in Construct Validity Study

View Fullscreen

DISCUSSION

The questionnaire was assessed for reliability, readability, ease of administration, and validity. Forty-nine patients completed the questionnaire at T1 and all were included in the construct validity study. However, a subgroup of only 17 patients was included in the criterion validity study because of recruitment difficulties. The questionnaire was posted to all of the original participants in order to undertake the test-retest reliability study (T2), and 31 responses were received within six weeks. Although the response rate of 63.3% might seem low, anything between 40% and 80% is not necessarily unacceptable,20 and certainly anything over 50% has been considered by some to be very good.21 Nonresponse is obviously important as it affects the quality of the data collected and the precision of any inferences. For example, response bias may be introduced if certain groups are less likely to respond.16 Methods suggested to improve response rates have included making the questionnaire short; using a personalized covering letter22; emphasizing confidentiality; allowing participants to withdraw at any time; and using colored ink,23 a bold logo, and a clear layout.24 Most of these techiques were used in the present study.

The design of the questionnaire included formatting the questions and responses not only for ease of analysis but also for readability. The latter was found to be at a level suitable for most patients undergoing orthodontic treatment. The wording of each question was structured so as to avoid jargon and therefore misinterpretation.22 The response format used was mainly closed throughout, although open responses were included and subsequently coded as being present or absent to allow the researcher to interpret them independently.

The ease of administration demonstrated that the questionnaire could be completed within 5 to 15 minutes; ideally, any questionnaire should be as succinct as possible to improve the response rate and ensure that the respondent is capable of answering the important research or audit question. The response rate in this case was reasonably good, which suggests that 10 to 15 minutes is a realistic length of time for this type of satisfaction questionnaire and in this age group.

The results also demonstrated that all sections of the questionnaire had satisfactory test-retest reliability, with the exception of section C. Here the two questions were “Whose idea was it for you to see the orthodontist?” and “What would you like your brace to do for you?” It is well known that many children referred for orthodontic treatment are unaware of the precise reason for their referral.25 The poor test-retest reliability for these two questions may have been due to a greater awareness of orthodontic treatment on the patient's behalf by the time the questionnaire was completed at T2. A time interval shorter than 12 to 14 weeks might have produced different results for at least one of these questions. Despite low levels of reliability, these questions addressed important aspects of the process of orthodontic care, as identified by patients in a previous qualitative study.18 However, the use of these data should be treated with caution when assessing the same respondent over time.

The face and content validity of this questionnaire were assumed to be good, as the questions were a reflection of the experiences discussed in the previous focus groups with child and adolescent patients.18 Although the questionnaire showed excellent construct validity, it showed poor criterion validity. There are three possible reasons for this. First, the measure itself is not valid. Second, there were methodologic differences between the telephone interviews and the written questionnaire. And third, a telephone interview may not be the best method of testing validity. An alternative would have been to assess the questionnaire against a similar measure in the published literature.17 However, there is currently no such gold standard measure available.

Construct validity is the extent to which the instrument tests the hypothesis or theory it is measuring. The scale should correlate with related variables; for example, a patient's report of length of treatment should be correlated with information recorded in their orthodontic notes. As already stated, the construct validity was excellent.

The present investigation illustrates the need to perform a pilot study in order to test parameters such as readability, ease of administration, reliability, and validity before the use of a patient satisfaction questionnaire if erroneous results are not to be produced. This has not always been the case with other questionnaire-based studies. For example, O'Connor3 used a clinician-based free-form questionnaire to assess patient perceptions of the delivery of orthodontic treatment. Depending on their stage of treatment, patients were asked to respond to their fears or apprehensions before treatment, greatest dislikes during treatment, and any recommendations for orthodontists after treatment. No mention of validity or reliability testing of these measures was made. Tung and Kiyak13 found that both patients and their parents expected that the greatest benefit from treatment would be an improvement in self-image and oral function. However, the questionnaire used was merely adapted from those used in other studies. The use of questionnaires in audit and research is important, as there is evidence that improving patient satisfaction may positively affect treatment outcome by influencing compliance and therefore treatment quality.26 Reliable information on patient satisfaction is also essential for quality assurance.27

As with all questionnaires, some caution should be exercised when interpreting the results. In this study the criterion validity was poor in 50% of responses, and it could be argued that the sample size was small. However, this was a pilot study to assess specific scientific parameters of a newly developed patient satisfaction questionnaire. It can be seen from the results that a questionnaire has been produced that is readable by the patient group of 10 years and older, is easy to administer, is reliable, and has good construct validity. It may therefore serve as a useful adjunct to existing clinical measures in future research and audit into patient satisfaction.

CONCLUSIONS

• The newly developed patient satisfaction questionnaire had good levels of readability, ease of administration, reliability, and construct validity.

• Test-retest reliability may be influenced by the patient experience and therefore the time interval between the test and retest.

• The response rate was disappointing, but it highlights the difficulties with recruitment in such studies.

• Criterion validity was poor in some instances, which may be related to sample size or interview method, but there is currently no gold standard available against which to test this.

• The results also illustrate the need to carefully pilot the scientific parameters of any questionnaire before use in order to avoid incorrect inferences being made as a result of any findings.