SciELO - Scientific Electronic Library Online

 
vol.20 número31Características morfométricas de los dientes multirradiculares a nivel de la zona de furcaciónPrevalencia de signos y síntomas de trastornos témporomandibulares en una población joven al inicio del tratamiento por drogodependencia índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Indicadores

Links relacionados

  • No hay articulos similaresSimilares en SciELO

Compartir


Odontoestomatología

versión impresa ISSN 0797-0374versión On-line ISSN 1688-9339

Odontoestomatología vol.20 no.31 Montevideo jun. 2018

http://dx.doi.org/10.22592/ode2018n31a4 

Research

Methodological quality of orthodontic clinical trials, 2012-2016

Camila Ruiz Chaura (orcid: 0000-0001-9618-5182)1  , Nicole Ulloa Riffo (orcid: 0000-0003-0264-7425)2  , Diego Halabí Robles (orcid: 0000-0002-1474-8066)3  , Helmuth Daniel Muñoz Martínez (orcid: 0000-0002-5650-0325)4 

1 School of Dentistry. Universidad Austral de Chile, Valdivia, Chile

2 School of Dentistry. Universidad Austral de Chile, Valdivia, Chile

3 Institute of Odonto-Stomatology, Universidad Austral de Chile, Valdivia, Chile.

4 Institute of Odonto-Stomatology, Universidad Austral de Chile, Valdivia, Chile

Resumen:

Objective:

To determine the methodological quality of orthodontics clinical studies in six journals with the highest impact indexed by the Institute for Scientific Information (ISI), between January 2012 and December 2016.

Methodology:

We performed a systematic search in PubMed and we reviewed the abstract and methodology of all the selected studies. We classified as clinical trials the studies that evaluated the effectiveness of a treatment, comparing two or more interventions to a control group. Results: We reviewed 221 clinical trials with the MinCir scale. Regarding methodological quality average, the European Journal of Orthodontics showed the best average score (15.6 ± 2.83). Regarding continent results, Europe has the highest average score of methodological quality with 14.88 points.

Conclusion:

The methodological quality of the clinical trials in orthodontics indexed in ISI journals presented some deficiencies. Therefore, we suggest this information should be considered to analyze its implications for orthodontics practice.

Keywords clinical trial; orthodontics; malocclusion; dentistry; publication bias

Introduction and background

Clinical trials are conducted to learn about the effectiveness of a treatment by testing it on humans. These studies are the basis for good decision-making about treatments since, according to evidence-based health care practice, clinicians should be able to apply the findings to their clinical practice after reviewing these studies. However, due to the methodological flaws in some studies of this and other kinds, the results obtained cannot be replicated or, even worse, cannot be extrapolated to reality1. In dentistry, publications do not necessarily have the clinical relevance the discipline demands, and in many cases, errors may occur in its methodology2. Therefore, it is essential to know whether studies have been properly conducted, considering the variables which may affect their results and interpretation1.

One of the main purposes of orthodontic trials is to assess interventions and reach valid conclusions about recommended treatment modalities3. Nowadays, however, only a small portion of the clinical trials reported in orthodontic literature are randomized clinical trials (RCTs), while the rest are prospective or retrospective nonrandomized clinical trials, with the latter being of a lesser methodological quality and, often, the basis for the decisions of clinicians4. Randomized clinical trials provide valid results if the bias is reduced or absent. It has been reported that the absence of a series of factors such as control, randomization, allocation concealment, blinding, and accounting for loss to follow-up in trials can introduce a bias and, thus, invalid results5. In view of this, it is important to conduct a continuous analysis of the scientific production in the field of orthodontics, as in other health science disciplines.

Bibliometric studies are tools used to provide quantitative analyses of the scientific production6 and also report on current publication trends to expand, compare and improve knowledge7. Thus, an analysis of biometric indicators provides objective information about what is being published and the chance of watching the trends followed by each journal. The data obtained and the conclusions reached can help editors improve the management of their journals, by facilitating the decision-making process on things such as article selection. There are instruments designed and validated for analyzing scientific production in dentistry. One of them is the MinCir scale (MS) which evaluates the methodological quality (MQ) of the articles by analyzing research design, sample size and methodology8, which is an objective assessment of the quality of the methodology used in the clinical trials published on this area.

Malocclusions are the third most prevalent alterations with the third greatest psychosocial impact9. According to Bilgic (2014), almost a third of the population evaluated have a very great need of orthodontic treatment10. In addition, a study conducted in Chile found that 96.2% of students examined had some kind of malocclusion9. Given the global importance of orthodontics issues, we must learn about the reality of this scientific field and form of knowledge dissemination. Therefore, the objective of this study is to describe the MQ of orthodontics clinical studies in six journals with the highest impact indexed by the Institute for Scientific Information (ISI), between January 2012 and December 2016.

Materials and methods

This study evaluated clinical trials published in the six journals with the highest impact factor (IF) between January 2012 and December 2016. All the journals assessed are indexed in the ISI Web of Knowledge, under the Dentistry, Oral Surgery and Medicine category. We selected journals featuring clinical trials published within the period evaluated, written in English and which included the words “orthodontic” and “orthopedics” in their titles. The journals selected were: American Journal of Orthodontics and Dentofacial Orthopedics (AJODO) IF=2.201, Angle Orthodontics (AO) IF=1.717, Orthodontics Craniofacial Research (OCFR) IF=1.617, European Journal of Orthodontics (EJO) IF=1.627, Journal of Orofacial Orthopedics/Fortschritte der Kieferorthopädie (JOOF) IF=1.073 and Australian Orthodontic Journal (AOJ) IF=0.452.

In order to select the articles in each journal, an advanced search was conducted in the Medline database, using the NLM Catalog: Journals referenced in the NCBI Databases. The following search filters were used: 1) Journal, 2) Article types: Clinical Study, Clinical Trial, Clinical Trial, Phase I, Clinical Trial, Phase II, Clinical Trial, Phase III, Clinical Trial, Phase IV, Multicenter Study and 3) Publication dates: 01 Jan 2012 to 31 Dec 2016.We reviewed the abstract and methodology of all the studies found. We classified as clinical trials those that evaluated the effectiveness of a treatment, comparing it to two or more interventions and using a control group. All articles that did not meet these criteria, even if they had been indexed in this manner in the NCBI database, were excluded.

All the studies selected were analyzed using an adaptation of the MS scale2 (Table 1), following the instructions set out by the authors11. Of the three domains that comprise the scale, only two were considered for this work: the first one, which evaluates the type of study design and the third one, which consists of four items that assess the methodology used in the study2. The Domain 2: Studied population by justification factor was not included. Thus, the final score varies between 10 and 24 points, minimum and maximum respectively.

The methodological quality evaluation of each article was conducted independently by each reviewer. The inter-reviewer kappa value obtained during the calibration process was 91.90%. It was calculated applying the MS evaluation instrument independently to 11 clinical trials selected randomly from the same journals selected for this work. Those articles were not included in this analysis. The articles analyzed were classified according to their geographical origin (continent), subject matter (corrective orthodontics, interceptive orthodontics, orthopedics, preventive orthodontics, orthodontic-surgical and others) and the journals included. The data were tabulated in a Google Docs electronic form (Mountain View, CA, USA). The statistical analysis was conducted using the R program, version 3.3.4, which showed significant differences in scores according to geographical origin. Levene’s test was used to check the homogeneity of score variances and, since p value > 0.05, homogeneity of variances is assumed. The Shapiro-Wilk test was then used to check the normality of the data, and p values < 0.05 were obtained, therefore, the Kruskal-Wallis test for non-parametric samples was used. The post hoc pairwise comparison with pooled SD was then used to look at the differences between continents.

Table 1:  MinCir Methodological Quality Scale, modified for clinical trials. 

Assigned Score
Domain 1. Study Design
Multicenter clinical trial 12
Double-masked, randomized controlled clinical trial 9
Clinical trial (simple or without masking; not randomized) 6
Domain 3. Methodology used
Item 1. Objectives
The objectives are clearly and specifically presented 3
The objectives are vaguely presented 2
The objectives are not presented 1
Item 2. Design
The design used is mentioned and justified 3
The design used is mentioned 2
The design used is neither mentioned nor justified 1
Item 3. Sample selection criteria
Inclusion and exclusion criteria are described 3
Inclusion or exclusion criteria are described 2
Inclusion or exclusion criteria are not described 1
Item 4. Sample size
Sample used is justified 3
It does not justify the sample used 1

Results

Study selection. A total of 271 articles were found: 89 articles which were defined as clinical trials in the AJODO journal, 49 in the EJO journal, 70 in the AO journal, 38 in the JOOF journal, 10 in the AOJ journal, 16 in the OCFR journal, and 50 articles were excluded because they did not meet the selection criteria to be classified as clinical trials. All of them were obtained as full text (Fig. 1).

Fig. 1: Flow chart of the number of articles obtained. 

Characteristics of the journals. The journal with the largest number of clinical trials published is AJODO, with 83 articles indexed under this category, whereas the journal with the fewest clinical trials published is AOJ, with only 7 articles (Fig. 2).

Fig. 2:  Number of clinical trials published per journal from January 2012 to December 2016 selected. 

As for the geographical origin, the continent with the largest number of clinical trials published was Europe, with 100 articles (45.25%), and the one with the fewest articles published was Africa, with 5 articles (2.26%). The MQ assessment shows the same distribution by continent. Europe has the highest MQ average with 14.88 points in the MS, while the continent that had the lowest MQ score was Africa, with an average of 11.8 points (p = 0.003) (Table 2) (Fig. 3).

Table 2:  Number of clinical trials published per continent and mean Methodological Quality score per continent. 

Continent n (%) Score SD
Europe 100 45.25 14.88a 2.8
Asia 64 28.96 14.31b 2.22
Oceania 7 3.17 14c 2
North America 20 9.05 13.55d 1.47
Latin America 25 11.31 13.44e 2.24
Africa 5 2.26 11.8f 0.84
TOTAL 221 100

Kruskal-Wallis test post-hoc Pairwise comparison with pooled SD (p = 0.003), f < a and b; c > d and e

Fig. 3:  methodological quality score, according to geographical distribution. The cut-off point is 60% (14.4 points) in the MinCir scale, represented in the chart with the red dashed line 

Fig. 3, methodological quality score, according to geographical distribution. The cut-off point is 60% (14.4 points) in the MinCir scale, represented in the chart with the red dashed line.

None of the articles analyzed obtained the maximum or the minimum score in the MS. The minimum score reached was 11 points, and the maximum was 23 points, which was an article in the EJO journal.

As for individual journals, EJO had the highest average score (15.6 ± 2.83), and JOOF had the lowest average score (13.09 ± 1.72) (Table 3).

Table 3: Clinical trials published per journal and mean Methodological Quality score per journal. 

Journal N % Score SD 5-y IF
European Journal of Orthodontics 36 16.29 15.6 2.83 1.627
American Journal of Orthodontics and Dentofacial Orthopedics 83 37.56 14.67 2.7 2.201
Orthodontics Craniofacial Research 11 4.98 14.27 2.45 1.617
Australian Orthodontic Journal 7 3.17 14 2.16 0.452
Angle Orthodontics 62 28.05 13.88 2.05 1.717
Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie 22 9.95 13.09 1.72 1.073
Total 221 100

5-y IF: 5-year impact factor, SD: standard deviation, Cut-off point: 14.4 points

The first domain of the MS considers the type of clinical trial conducted, of which the clinical trial (simple or without masking; not randomized) was the most common one in orthodontics, with 189 articles (85.52%); and multicenter clinical trial was the least frequent type with 15 articles (6.79%) (Table 4).

Table 4:  Number of articles classified per type of clinical trial, design and selection criterion. 

n (%)
Clinical trial Multicenter clinical trial 15 6.79
Double-masked, randomized controlled clinical trial 17 7.69
Clinical trial (simple or without masking; not randomized) 189 85.52
Design The design used is mentioned and justified 3 1.36
The design used is mentioned 136 61.54
The design used is neither mentioned nor justified 82 37.1
Selection Inclusion and exclusion criteria are described 79 35.75
criteria Inclusion or exclusion criteria are described 113 51.13
Inclusion or exclusion criteria are not described 29 13.12

As for design analysis, only 61.54% mentioned the design, while the design used was mentioned and justified in 1.36% of cases. In the remaining percentage, the design used was neither mentioned nor justified.

Regarding the selection criteria item, inclusion or exclusion criteria are described in 52.38% of studies, and they are neither described nor mentioned in 12.86% of articles (Table 3).

When analyzing the objectives set out in the articles using the MinCir scale we found that vague objectives were set out in 96.77% of the clinical studies published in the AO journal. On the other hand, 100% of the works of the of the same kind published in the JOOF, AOJ and OCFR journals reached a score of 2 in this item, that is, their objectives were set out vaguely (Table 5).

Table 5:  Number of clinical trials published per journal per presentation of the study objective. 

Journals The objectives are clearly and specifically presented The objectives are vaguely presented The objectives are not presented Total
n % N % n % n %
American Journal of Orthodontics and Dentofacial Orthopedics 0 0 80 96.39 3 3.61 83 37.56
European Journal of Orthodontics 1 2.78 34 94.44 1 2.78 36 16.29
Angle Orthodontics 0 0 60 96.77 2 3.23 62 28.05
Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie 0 0 22 100 0 0 22 9.95
Australian Orthodontic Journal 0 0 7 100 0 0 7 3.17
Orthodontics Craniofacial Research 0 0 11 100 0 0 11 4.98
Total 1 0.45 214 96.83 6 2.71 221 100

Finally, in the sample size item, the EJO journal stood out because the sample used was justified in 63.89% of the articles (Table 6).

Table 6: Number of clinical trials published per journal per justification of the sample size. 

Journals Sample is justified Sample is not justified Total
N % n % n %
American Journal of Orthodontics and Dentofacial Orthopedics 38 45.78 45 54.22 83 37.56
European Journal of Orthodontics 23 63.89 13 36.11 36 16.29
Angle Orthodontics 24 38.71 38 61.29 62 28.05
Journal of Orofacial Orthopedics / Fortschritte der Kieferorthopädie 3 13.64 19 86.36 22 9.95
Australian Orthodontic Journal 2 28.57 5 71.43 7 3.17
Orthodontics Craniofacial Research 4 36.36 7 63.64 11 4.98
Total 94 42,53 127 57,47 221 100

Discussion

The subject matter was defined as the area of knowledge related to orthodontics the paper was dedicated to. Articles were then grouped as follows: preventive, corrective, interceptive orthodontics, orthopedics, surgical orthodontics and other topics, only for the purpose of organizing the information.

The instrument used was the MS, which has been shown to have adequate psychometric properties, which makes it a valid option for assessing the MQ in dental therapy research articles8. In this study we used an adaptation of the MS and assessed two of the three domains of the scale. Domain 2, which looks at the sample size, was excluded because it assigns a high score to studies with a larger number of participants, and a lower score to those with a smaller number of participants, although we did not prove the existence of a proportional relation between the number of subjects included in a sample and the quality of the study. Thus, according to this domain of the MS, a double-blind, randomized, multicenter clinical trial comparing two interventions can require 100 patients (6 points) or less to obtain valid results, while a study of a series of cases can include 500 patients (12 points) and still produce low-quality results. Cartes et al. propose, as a solution to this issue, giving more weight to the justification of the sample size, than to the number itself13.

A geographical analysis of the scientific production has shown that the best indicators for medical and dental research can be found in Europe, North America, Australia and Oceania, versus Asia, Africa and South America12. Nevertheless, our results showed that research in orthodontics is more developed in Europe and Asia, since these two continents stand out both regarding the number of studies published and their quality.

The MS adaptation provided a score ranging from 10 to 24 points, and the average MS score for the clinical trials in the six journals analyzed was 14.33. The cut-off point used for determining the MS was 14.4 points. A score above 14.4 is considered a good MQ, therefore, Europe and Asia were the continents that stood out the most with the largest number of studies over the cut-off point.

It is also worth noting that the IF for each journal is not directly related to the average MQ of its clinical trials, as the AJODO journal has the highest IF of 2.201, but did not reach the highest score, with only 14.67 points. This can be interpreted in many ways, the first of which is that perhaps the authors who are citing the papers in that journal are not aware of the fact that the methodological designs used to conduct the clinical trials are not optimal and/or they prioritize other aspects that they believe to be more important at the time of citing research. It could also be concluded that authors are aware of the poor methodological quality of these trials and are citing them to emphasize the mistakes in those publications, or that the authors know that the methodological quality of the clinical trials is deficient, but they are citing other kinds of studies published in those journals whose methodological quality is good. In view of this, we suggest conducting systematic reviews of the different therapies used currently in orthodontics.

Another interesting finding in this analysis is that in the MS objectives, those that were presented were, mostly, vague (96.83%). This is explained by the fact that the MS requires for an objective to be considered clear and specific to include what will be measured, by what means, by whom, where and when the measurement will take place. In the case of a therapeutic procedure or intervention, the objective must state what it is, whether it has a comparator, who applies it, to whom it is applied, when and how it is applied11. It is also worth noting that if any of the criteria detailed above are not met, the objective is classified as vague. Only one article complied with all of these items, the others usually only mention what will be measured, when it will be measured, the therapeutic intervention and the comparison.

Among the different study designs, the RCT is considered to potentially provide the best quality evidence, but there is substantial evidence in the biomedical literature that the quality of the RCTs published is suboptimal1. In this study we found mostly (85.52%) RCTs in which the study design was not explicitly mentioned, leading the reader to infer this information, which can cause interpretation problems with less experienced readers. In addition, multicenter studies were the least, which could be due to the complexity of their execution: taking into account all the parameters that should be considered for the procedure standardization, planning, execution and funding2.

Masking is considered one of the most important procedures to obtain impartial results. We must also remember that it is not always possible to blind participants and/or reviewers; it is often possible to blind the reviewers of results and data analysts1. This is what happened in most cases.

According to the parameters set by evidence-based dentistry, clinical decision-making must be based on the best and most current evidence available, yet there are few studies focused on evaluating this evidence, which leaves unexperienced readers with the dilemma of whether to accept or not the data published. Therefore, we recommend conducting further studies to assess other areas of dental sciences, to provide data that will guide the decision-making process of readers to determine how reliable and applicable to other cases the results found in a scientific paper can be.

Conclusions

Implications for clinical practice. After analyzing the methodological quality of the clinical trials in orthodontics indexed in ISI journals between 2012 and 2016, and concluding that it presented some deficiencies when the MinCir scale is applied, we suggest this information should be considered to analyze its implications for orthodontics practice.

Implications for research. We suggest researchers conduct controlled trials taking into account the methodological quality, including, for example, masked examiners to reduce biases.

Funding. Study funded by the School of Dentistry, School of Medicine, Universidad Austral de Chile.

Acknowledgments. To the School of Dentistry, School of Medicine, Universidad Austral de Chile, for its financial cooperation to this research

Referencias

1 Pandis N, Polychronopoulou A, Eliades T. An assessment of quality characteristics of randomised control trials published in dental journals. J Dent 2010. 38(9):713-21. [ Links ]

2 Morales K, Montoya N, Pradenas I, Muñoz H. Methodological quality of periodontal clinical trials published in isi journals.Odontoestomatología, 2016. 18(27). [ Links ]

3 Baumgartnera S, Pandis N, Eliades T. Exploring the publications in three major orthodontic journals A comparative analysis of two 5-year periods. Angle Orthodont 2014. 84(3):397-403. [ Links ]

4 Papageorgiou S, Xavierd G, Cobourned M. Basic study design influences the results of orthodontic clinical investigations. J Clin Epidemiol 2015;68(12):1512-22. [ Links ]

5 Pandis N. Randomized clinical trials (RCTs) and systematic reviews (SRs) in the context of evidence-based orthodontics (EBO). Seminar Orthodont 2013. 19(3): 142-157. [ Links ]

6 Patrón C, López J, Piovesan S, Demaría B. Bibliometric analysis of the scientific production in Odontoestomatología journal. Odontoestomatología 2014; 16(23). [ Links ]

7 Restrepo- Valencia L, Cano A, Castañeda C, Sánchez R, González-Ariza SE. Análisis de la producción científica de la revista ces odontología en los últimos 10 años. Rev. CES Odont 2015; 28(2): 119-131. [ Links ]

8 Cartes-Velásquez R, Manterola C, Aravena P, Moraga J. Reliability and validity of MINCIR scale for methodological quality in dental therapy research. Braz Oral Res 2014;28(1):1-5. [ Links ]

9 Burgos D. Prevalencia de maloclusiones en niños y adolescentes de 6 a 15 años en Frutillar, Chile. Int. J. Odontostomat, 2014. 8(1):13-19. [ Links ]

10 Bilgic F, Gelgor IE, Celebi AA. Malocclusion prevalence and orthodontic treatment need in central Anatolian adolescents compared to European and other nations’ adolescents. Dental Press J Orthod. 2015 NovDec;20(6):75-81. [ Links ]

11 Moraga J, Manterola C, Cartes-Velasquez R, Burgos ME, Aravena P, Urrutia S, Grupo MINCIR. Instrucciones para la utilización de la escala MINCIR para valorar calidad metodológica de estudios de terapia. Int. J. Morphol, 2014; 32(1):294-298. [ Links ]

12 Manterola C, Vial M, Pineda V, Sanhueza A. Systematic review of literature with different types of designs. Int. J. Morphol, 2009; 27(4):1179-1186. [ Links ]

13 Feldens C, Kramer P, Feldens E. Exploring the profile of articles on traumatic dental injuries in pediatric dental journals. Dental Traumatology 2013; 29: 172-177. [ Links ]

Received: September 26, 2017; Accepted: April 03, 2018

Creative Commons License Este es un artículo publicado en acceso abierto bajo una licencia Creative Commons