Introduction and background

Clinical trials are conducted to learn about the effectiveness of a treatment by testing it on humans. These studies are the basis for good decision-making about treatments since, according to evidence-based health care practice, clinicians should be able to apply the findings to their clinical practice after reviewing these studies. However, due to the methodological flaws in some studies of this and other kinds, the results obtained cannot be replicated or, even worse, cannot be extrapolated to reality¹. In dentistry, publications do not necessarily have the clinical relevance the discipline demands, and in many cases, errors may occur in its methodology². Therefore, it is essential to know whether studies have been properly conducted, considering the variables which may affect their results and interpretation¹.

One of the main purposes of orthodontic trials is to assess interventions and reach valid conclusions about recommended treatment modalities³. Nowadays, however, only a small portion of the clinical trials reported in orthodontic literature are randomized clinical trials (RCTs), while the rest are prospective or retrospective nonrandomized clinical trials, with the latter being of a lesser methodological quality and, often, the basis for the decisions of clinicians⁴. Randomized clinical trials provide valid results if the bias is reduced or absent. It has been reported that the absence of a series of factors such as control, randomization, allocation concealment, blinding, and accounting for loss to follow-up in trials can introduce a bias and, thus, invalid results⁵. In view of this, it is important to conduct a continuous analysis of the scientific production in the field of orthodontics, as in other health science disciplines.

Bibliometric studies are tools used to provide quantitative analyses of the scientific production⁶ and also report on current publication trends to expand, compare and improve knowledge⁷. Thus, an analysis of biometric indicators provides objective information about what is being published and the chance of watching the trends followed by each journal. The data obtained and the conclusions reached can help editors improve the management of their journals, by facilitating the decision-making process on things such as article selection. There are instruments designed and validated for analyzing scientific production in dentistry. One of them is the MinCir scale (MS) which evaluates the methodological quality (MQ) of the articles by analyzing research design, sample size and methodology⁸, which is an objective assessment of the quality of the methodology used in the clinical trials published on this area.

Malocclusions are the third most prevalent alterations with the third greatest psychosocial impact⁹. According to Bilgic (2014), almost a third of the population evaluated have a very great need of orthodontic treatment¹⁰. In addition, a study conducted in Chile found that 96.2% of students examined had some kind of malocclusion⁹. Given the global importance of orthodontics issues, we must learn about the reality of this scientific field and form of knowledge dissemination. Therefore, the objective of this study is to describe the MQ of orthodontics clinical studies in six journals with the highest impact indexed by the Institute for Scientific Information (ISI), between January 2012 and December 2016.

Materials and methods

This study evaluated clinical trials published in the six journals with the highest impact factor (IF) between January 2012 and December 2016. All the journals assessed are indexed in the ISI Web of Knowledge, under the Dentistry, Oral Surgery and Medicine category. We selected journals featuring clinical trials published within the period evaluated, written in English and which included the words “orthodontic” and “orthopedics” in their titles. The journals selected were: American Journal of Orthodontics and Dentofacial Orthopedics (AJODO) IF=2.201, Angle Orthodontics (AO) IF=1.717, Orthodontics Craniofacial Research (OCFR) IF=1.617, European Journal of Orthodontics (EJO) IF=1.627, Journal of Orofacial Orthopedics/Fortschritte der Kieferorthopädie (JOOF) IF=1.073 and Australian Orthodontic Journal (AOJ) IF=0.452.

In order to select the articles in each journal, an advanced search was conducted in the Medline database, using the NLM Catalog: Journals referenced in the NCBI Databases. The following search filters were used: 1) Journal, 2) Article types: Clinical Study, Clinical Trial, Clinical Trial, Phase I, Clinical Trial, Phase II, Clinical Trial, Phase III, Clinical Trial, Phase IV, Multicenter Study and 3) Publication dates: 01 Jan 2012 to 31 Dec 2016.We reviewed the abstract and methodology of all the studies found. We classified as clinical trials those that evaluated the effectiveness of a treatment, comparing it to two or more interventions and using a control group. All articles that did not meet these criteria, even if they had been indexed in this manner in the NCBI database, were excluded.

All the studies selected were analyzed using an adaptation of the MS scale² (Table 1), following the instructions set out by the authors¹¹. Of the three domains that comprise the scale, only two were considered for this work: the first one, which evaluates the type of study design and the third one, which consists of four items that assess the methodology used in the study². The Domain 2: Studied population by justification factor was not included. Thus, the final score varies between 10 and 24 points, minimum and maximum respectively.

The methodological quality evaluation of each article was conducted independently by each reviewer. The inter-reviewer kappa value obtained during the calibration process was 91.90%. It was calculated applying the MS evaluation instrument independently to 11 clinical trials selected randomly from the same journals selected for this work. Those articles were not included in this analysis. The articles analyzed were classified according to their geographical origin (continent), subject matter (corrective orthodontics, interceptive orthodontics, orthopedics, preventive orthodontics, orthodontic-surgical and others) and the journals included. The data were tabulated in a Google Docs electronic form (Mountain View, CA, USA). The statistical analysis was conducted using the R program, version 3.3.4, which showed significant differences in scores according to geographical origin. Levene’s test was used to check the homogeneity of score variances and, since p value > 0.05, homogeneity of variances is assumed. The Shapiro-Wilk test was then used to check the normality of the data, and p values < 0.05 were obtained, therefore, the Kruskal-Wallis test for non-parametric samples was used. The post hoc pairwise comparison with pooled SD was then used to look at the differences between continents.

Results

Study selection. A total of 271 articles were found: 89 articles which were defined as clinical trials in the AJODO journal, 49 in the EJO journal, 70 in the AO journal, 38 in the JOOF journal, 10 in the AOJ journal, 16 in the OCFR journal, and 50 articles were excluded because they did not meet the selection criteria to be classified as clinical trials. All of them were obtained as full text (Fig. 1).

Characteristics of the journals. The journal with the largest number of clinical trials published is AJODO, with 83 articles indexed under this category, whereas the journal with the fewest clinical trials published is AOJ, with only 7 articles (Fig. 2).

As for the geographical origin, the continent with the largest number of clinical trials published was Europe, with 100 articles (45.25%), and the one with the fewest articles published was Africa, with 5 articles (2.26%). The MQ assessment shows the same distribution by continent. Europe has the highest MQ average with 14.88 points in the MS, while the continent that had the lowest MQ score was Africa, with an average of 11.8 points (p = 0.003) (Table 2) (Fig. 3).

Fig. 3, methodological quality score, according to geographical distribution. The cut-off point is 60% (14.4 points) in the MinCir scale, represented in the chart with the red dashed line.

None of the articles analyzed obtained the maximum or the minimum score in the MS. The minimum score reached was 11 points, and the maximum was 23 points, which was an article in the EJO journal.

As for individual journals, EJO had the highest average score (15.6 ± 2.83), and JOOF had the lowest average score (13.09 ± 1.72) (Table 3).

The first domain of the MS considers the type of clinical trial conducted, of which the clinical trial (simple or without masking; not randomized) was the most common one in orthodontics, with 189 articles (85.52%); and multicenter clinical trial was the least frequent type with 15 articles (6.79%) (Table 4).

As for design analysis, only 61.54% mentioned the design, while the design used was mentioned and justified in 1.36% of cases. In the remaining percentage, the design used was neither mentioned nor justified.

Regarding the selection criteria item, inclusion or exclusion criteria are described in 52.38% of studies, and they are neither described nor mentioned in 12.86% of articles (Table 3).

When analyzing the objectives set out in the articles using the MinCir scale we found that vague objectives were set out in 96.77% of the clinical studies published in the AO journal. On the other hand, 100% of the works of the of the same kind published in the JOOF, AOJ and OCFR journals reached a score of 2 in this item, that is, their objectives were set out vaguely (Table 5).

Finally, in the sample size item, the EJO journal stood out because the sample used was justified in 63.89% of the articles (Table 6).

Discussion

The subject matter was defined as the area of knowledge related to orthodontics the paper was dedicated to. Articles were then grouped as follows: preventive, corrective, interceptive orthodontics, orthopedics, surgical orthodontics and other topics, only for the purpose of organizing the information.

The instrument used was the MS, which has been shown to have adequate psychometric properties, which makes it a valid option for assessing the MQ in dental therapy research articles⁸. In this study we used an adaptation of the MS and assessed two of the three domains of the scale. Domain 2, which looks at the sample size, was excluded because it assigns a high score to studies with a larger number of participants, and a lower score to those with a smaller number of participants, although we did not prove the existence of a proportional relation between the number of subjects included in a sample and the quality of the study. Thus, according to this domain of the MS, a double-blind, randomized, multicenter clinical trial comparing two interventions can require 100 patients (6 points) or less to obtain valid results, while a study of a series of cases can include 500 patients (12 points) and still produce low-quality results. Cartes et al. propose, as a solution to this issue, giving more weight to the justification of the sample size, than to the number itself¹³.

A geographical analysis of the scientific production has shown that the best indicators for medical and dental research can be found in Europe, North America, Australia and Oceania, versus Asia, Africa and South America¹². Nevertheless, our results showed that research in orthodontics is more developed in Europe and Asia, since these two continents stand out both regarding the number of studies published and their quality.

The MS adaptation provided a score ranging from 10 to 24 points, and the average MS score for the clinical trials in the six journals analyzed was 14.33. The cut-off point used for determining the MS was 14.4 points. A score above 14.4 is considered a good MQ, therefore, Europe and Asia were the continents that stood out the most with the largest number of studies over the cut-off point.

It is also worth noting that the IF for each journal is not directly related to the average MQ of its clinical trials, as the AJODO journal has the highest IF of 2.201, but did not reach the highest score, with only 14.67 points. This can be interpreted in many ways, the first of which is that perhaps the authors who are citing the papers in that journal are not aware of the fact that the methodological designs used to conduct the clinical trials are not optimal and/or they prioritize other aspects that they believe to be more important at the time of citing research. It could also be concluded that authors are aware of the poor methodological quality of these trials and are citing them to emphasize the mistakes in those publications, or that the authors know that the methodological quality of the clinical trials is deficient, but they are citing other kinds of studies published in those journals whose methodological quality is good. In view of this, we suggest conducting systematic reviews of the different therapies used currently in orthodontics.

Another interesting finding in this analysis is that in the MS objectives, those that were presented were, mostly, vague (96.83%). This is explained by the fact that the MS requires for an objective to be considered clear and specific to include what will be measured, by what means, by whom, where and when the measurement will take place. In the case of a therapeutic procedure or intervention, the objective must state what it is, whether it has a comparator, who applies it, to whom it is applied, when and how it is applied¹¹. It is also worth noting that if any of the criteria detailed above are not met, the objective is classified as vague. Only one article complied with all of these items, the others usually only mention what will be measured, when it will be measured, the therapeutic intervention and the comparison.

Among the different study designs, the RCT is considered to potentially provide the best quality evidence, but there is substantial evidence in the biomedical literature that the quality of the RCTs published is suboptimal¹. In this study we found mostly (85.52%) RCTs in which the study design was not explicitly mentioned, leading the reader to infer this information, which can cause interpretation problems with less experienced readers. In addition, multicenter studies were the least, which could be due to the complexity of their execution: taking into account all the parameters that should be considered for the procedure standardization, planning, execution and funding².

Masking is considered one of the most important procedures to obtain impartial results. We must also remember that it is not always possible to blind participants and/or reviewers; it is often possible to blind the reviewers of results and data analysts¹. This is what happened in most cases.

According to the parameters set by evidence-based dentistry, clinical decision-making must be based on the best and most current evidence available, yet there are few studies focused on evaluating this evidence, which leaves unexperienced readers with the dilemma of whether to accept or not the data published. Therefore, we recommend conducting further studies to assess other areas of dental sciences, to provide data that will guide the decision-making process of readers to determine how reliable and applicable to other cases the results found in a scientific paper can be.

Conclusions

Implications for clinical practice. After analyzing the methodological quality of the clinical trials in orthodontics indexed in ISI journals between 2012 and 2016, and concluding that it presented some deficiencies when the MinCir scale is applied, we suggest this information should be considered to analyze its implications for orthodontics practice.

Implications for research. We suggest researchers conduct controlled trials taking into account the methodological quality, including, for example, masked examiners to reduce biases.

Funding. Study funded by the School of Dentistry, School of Medicine, Universidad Austral de Chile.

Acknowledgments. To the School of Dentistry, School of Medicine, Universidad Austral de Chile, for its financial cooperation to this research