Services on Demand
Journal
Article
text in 
English (pdf)
Article in xml format
Article references
Automatic translationRelated links
Share
Permalink
Odontoestomatología
Print version ISSN 0797-0374On-line version ISSN 1688-9339
Odontoestomatología vol.18 no.27 Montevideo May 2016
Methodological quality of periodontal clinical trials published in ISI Journals
Morales, Katherine*, Montoya, Navy*, Pradenas, Ignacio*, Muñoz, Helmuth**
*School of Dentistry. Universidad Austral de Chile.
** Dental Surgeon. Assistant Professor. Odontostomatology Institute. Universidad Austral de Chile.
Helmuth Muñoz: helmuth.munoz@uach.cl
Abstract
Objective: To describe the methodological quality of periodontal clinical trials in journals indexed by the Institute for Scientific Information (ISI).
Materials and methods: Descriptive observational study in which the clinical trials published in four ISI-indexed periodontics journals between January 2008 and December 2012 were evaluated. The journals were selected as they had had the highest impact factor in the last 5 years according to the Journal Citation Reports.
For each journal, papers were searched for in PubMed. Only clinical trials were selected to conduct the qualitative analysis using a methodological quality scale. Statistical analysis was conducted with SPSS Statistics 20 for Windows (IBM, Chicago, USA), and the data were presented through descriptive statistics.
Results: The following journals fulfilled the selection criteria: Journal of Clinical Periodontology, Journal of Periodontology, Journal of Periodontal Research and International Journal of Periodontics & Restorative Dentistry. From these journals, 387 clinical trials were analyzed. Of the maximum of 24 points in the scale, the mean reached was 17.45 points.
Conclusion: The methodological quality of periodontal clinical trials indexed in ISI is deficient.
Keywords: Methodological quality, Clinical trials, Periodontics.
Received on: 03 May 15 Accepted on: 25 Nov 2015
Introduction
In scientific literature, the impact factor is defined as the frequency with which a journal’s article is cited (1). This is considered a quantitative measurement of the quality of the journal (2). However, there is a conflict regarding the correlation between the impact factor of the journal and the quality of its publications (3). This is due to the fact that the number of citations in an article does not necessarily reflect its methodological quality, since an article may be cited not only to highlight its positive characteristics but also to emphasize the negative ones.
As of the emergence of Evidence-Based Dentistry, literature has become an essential tool in the process of making clinical decisions. Despite this, in dentistry, publications do not necessarily have the clinical relevance the discipline demands, and in many cases, errors may occur in its methodology (4). In general, periodontics journals have a high impact factor. However, there are no studies that evaluate the methodological quality of the reports therein published. The evaluation of the methodological quality of an article is controversial. The methods and techniques used are a source of continuous debate, not only by the scientific community but also by editors (5). Therefore, it is important to judge and evaluate the quality of what is reported in scientific articles in order to identify the reliability of the information before its clinical application, as errors in design may alter results.
The aim of this study is to describe the methodological quality of periodontal clinical trials in journals indexed by the Institute for Scientific Information (ISI) published between 2008 and 2012.
Materials and Methods
This is a descriptive observational study in which periodontics journals were searched in the Journal Citation Reports. They were ranked according to the impact factor of the last five years. The four with the highest impact factor indexed to ISI were selected. They included clinical trials, were written in English and included the following words in their headings: periodontics, periodontal, periodontology.
In order to select the articles, an advanced search of each journal was conducted in PubMed, using as a search criterion the name of the journal and the following search filters: journal, article types, clinical trial and publication dates: 01-01-2008 to 31-12-2012.
The materials and methods were analyzed to verify that the studies were clinical trials; in other words, studies carried out with patients, where they underwent a diagnostic test or therapeutic treatment. The articles that did not qualify as clinical trials were excluded even when they had been indexed as clinical trials in PubMed.
Every study was submitted to an analysis of methodological quality through a scoring system designed for the evaluation of clinical trials in surgery and validated to be used in clinical studies in dentistry (6-7) (Table 1). This system contains originally three sections. However, only two were used in this study. The first section was related to the type of study design, and the second was related to the methodology description used in the study (description of the objectives, design justification, eligibility requirements and sample justification). Thus, a final score between 10 and 24 points, minimum and maximum respectively, was allocated.
The methodological quality evaluation of each article was conducted independently by two reviewers and a kappa index >0.8 was achieved in the calibration process. This consisted in applying the methodological quality scale to 20 periodontics clinical trials from journals that were not part of the study.
The articles analyzed were organized according to their geographical origin and the journals included in the evaluation. The mean and the standard deviation of the methodological quality score were also stated.
The data were tabulated in a Google Docs electronic form (Mountain View, CA, USA). Then, they were analyzed using descriptive statistics with SPSS 20 for Windows statistical software (IBM, Chicago, USA).
Results
The four journals with the highest periodontics impact that included clinical trials were: Journal of Periodontology, Journal of Clinical Periodontology, Journal of Periodontal Research and The International Journal of Periodontics and Restorative Dentistry.
After each journal was searched in PubMed, 493 clinical trials were obtained. Their full texts were analyzed and, through the analysis of their materials and methods, it was determined that only 78% (387) of the articles were actually clinical trials (Figure 1).
Figure 1. Flow chart of the number of articles obtained.
The continent with the highest number of published clinical trials was Europe (47.8%) and the lowest was Oceania with only one clinical trial published in the selected journals between 2008 and 2012 (Table 2).
The following image shows the number of clinical trials published per journal from January 2008 to December 2012.
Of the maximum of 24 points in the scale, the mean reached was 17.45 points. Out of the articles analyzed, 3.3% (13) reached the highest score. These were published in the Journal of Periodontology (n=5) and the Journal of Clinical Periodontology (n=8).
Based on the analysis conducted, it was determined that the Journal of Clinical Periodontology had the highest average score (18.06 points). Table 3 specifies the Methodological Quality mean score per journal. These scores are compared in figure 3.
Regarding the sections analyzed with the scale, the ones with the lowest score were:
a. in the study design variable, the multicenter clinical trials were the least frequent, with only 7.2% of the articles analyzed (Table 4);
b. the lack of explicit sample inclusion and exclusion selection criteria regarding sample selection, since almost 30% of the articles mention these criteria incompletely (Table 4);
c. the lack of justification of the sample size, where 47.3% of the articles did not justify the sample used. The journal with the most deficiencies in this area was the Journal of Periodontics and Restorative Dentistry, with 95.3% of its articles having this problem (Table 5).
As for the objectives of the articles analyzed, 92.2% of them clearly and specifically presented their objectives, this figure reaching 97% in the Journal of Periodontology (Table 6).
Discussion
The mean reached by the articles evaluated from the four journals selected through the selection criteria and to which the scale was applied was 17.45, out a maximum of 24 points.
Based on the analysis of the clinical trials between 2008 and 2012, the source continent with the highest publication is Europe (47.8%), which may be due to the fact that the journals studied are European. However, in comparison to the other continents, the mean score in the Methodological Quality Scale is lower than the average score of all the continents. Oceania is a different case, since it has the highest score but only one article published in the journals studied, therefore it cannot be considered representative.
The Journal of Periodontology had the highest number of publications of clinical trials. Still, it did not reach the highest mean out of all the journals. Based on this, the question is: why does a scientific journal publish more clinical trials than another? It may be due to its reputation assigned by its readers, the prestige of its publications, the reputation of its editors or the impact factor, which can be even more important. If a journal has a higher impact factor, it is likely that the authors would like to publish in said journal, disregarding the fact that the design of the articles may be deficient.
The lower frequency of the multicenter studies could be due to the complexity of their execution: taking into account all the parameters that should be considered for the procedure standardization, planning and execution. In addition, the reason for a higher frequency of the unmasked or single-masked clinical studies can be that their execution feasibility is higher than that of the multicenter study.
The lowest score was reached in the sample size variable, with 47% (87) of the studies incorrectly justifying the sample. Not being clear about it can lead to a distortion when the study is replicated or when the results are extrapolated to reality. In fact, if the sample used cannot be justified, the study loses credibility and leads to misinterpretations (8). Authors must understand that the impact of the sample size when interpreting the data of a study is a prerequisite for high-quality, innovative and valid research (9).
As for the sample selection criteria variable, 28.4% (n=110) of the papers only describe inclusion or exclusion criteria. When only one variable is described, the other variable must be interpreted by the reader. An ambiguous definition of the sample selection criteria will not allow us to replicate the study since there is no certainty about the selection of the study population.
In a study conducted by Clovis (2012), which evaluated periodontal clinical trial abstracts using the CONSORT (10) statement, it was concluded that that their quality can be improved (11). This agrees with the results of this evaluation, since according to the abstracts; the methodological quality of the articles is clearly low.
When entering the filter "clinical trial" in PubMed and after thoroughly analyzing the texts obtained, researchers verified that the results obtained from the search engine were not all clinical trials, since many of them did not compare the therapeutic procedures, which means that some articles are being incorrectly indexed.
An important limitation is that in dentistry, so far, there are no similar studies that evaluate the methodological quality of the entire text with which to contrast the results obtained in this study. Therefore, it cannot be established whether there was an improvement in the study methodology or not.
It is questionable whether the great numbers of articles that are currently published are actually useful and relevant to dentistry as a field of knowledge. As mentioned above, there is a larger percentage of methodologically deficient articles, which can lead readers to make mistakes in their decision-making processes. In this context, we could suggest improving the research protocols and the strictness of the editors before approving the publication of an article. In addition, we recommend the creation of a new reliable, validated and standardized evaluation scale to conduct a periodical analysis of the methodological quality of each journal (6).
In the meantime, clinical trial authors should follow the CONSORT statement (10) guidelines, which would allow us to standardize and compare articles in the future. Additionally, reviewers would be asked to promote clearer rules to improve the authors’ compliance with them. We recommend that the editors hire scientific research specialized reviewers, said expertise being qualitative or quantitative, depending on the article to analyze.
For future studies, we suggest that articles such as this be written in order to analyze whether there is an improvement in article quality over time, and also that we study whether the highest impact factor journals are the ones that actually include articles with better methodological quality.
Conclusion
The methodological quality of periodontal clinical trials indexed in ISI journals is deficient.
References
1. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc [en línea] 2003; 91(1): 42–46. Consultado noviembre 2015. Disponible en: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC141186/pdf/i0025-7338-091-01-0042.pdf
Morales, Katherine*, Montoya, Navy*, Pradenas, Ignacio*, Muñoz, Helmuth**
*School of Dentistry. Universidad Austral de Chile.
** Dental Surgeon. Assistant Professor. Odontostomatology Institute. Universidad Austral de Chile.
Helmuth Muñoz: helmuth.munoz@uach.cl
Abstract
Objective: To describe the methodological quality of periodontal clinical trials in journals indexed by the Institute for Scientific Information (ISI).
Materials and methods: Descriptive observational study in which the clinical trials published in four ISI-indexed periodontics journals between January 2008 and December 2012 were evaluated. The journals were selected as they had had the highest impact factor in the last 5 years according to the Journal Citation Reports.
For each journal, papers were searched for in PubMed. Only clinical trials were selected to conduct the qualitative analysis using a methodological quality scale. Statistical analysis was conducted with SPSS Statistics 20 for Windows (IBM, Chicago, USA), and the data were presented through descriptive statistics.
Results: The following journals fulfilled the selection criteria: Journal of Clinical Periodontology, Journal of Periodontology, Journal of Periodontal Research and International Journal of Periodontics & Restorative Dentistry. From these journals, 387 clinical trials were analyzed. Of the maximum of 24 points in the scale, the mean reached was 17.45 points.
Conclusion: The methodological quality of periodontal clinical trials indexed in ISI is deficient.
Keywords: Methodological quality, Clinical trials, Periodontics.
Received on: 03 May 15 Accepted on: 25 Nov 2015
Introduction
In scientific literature, the impact factor is defined as the frequency with which a journal’s article is cited (1). This is considered a quantitative measurement of the quality of the journal (2). However, there is a conflict regarding the correlation between the impact factor of the journal and the quality of its publications (3). This is due to the fact that the number of citations in an article does not necessarily reflect its methodological quality, since an article may be cited not only to highlight its positive characteristics but also to emphasize the negative ones.
As of the emergence of Evidence-Based Dentistry, literature has become an essential tool in the process of making clinical decisions. Despite this, in dentistry, publications do not necessarily have the clinical relevance the discipline demands, and in many cases, errors may occur in its methodology (4). In general, periodontics journals have a high impact factor. However, there are no studies that evaluate the methodological quality of the reports therein published. The evaluation of the methodological quality of an article is controversial. The methods and techniques used are a source of continuous debate, not only by the scientific community but also by editors (5). Therefore, it is important to judge and evaluate the quality of what is reported in scientific articles in order to identify the reliability of the information before its clinical application, as errors in design may alter results.
The aim of this study is to describe the methodological quality of periodontal clinical trials in journals indexed by the Institute for Scientific Information (ISI) published between 2008 and 2012.
Materials and Methods
This is a descriptive observational study in which periodontics journals were searched in the Journal Citation Reports. They were ranked according to the impact factor of the last five years. The four with the highest impact factor indexed to ISI were selected. They included clinical trials, were written in English and included the following words in their headings: periodontics, periodontal, periodontology.
In order to select the articles, an advanced search of each journal was conducted in PubMed, using as a search criterion the name of the journal and the following search filters: journal, article types, clinical trial and publication dates: 01-01-2008 to 31-12-2012.
The materials and methods were analyzed to verify that the studies were clinical trials; in other words, studies carried out with patients, where they underwent a diagnostic test or therapeutic treatment. The articles that did not qualify as clinical trials were excluded even when they had been indexed as clinical trials in PubMed.
Every study was submitted to an analysis of methodological quality through a scoring system designed for the evaluation of clinical trials in surgery and validated to be used in clinical studies in dentistry (6-7) (Table 1). This system contains originally three sections. However, only two were used in this study. The first section was related to the type of study design, and the second was related to the methodology description used in the study (description of the objectives, design justification, eligibility requirements and sample justification). Thus, a final score between 10 and 24 points, minimum and maximum respectively, was allocated.
The methodological quality evaluation of each article was conducted independently by two reviewers and a kappa index >0.8 was achieved in the calibration process. This consisted in applying the methodological quality scale to 20 periodontics clinical trials from journals that were not part of the study.
The articles analyzed were organized according to their geographical origin and the journals included in the evaluation. The mean and the standard deviation of the methodological quality score were also stated.
The data were tabulated in a Google Docs electronic form (Mountain View, CA, USA). Then, they were analyzed using descriptive statistics with SPSS 20 for Windows statistical software (IBM, Chicago, USA).
Results
The four journals with the highest periodontics impact that included clinical trials were: Journal of Periodontology, Journal of Clinical Periodontology, Journal of Periodontal Research and The International Journal of Periodontics and Restorative Dentistry.
After each journal was searched in PubMed, 493 clinical trials were obtained. Their full texts were analyzed and, through the analysis of their materials and methods, it was determined that only 78% (387) of the articles were actually clinical trials (Figure 1).
Figure 1. Flow chart of the number of articles obtained.
The continent with the highest number of published clinical trials was Europe (47.8%) and the lowest was Oceania with only one clinical trial published in the selected journals between 2008 and 2012 (Table 2).
The following image shows the number of clinical trials published per journal from January 2008 to December 2012.
Of the maximum of 24 points in the scale, the mean reached was 17.45 points. Out of the articles analyzed, 3.3% (13) reached the highest score. These were published in the Journal of Periodontology (n=5) and the Journal of Clinical Periodontology (n=8).
Based on the analysis conducted, it was determined that the Journal of Clinical Periodontology had the highest average score (18.06 points). Table 3 specifies the Methodological Quality mean score per journal. These scores are compared in figure 3.
Regarding the sections analyzed with the scale, the ones with the lowest score were:
a. in the study design variable, the multicenter clinical trials were the least frequent, with only 7.2% of the articles analyzed (Table 4);
b. the lack of explicit sample inclusion and exclusion selection criteria regarding sample selection, since almost 30% of the articles mention these criteria incompletely (Table 4);
c. the lack of justification of the sample size, where 47.3% of the articles did not justify the sample used. The journal with the most deficiencies in this area was the Journal of Periodontics and Restorative Dentistry, with 95.3% of its articles having this problem (Table 5).
As for the objectives of the articles analyzed, 92.2% of them clearly and specifically presented their objectives, this figure reaching 97% in the Journal of Periodontology (Table 6).
Discussion
The mean reached by the articles evaluated from the four journals selected through the selection criteria and to which the scale was applied was 17.45, out a maximum of 24 points.
Based on the analysis of the clinical trials between 2008 and 2012, the source continent with the highest publication is Europe (47.8%), which may be due to the fact that the journals studied are European. However, in comparison to the other continents, the mean score in the Methodological Quality Scale is lower than the average score of all the continents. Oceania is a different case, since it has the highest score but only one article published in the journals studied, therefore it cannot be considered representative.
The Journal of Periodontology had the highest number of publications of clinical trials. Still, it did not reach the highest mean out of all the journals. Based on this, the question is: why does a scientific journal publish more clinical trials than another? It may be due to its reputation assigned by its readers, the prestige of its publications, the reputation of its editors or the impact factor, which can be even more important. If a journal has a higher impact factor, it is likely that the authors would like to publish in said journal, disregarding the fact that the design of the articles may be deficient.
The lower frequency of the multicenter studies could be due to the complexity of their execution: taking into account all the parameters that should be considered for the procedure standardization, planning and execution. In addition, the reason for a higher frequency of the unmasked or single-masked clinical studies can be that their execution feasibility is higher than that of the multicenter study.
The lowest score was reached in the sample size variable, with 47% (87) of the studies incorrectly justifying the sample. Not being clear about it can lead to a distortion when the study is replicated or when the results are extrapolated to reality. In fact, if the sample used cannot be justified, the study loses credibility and leads to misinterpretations (8). Authors must understand that the impact of the sample size when interpreting the data of a study is a prerequisite for high-quality, innovative and valid research (9).
As for the sample selection criteria variable, 28.4% (n=110) of the papers only describe inclusion or exclusion criteria. When only one variable is described, the other variable must be interpreted by the reader. An ambiguous definition of the sample selection criteria will not allow us to replicate the study since there is no certainty about the selection of the study population.
In a study conducted by Clovis (2012), which evaluated periodontal clinical trial abstracts using the CONSORT (10) statement, it was concluded that that their quality can be improved (11). This agrees with the results of this evaluation, since according to the abstracts; the methodological quality of the articles is clearly low.
When entering the filter "clinical trial" in PubMed and after thoroughly analyzing the texts obtained, researchers verified that the results obtained from the search engine were not all clinical trials, since many of them did not compare the therapeutic procedures, which means that some articles are being incorrectly indexed.
An important limitation is that in dentistry, so far, there are no similar studies that evaluate the methodological quality of the entire text with which to contrast the results obtained in this study. Therefore, it cannot be established whether there was an improvement in the study methodology or not.
It is questionable whether the great numbers of articles that are currently published are actually useful and relevant to dentistry as a field of knowledge. As mentioned above, there is a larger percentage of methodologically deficient articles, which can lead readers to make mistakes in their decision-making processes. In this context, we could suggest improving the research protocols and the strictness of the editors before approving the publication of an article. In addition, we recommend the creation of a new reliable, validated and standardized evaluation scale to conduct a periodical analysis of the methodological quality of each journal (6).
In the meantime, clinical trial authors should follow the CONSORT statement (10) guidelines, which would allow us to standardize and compare articles in the future. Additionally, reviewers would be asked to promote clearer rules to improve the authors’ compliance with them. We recommend that the editors hire scientific research specialized reviewers, said expertise being qualitative or quantitative, depending on the article to analyze.
For future studies, we suggest that articles such as this be written in order to analyze whether there is an improvement in article quality over time, and also that we study whether the highest impact factor journals are the ones that actually include articles with better methodological quality.
Conclusion
The methodological quality of periodontal clinical trials indexed in ISI journals is deficient.
References
1. Saha S, Saint S, Christakis DA. Impact factor: a valid measure of journal quality? J Med Libr Assoc [en línea] 2003; 91(1): 42–46. Consultado noviembre 2015. Disponible en: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC141186/pdf/i0025-7338-091-01-0042.pdf
2. Garfield E. How can impact factors be improved? BMJ 1996 Aug 17;313(7054):411-3.
3. Martínez-Fuentes J, Meroño A, Ríos-Díaz J. El factor de impacto como criterio para la evaluación de la producción y la calidad científica. Rev Iberoam Fisioter Kinesiol [en línea] 2010; 13(1):29–36. Consultado noviembre 2015. Disponible en:
http://www.elsevier.es/es-revista-revista-iberoamericana-fisioterapia-kinesiologia-176-articulo-el-factor-impacto-como-criterio-13152251?referer=buscador
4. Uribe S. ¿Qué es la Odontología Basada en la Evidencia? Rev Fac Odontol UV. 2000; 2(4): 281-287
5. Ismail AI, Bader JD. Evidence-based dentistry in clinical practice. J Am Dent Assoc. 2004; 135(1): 78-83
6. Manterola C, Grande L. Methodological quality of articles on therapeutic procedures published in Cirugía Española. Evaluation of the period 2005–2008. Cir Esp [en línea] 2010; 87(4):244–250. Consultado noviembre 2015. Disponible en: http://www.elsevier.es/en/linksolver/ft/pii/S0009-739X(09)00567-3
7. Cartes -Velásquez R, Manterola C, Aravena P, Moraga J. Reliability and validity of MINCIR scale for methodological quality in dental therapy research. Braz Oral Res 2014; 28(1):1-5.
8. Schulgen G, Olschewski M, Krane V, Wanner C, Ruf G, Schumacher M. Sample sizes for clinical trials with time-to-event endpoints and competing risks. Contemp Clin Trials [en línea] 2005 Jun; 26(3):386-96. Consultado: noviembre 2015. Disponible en: http://dx.doi.org/10.1016/j.cct.2005.01.010
9. Edmiston CE Jr, Josephson A, Pottinger J, Ciacco-Tsivitis M, Palenik C. The numbers game: sample-size determination. Am J Infect Control [en línea] 1993; 21(3):151-4. Consultado: noviembre 2015. Disponible en: http://dx.doi.org/10.1016/0196-6553(93)90007-Q
10. CONSORT. CONSORT endorsers-journals. Consultado: noviembre 2015. Disponible en: http://www.consort-statement.org/downloads/consort-statement
11. Faggion CM Jr, Giannakopoulos NN. Quality of reporting in abstracts of randomized controlled trials published in leading journals of periodontology and implant dentistry: a survey. J Periodontol 2012; 83(10):1251-6
