SciELO - Scientific Electronic Library Online

 
vol.18 issue1Sociodemographic variables and prejudices perception from disabled peoplePredictive ability of cognitive flexibility and planning in early mathematical competencies author indexsubject indexarticles search
Home Pagealphabetic serial listing  

Services on Demand

Journal

Article

Related links

Share


Ciencias Psicológicas

Print version ISSN 1688-4094On-line version ISSN 1688-4221

Cienc. Psicol. vol.18 no.1 Montevideo  2024  Epub June 01, 2024

https://doi.org/10.22235/cp.v18i1.3178 

Original Articles

Instruments for evaluation of aggressiveness in sports fans: a systematic review

Geovani Garcia Zeferino1 
http://orcid.org/0000-0002-8245-668X

Anna Clara Santos da Silva2 
http://orcid.org/0000-0002-0831-6793

Marcela Mansur-Alves3 
http://orcid.org/0000-0002-3961-3475

1Universidade Federal de Minas Gerais, Brasil, geogarciapsico@gmail.com

2 Universidade Federal de São João del-Rei, Brasil

3Universidade Federal de Minas Gerais, Brasil


Abstract:

The aim of the study was to identify instruments used to assess the aggressiveness of sports fans and describe their operational and psychometric characteristics. A systematic literature review was carried out, following PRISMA guidelines, in the BVS, PsycINFO, PubMed/MedLine, Science Direct, Scopus and Web of Science databases, using the Boolean descriptors and operators: scale OR test OR inventory OR questionnaire AND aggression OR violence OR aggressiveness AND spectators OR fans. 198 studies were found and, after exclusion criteria, 15 remained. In these studies, 11 instruments were identified. These instruments showed differences in theoretical bases, number and content of dimensions, administration method and type of response. In order to verify the robustness of the instruments, the psychometric evidences were scored. The findings of the present review showed the lack of specific instruments to assess the aggressiveness of fans, which can help researchers in the potential creation of measures for this purpose.

Keywords: aggressiveness; fans; measurement instruments; assessment; sports

Resumo:

O objetivo do estudo foi identificar instrumentos utilizados para avaliar a agressividade de torcedores esportivos e descrever suas características operacionais e psicométricas. Foi realizada uma revisão sistemática na literatura, seguindo diretrizes do PRISMA, nas bases de dados BVS, PsycINFO, PubMed/MedLine, Science Direct, Scopus e Web of Science, utilizando os descritores e operadores booleanos: scale OR test OR inventory OR questionnaire AND aggression OR violence OR aggressiveness AND spectators OR fans. Foram encontrados 198 estudos e após critérios de exclusão, restaram 15. Nesses estudos, foram identificados 11 instrumentos. Esses instrumentos, apresentaram diferenças em bases teóricas, número e conteúdo das dimensões, modo de administração e tipo de resposta. No intuito de verificar a robustez dos instrumentos, as evidências psicométricas foram pontuadas. As descobertas da presente revisão evidenciaram carência de instrumentos específicos para avaliar agressividade de torcedores e poderão auxiliar pesquisadores em possíveis criações de medidas com essa finalidade.

Palavras-chave: agressividade; torcedores; instrumentos de medida; avaliação; esportivos

Resumen:

El objetivo del estudio fue identificar instrumentos utilizados para evaluar la agresividad de los aficionados al deporte y describir sus características operativas y psicométricas. Se realizó una revisión sistemática de la literatura, siguiendo las pautas PRISMA, en las bases de datos BVS, PsycINFO, PubMed/MedLine, Science Direct, Scopus y Web of Science, utilizando los descriptores y operadores booleanos: escala OR test OR inventario OR cuestionario Y agresión OR violencia OR agresividad OR espectadores OR aficionados. Fueron encontrados 198 estudios y después de los criterios de exclusión quedaron 15. En ellos se identificados 11 instrumentos. Estos instrumentos mostraron diferencias en las bases teóricas, número y contenido de dimensiones, modo de administración y tipo de respuesta. Para verificar la robustez de los instrumentos, se puntuaron las evidencias psicométricas. Los hallazgos de la presente revisión mostraron la falta de instrumentos específicos para evaluar la agresividad de los fanáticos, que pueden ayudar a los investigadores en la posible creación de medidas para este propósito.

Palabras clave: agresividad; aficionados; instrumentos de medición; evaluación; deporte

Sport is undoubtedly the cultural expression that has strengthened the most in recent decades (Rubio & Camilo, 2019), currently boasting billions of fans worldwide (Budevici-Puiu et al., 2020). Fans are those individuals motivated to regularly follow sports competitions, whether in person or through media (Ranjan & Muraleedharan, 2021). Sports events, on the other hand, are occasions that elicit various feelings and manifestations among their fans. While these competitions provide joy and positive emotions (Budevici-Puiu et al., 2020), they also replicate the aggression seen in the social everyday life of many individuals (Murad, 2017).

Specifically in sports, aggressions have occurred in professional, amateur, university, and youth competitions (Lake, 2020), spanning various disciplines (Turğut et al., 2018). Aggressiveness is a construct that presents several distinct definitions. Generally, it can be understood as an intentional action aimed at causing harm to another person or oneself (Allen & Anderson, 2017), which can be expressed physically, verbally, psychologically, among other forms (Anderson & Huesmann, 2003). In sports context, aggression by fans is a phenomenon that occurs worldwide (Brandão et al., 2020) and has become a concerning issue due to its various detrimental effects and negative consequences (Murad, 2017).

Among the most cited damages and consequences are the thousands of deaths and injuries resulting from fights involving fans (Lake, 2020; Murad, 2017). Concerning injuries, it is noted that there are impacts on the economy of the state, as hospitals provide necessary treatment and medicine to the injured, incurring additional expenses (Maciel et al., 2016). Moreover, in public security, according to Murad (2017), there are instances of aggressive behavior by police officers too, leading to clashes with fans. Consequently, the security staff get injured as well, are taken to public hospitals, and in some cases, are relieved of their duties.

Other impacts are observed in leisure activities of some fans, who refrain from or reduce their attendance at competition venues (stadiums, arenas, among others) due to these disturbances. Consequently, clubs experience reduction in ticket sales (Toder-Alon et al., 2018). Furthermore, athletes are also affected, as pressure, excessive demands, and aggression by fans have led to decreased performance and, in some cases, common mental disorders due to high levels of anxiety and stress (Albino & Conde, 2019).

In the light of the foregoing, it is emphasized that aggression by fans has led to negative consequences for clubs, other fans, and society at large (Lake, 2020; Murad, 2017). Therefore, tools to assess aggression by fans are of utmost importance to identify aggressive profiles and subsequently consider and promote alternatives to mitigate aggressive behavior. In this regard, Zeferino et al. (2021) highlighted the urgency of discussing strategies to reduce aggression. However, the authors mentioned the difficulty in measuring aggression by fans due to the lack of specific instruments.

Furthermore, concerning specific instruments for assessing aggression by fans, several studies (Carriedo et al., 2020; Turğut et al., 2018; Moore et al., 2007; Zeferino et al., 2021) used instruments without a focus on fans, such as the Buss and Perry Aggression Questionnaire (Buss & Perry, 1992) and the Buss-Durkee Hostility Inventory (Buss & Durkee, 1957). Meanwhile, this has become a concern because the widespread use of instruments that disregard the specificities of sports (Carriedo et al., 2020; Zeferino et al., 2021). Additionally, it is crucial to use specific instruments to the target population with norms to eliminate biases due to various differences (e.g., cultural, linguistic, among others) that can affect results (American Educational Research Association (AERA) et al., 2014), and disregard the specificities, behaviors, and reality of the sports context (Wachelke et al., 2008).

In light of the above, the aim of this study was to identify, through a systematic review, instruments used to measure the aggression by sports fans and analyze their theoretical bases, operational characteristics (e.g., quantity and content of dimensions, population, number of items, and type of response), and psychometric evidence.

Method

A systematic literature review was conducted following the guidelines of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA Statement; Moher et al., 2015) to systematize the review, but reducing biases and enhancing methodological quality. Operational and psychometric characteristics of the reviewed instruments were evaluated based on the study by Silva et al. (2018).

Search and Selection of Instruments

Search was conducted in the following databases, namely, BVS, PsycINFO, PubMed/MedLine, Science Direct, Scopus, and Web of Science, using the following combination of descriptors (only in English) and Boolean operators: scale OR test OR inventory OR questionnaire AND aggression OR violence OR aggressiveness AND spectators OR fans. To ensure methodological rigor, there was a consultation of studies on this topic and keyword indexes as Health Sciences Descriptors (DeCS) and Medical Subject Headings (MeSH) for the selection of descriptors. For each database, article search was conducted independently by two researchers. Additionally, they managed references using EndNote Web®, available at https://www.myendnoteweb.com.

The search was done on December 28, 2021, aiming to identify studies that used scales, inventories, questionnaires, or other measures to assess aggressiveness by fans. It's worth noting that no publication period for studies was delimited to broaden the scope. Furthermore, there was an article reference section tracking that met the eligibility criteria for the review to identify additional studies that were aligned with the proposed objective, the insertion of the inclusion and exclusion criteria.

It is highlighted that the inclusion criteria were: (1) empirical articles and (2) articles that used instruments to assess aggressiveness by fans. The exclusion criteria were: (1) articles published in languages other than Portuguese, English, Spanish, and French; (2) studies with non-psychometric instruments; and (3) studies that presented instruments without the possibility of accessing the full text. The selected articles were read in full and grouped according to the categories of interest (e.g., theory underlying the instrument, dimensionality, target population, type of response scale, type of application, among others).

Definition of Criteria for Psychometric Quality Assessment

In order to identify the psychometric quality of the instruments that were found, scoring criteria were developed following recommendations from the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (AERA et al., 2014). These criteria classified the instruments according to types of validity and reliability. To accomplish this, a scoring scale (ranging from zero to four points) was used, progressing in quality order. Therefore, the score zero indicated a lack of psychometric evidence, while the score four denoted more robust instruments concerning their validity or reliability. In other words, these instruments employed more modern, advanced, and rigorous statistical methods and analyses. Specifically, in evaluating criterion-related validity and reliability (temporal stability and internal consistency), only three scoring possibilities were adopted (0, 2, and 4). The descriptions used for the instrument classification are detailed in Table 1.

Table 1: Criteria Definition for Psychometric Characteristics Scoring for the Instruments Analyzed in the Review 

Source:Notes: Based on the study by Silva et al. (2018) 1 Content validity sources: consistent theoretical basis, use of items validated in other instruments, instrument analysis by experts, pilot study, semantic analysis of items; 2Dimensionality and relationships between scores or subscores of the same inventory; 3Specificities: sports discipline, belonging to groups of fans.

Results

Regarding the articles that were found, there was total agreement (100%) between the two researchers. With the defined descriptors and operational criteria, 198 studies were identified. After removing duplicates, 155 remained, of which 140 were excluded because: (a) they were not empirical articles (n = 3); (b) there was no access to the instrument (n = 1); (c) they were books or chapters (n= 17); (d) dissertations or theses (n = 2); (e) published in languages other than Portuguese, English, Spanish, or French (n = 5); (f) they addressed other subjects (n = 74); (g) the instrument used was created only for the study and/or did not report the use of at least one psychometric instrument that assessed aggression (n = 10); or (h) they did not measure aggression (n = 28). In total, 15 studies met the inclusion criteria (Figure 1).

Figure 1: Flowchart of the systematic search process (PRISMA) 

The methods in the 15 articles were read to identify and count the measurement instruments used to assess aggression and verify the reported psychometric evidence. Seven instruments were identified, namely: (1) Buss-Durkee Hostility Inventory (BDHI; Buss & Durkee, 1957 ) (f = 5); (2) Buss and Perry Aggression Questionnaire (BPAQ; Buss & Perry, 1992) (f = 4); (3) Habitus of Chinese Football (HCF; Hu & Cui, 2020) (f = 1); (4) Fan Behavior Measure (FBM; Rocca & Vogl‐Bauer, 1999) (f = 2); (5) Hostile and Instrumental Aggression of Spectators Questionnaire (HIASQ; Wann, Fahl, et al., 1999) (f= 3); (6) Team Sport Club Aggression Scale (TSCAS; Shuv-Ami & Toder-Alon, 2021) (f = 1); and (7) Violence Tendency Scale (VTS; Yalcin et al., 2021) (f = 1). Furthermore, it is noteworthy that in the study by Shuv-Ami and Toder-Alon (2021), three instruments were used. Table 2 presents the study descriptions.

Table 2: Description of studies in which the instruments were initially found 

Note: Instrum#: Instrument; BDHI: Buss-Durkee Hostility Inventory; BPAQ: Buss-Perry Aggression Questionnaire; HCF: Habitus of Chinese Football; FBM: Fan Behavior Measure; HIASQ: Hostile and Instrumental Aggression Spectators Questionnaire; TSCAS: Team Sport Club Aggression Scale; VTS: Violence Tendency Scale; * Contributed to the development of items (physical and verbal aggression) for the construction of a new instrument; M; average age; SD: standard deviation.

Additionally, four more instruments were found in other information sources (reference sections of the included studies), namely: (1) Brief Aggression Questionnaire (BAQ; Vitoratou et al., 2009); (2) Competitive Aggressiveness and Anger Scale (CAAS; Maxwell & Moores, 2007); (3) The Reactive-Proactive Questionnaire (RPQ; Raine et al., 2006); and (4) The Verbal Aggression Scale (VAS; Infante & Wigley, 1986). Therefore, a total of 11 instruments was selected for a qualitative analysis of their descriptive and psychometric characteristics.

Operational characteristics of the instruments

Regarding the theoretical foundations, the General Aggression Model (GAM; Anderson & Bushman, 2002) was the most used reference point. Other models such as Social Learning, Neo-Associationist Cognitive Theory, Social Identification, and Practice Theory were also employed. The dimensionality varied; however, the multidimensional format was the most prevalent, and used in six instruments. Among the multidimensional instruments, three of them, Buss-Durkee Hostility Inventory (Buss & Durkee, 1957 ), Buss and Perry Aggression Questionnaire (Buss & Perry, 1992), and Brief Aggression Questionnaire (Vitoratou et al., 2009), are "progressive" versions, meaning revised and reduced versions of the original instrument.

The most employed dimensions in the instruments were verbal aggression (f= 5, representing 45% of the instruments) and physical aggression, anger, and hostility (f= 2, equivalent to 18 % of the instruments). Other forms of aggression were also measured by the instruments, but less frequently. In some cases, the measures were more general (e.g., subscales evaluating aggression without differentiation, aggression towards the team and fans, in these cases without reporting the type of behavior). There were also measures/subscales assessing other specificities (e.g., irritability, resentment, emotional reaction, guilt, negativism, tendency, based on the identification with the team, reactions during victories and defeats, etc.).

Regarding the study populations, seven studies specifically focused on sports fans, five of them in soccer (Bensimon & Bodner, 2011; Hu & Cui, 2020; Shuv-Ami & Toder-Alon, 2021; Yalcin et al., 2021; Zeferino et al., 2021), one in hockey (Harrell, 1981), and one in rugby (Moore et al., 2007). Six studies focused on university students (Bernache-Assollant & Chantal, 2009; End & Foster, 2010; Rocca & Vogl‐Bauer, 1999; Wann, 1994; Wann, Carlson et al., 1999; Wann, Fahl, et al., 1999).

There was also a study with a population of men attending a hockey match (Russell & Arms, 1995), students who are supporters of a university basketball (Wann, Carlson, et al., 1999), and parents of young baseball athletes (Hennessy & Schwartz, 2007). Concerning additional instruments (found through other sources of information), one of them investigated competitive athletes (CAAS; Maxwell & Moores, 2007), two of them were about general population (BAQ; Vitoratou et al., 2009; VAS; Infante & Wigley, 1986), and another one was about teenagers (RPQ; Raine et al., 2006). Consequently, those cases in which the sports context was not addressed, the item responses in the measurement instruments might have been influenced by response biases. Also related to the items, the quantity varied from 6 to 75, with the majority of them ranging from 20 to 29.

Regarding the mode of application, all instruments are self-administered. Regarding the type of response, there was some variation, most of them in Likert format (f= 8). Within these eight instruments, in only one of them the answer options ranged from zero to two points, while in the others, they ranged from one to five points. In two instruments, the answer options were scored on an anchoring scale (in one of them, from 1 to 8 points, and in another, from 1 to 10 points), while in one study the answers were dichotomous (where the respondent indicated true or false for the items). Table 3 contains an operational description of the aggressiveness measurement instruments analyzed in this review.

Table 3: Descriptive and operational characteristics of the instruments analyzed in the review 

Note: Instrum#: Instrument; BDHI: Buss-Durkee Hostility Inventory; BPAQ: Buss-Perry Aggression Questionnaire; HCF: Habitus of Chinese Football; FBM: Fan Behavior Measure; HIASQ: Hostile and Instrumental Aggression Spectators Questionnaire; TSCAS: Team Sport Club Aggression Scale; VTS: Violence Tendency Scale; BAQ: Brief Aggression Questionnaire; CAAS: Competitive Aggressiveness and Anger Scale; RPQ: Reactive-Proactive Questionnaire; VAS: Verbal Aggression Scale; *MGA: General Aggression Model; * Extra instruments, included by other sources (from the reference section of the initially included studies).

Psychometric Qualities

Psychometric qualities were assessed to determine the accuracy and robustness of the instruments included in this review. The analysis indicated that none of the eleven instruments reached the maximum score (24 points). The highest scores were obtained by the Buss-Perry Aggression Questionnaire (Buss & Perry, 1992), with 19 points, followed by The Reactive-Proactive Questionnaire (Raine et al., 2006) and the Brief Aggression Questionnaire (Vitoratou et al., 2009), both with 18 points. The Competitive Aggressiveness and Anger Scale (Maxwell & Moores, 2007) also scored 17 points. The Verbal Aggression Scale (Infante & Wigley, 1986) received 14 points, while the other instruments presented unsatisfactory psychometric evidence (12 points or less), mainly in construct and criterion validities, as well as reliability in relation to temporal stability.

As for content-based validity evidence, five instruments -Brief Aggression Questionnaire (Vitoratou et al., 2009), Buss-Perry Aggression Questionnaire (Buss & Perry, 1992), Competitive Aggressiveness and Anger Scale (Maxwell & Moores, 2007), Team Sport Club Aggression Scale (Shuv-Ami & Toder-Alon, 2021), and The Reactive-Proactive Questionnaire (Raine et al., 2006)- obtained 3 points, as they presented theoretical bases and used previously tested items. In addition to these instruments, the Buss-Durkee Hostility Inventory (Buss & Durkee, 1957) and the Violence Tendency Scale (Yalcin et al., 2021) presented other types of evidence, such as semantic analysis of the items and evaluation by experts, achieving the maximum score, 4 points. The remaining instruments obtained 2 points.

Regarding the evidence of internal structure, only one instrument (HIASQ; Wann, Carlson et al., 1999) did not score, two of them (RPQ; Raine et al., 2006 and VAS; Infante & Wigley, 1986) obtained three points. The other eight reached the full score. Three instruments (BDHI; Buss & Durkee, 1957, HCF; Hu & Cui, 2020 and FBM; Rocca & Vogl‐Bauer, 1999) scored for performing exploratory factor analysis, and three of the, (BAQ; Vitoratou et al., 2009, RPQ; Raine et al., 2006 and VAS; Infante & Wigley, 1986) for conducting confirmatory factor analysis. Four instruments (BPAQ; Buss & Perry, 1992, CAAS; Maxwell & Moores, 2007, TSCAS; Shuv-Ami & Toder-Alon, 2021 and Yalcin et al., 2021) employed both factor analysis methods. In addition, the Brief Aggression Questionnaire (Vitoratou et al., 2009) used Item Response Theory (IRT), and the Team Sport Club Aggression Scale (Shuv-Ami & Toder-Alon, 2021) employed nomological network. Structural equation modeling was applied at two scales (CAAS; Maxwell & Moores, 2007 and TSCAS; Shuv-Ami & Toder-Alon, 2021).

As for the validity evidence based on the relation to conceptually related constructs, only two instruments (BAQ; Vitoratou et al., 2009 and RPQ; Raine et al., 2006) received the maximum score (four points) because they showed significant correlations with other instruments and were used to validate others. Three other instruments also scored, with The Verbal Aggression Scale (Infante & Wigley, 1986) showing correlations with two measurement instruments (three points), and the Buss-Perry Aggression Questionnaire (Buss & Perry, 1992) and the Competitive Aggressiveness and Anger Scale (Maxwell & Moores, 2007) having correlations with one instrument (two points). Six instruments were not scored in this evidence of validity.

In the validity analyses based on the relation to the criterion, only the Competitive Aggressiveness and Anger Scale (Maxwell & Moores, 2007) and The Reactive-Proactive Questionnaire (Raine et al., 2006) achieved the maximum score. These two instruments were subjected to comparative analyses of sports specificities and associations with variables such as age, gender, education and/or socioeconomic status. The Brief Aggression (Vitoratou et al., 2009) and Buss-Perry Aggression (Buss & Perry, 1992) questionnaires, as well as The Verbal Aggression Scale (Infante & Wigley, 1986), received two points due to comparisons of respondents' means and gender differences. The other six instruments did not score in this evidence of validity.

Evidence of reliability was assessed by two methods: temporal stability and internal consistency. Regarding temporal stability, three instruments (BPAQ; Buss & Perry, 1992, CAAS; Maxwell & Moores, 2007 and VAS; Infante & Wigley, 1986) achieved the maximum score, presenting satisfactory indicators as expected (correlation above 0.7). The Brief Aggression Questionnaire (Vitoratou et al., 2009) also scored on this evidence, but its values were below 0.7, obtaining only 2 points. The other instruments did not score. In the second method (internal consistency), two instruments (CAAS; Maxwell & Moores, 2007 and VAS; Infante & Wigley, 1986) did not receive scores, and two studies (BDHI; Buss & Durkee, 1957 and BAQ; Vitoratou et al., 2009) presented values below 0.7, obtaining 2 points. The other seven instruments reached the total score (four points), as they demonstrated internal consistency above 0.7 for the dimensions of the instrument. Table 4 shows the scores (partial and total) of each instrument analyzed.

Table 4: Score of the psychometric evidence of the instruments present in the review 

Notes: Instrum #; BDHI: Buss-Durkee Hostility Inventory; BPAQ: Buss-Perry Aggression Questionnaire; HCF: Habitus of Chinese Football; FBM: Fan Behavior Measure; HIASQ: Hostile and Instrumental Aggression Spectators Questionnaire; TSCAS: Team Sport Club Aggression Scale; VTS: Violence Tendency Scale; BAQ: Brief Aggression Questionnaire; CAAS: Competitive Aggressiveness and Anger Scale; RPQ: Reactive-Proactive Questionnaire; VAS: Verbal Aggression Scale; *Instruments included after inclusion criteria.

In summary, it was observed that most of the instruments presented evidence of content validity through a consistent theoretical basis and empirical experience, using items previously tested in other instruments. In addition, some studies have employed techniques such as semantic analysis of items and analysis by expert judges. In the evaluation of the internal structure of the instruments, it was found that factor analysis (exploratory and/or confirmatory) was the most used technique, appearing in ten of the eleven instruments, in some cases with the use of modern psychometric methods, such as item response theory and nomological network. Regarding the evidence of validity of conceptually related constructs and criteria, there was a low level of exploration by the authors, being carried out in only five instruments. Regarding the evidence of reliability, only four instruments assessed temporal stability (test-retest). However, in terms of internal consistency, only one study did not address this evidence.

Discussion

The aim of this study was to conduct a systematic review of the literature on the instruments used to assess the aggressiveness by sports fans. No specific period was established for the search, with the intention of accumulating information about the instruments. This fact allowed to obtain knowledge about the theoretical bases, operational and psychometric characteristics of these measures, in order to discuss, both theoretically and psychometrically, the instruments, with the purpose of supporting the creation of a specific instrument.

Eleven instruments were identified that assessed aggressiveness by fans. However, only four of them (HCF; Hu & Cui, 2020, FBM; Rocca & Vogl‐Bauer, 1999, HIASQ; Wann, Carlson et al., 1999 and TSCAS; Shuv-Ami & Toder-Alon, 2021) are specific to the population of fans. In addition, on a scale (CAAS; Maxwell & Moores, 2007), the population was composed of athletes. The small number of specific instruments found to assess aggressiveness by fans corroborates what was presented by Zeferino et al. (2021).

In the specific instruments, differences in the focus of investigation were identified: in one case (HCF; Hu & Cui, 2020), aggressiveness was measured in the cruelest/most brutal form and its manifestations after the team's victory or defeat, excluding other forms; in another (TSCAS; Shuv-Ami & Toder-Alon, 2021), respondents did not report about themselves, but the items were directed at the behavior of other fans. In addition, two studies (FBM; Rocca & Vogl‐Bauer, 1999 and HIASQ; Wann, Carlson et al., 1999) assessed aggression directed at officials/authorities and adversaries. These facts corroborate authors in the area who have highlighted that some studies ignore important factors and contexts for understanding these fan manifestations (Brandão et al., 2020; Murad, 2017; Zeferino et al., 2021), with a strong tendency in Psychology to restrict itself to the clinical context (Tertuliano & Machado, 2019). In addition, the limitation in capturing the details, behaviors, and reality of fans more accurately (Zeferino et al., 2021) is pointed up.

Regarding the theoretical bases, the General Aggression Model (GAM) by Anderson and Bushman (2002) was the most used. The GAM is based on an interactive perspective between biological, psychological, social, and cultural elements, considering that the understanding of aggressiveness must be multidimensional. Additionally, it is important to note that this model does not rule out the others; The authors' proposal was to theoretically broaden the understanding of aggressiveness, including some aspects of other theories (e.g., Social Learning, Neo-Associationist Cognitive, among others). Thus, authors in the field (Allen & Anderson, 2017; Bushman & Anderson, 2020) emphasized that the model is the most complete and accepted due to this theoretical expansion and for reducing fragmented and/or reductionist conceptions. Therefore, it is believed that the GAM would be the most suitable for the creation of a new instrument for this purpose.

With regard to the structure of the instruments, two studies examined the respondents' perception of the behavior of other fans (TSCAS; Shuv-Ami & Toder-Alon, 2021 and Rocca & Vogl‐Bauer, 1999), while another one adopted one-dimensional structure (VAS; Infante & Wigley, 1986) and four of them, a two-dimensional structure (Raine et al., 2006, CAAS; Maxwell & Moores, 2007, TSCAS; Shuv-Ami & Toder-Alon, 2021 and HIASQ; Wann, Carlson et al., 1999). These instruments were considered limited, since aggressiveness by fans is a multidimensional phenomenon (Brandão et al., 2020; Moore et al., 2007; Zeferino et al., 2021) and composed of related components (Buss & Perry, 1992). For these authors, aggressiveness involves a stimulus, which is received by the cognitive component (hostility), triggering an emotional reaction (anger), and this reaction can produce an instrumental or motor component of the behavior (physical or verbal aggression).

Regarding the dimensionality of the instruments, a variety in the quantity and content of the dimensions was identified. This fact may be reflecting the extensive nature of the concept of aggressiveness and, at the same time, justifying the separation into sets of items (dimensions) to attenuate the complexity of the object of study and obtain a better understanding (Tay & Jebb, 2017), it means, to contribute to the evaluation and understanding of aggressiveness and its manifestation. In the analysis carried out, it was found that hostility, anger and, especially, verbal, and physical aggression are the most evaluated dimensions in the instruments. However, in some cases, the type of aggressiveness (either through a tendency or a concrete act) is not specified; in others, social or team identification (fanaticism) was verified; and there were cases in which the dimensions evaluated forms of aggressiveness, such as: indirect, directed (towards employees, authorities, and adversaries), proactive-reactive, and frustration-aggression.

Regarding psychometric properties, six of the eleven instruments analyzed did not present satisfactory indexes, considering the recommendations of AERA et al. (2014). Out of these six instruments, there are only four of them which were specific to populations of fans (HCF; Hu & Cui, 2020, FBM; Rocca & Vogl‐Bauer, 1999, HIASQ; Wann, Carlson et al., 1999 and TSCAS; Shuv-Ami & Toder-Alon, 2021). In all these cases, the scores were zero on the evidence of validity of conceptually related constructs and on the evidence of reliability by temporal stability.

Considering specific instruments for the context of fans, all of them had low scores (12 points or less), corresponding to half of the score in relation to the psychometric evidence. In addition, the Buss-Durkee Hostility Inventory (BDHI; Buss & Durkee, 1957 ), one of the instruments with the lowest score (ten points), was the most widely used. The Verbal Aggression Scale (Infante & Wigley, 1986) had a score considered medium (14 points). Based on the results, it was also found that four of the measurement instruments evaluated (BPAQ; Buss & Perry, 1992, BAQ; Vitoratou et al., 2009, CAAS; Maxwell & Moores, 2007 and RPQ; Raine et al., 2006) were considered satisfactory, i.e., they were in greater agreement with the literature in the area (DeVellis, 2003; Pasquali, 2010; Tay & Jebb, 2017), being elaborated and psychometrically analyzed with more theoretical and methodological rigor.

Regarding the use frequency of the instruments, the Buss-Durkee Hostility Inventory (f= 5) and the Buss-Perry Aggression Questionnaire (f= 4) were the most used ones. However, as pointed out by Maxwell and Moores (2007), the use of these two instruments in sports is troublesome, as they do not have specific and applicable items for the context in question. This fact reinforces the need to create a specific instrument, as in the absence of this measure, the authors end up resorting to instruments that presented limitations for the use in sports.

As for the evidence of content validity, six out of the eleven instruments were developed from the literature in the area, following theories about aggressiveness and other instruments. Only the Buss-Durkee Hostility Inventory (Buss & Durkee, 1957 ) and the Violence Tendency Scale (VTS; Yalcin et al., 2021) obtained the maximum score, as they had other types of evidence, such as semantic analysis of the items, to ensure that they were clear to members of the target population with a lower level of education, as recommended by Pasquali (2010). In addition, VTS had an analysis by expert judges. This technique is indicated to contemplate the considerations of scholars in the area and verify the relevance of the items to the theory, whether the items are being able to represent the construct evaluated and how clear the wording is (AERA et al., 2014; DeVellis, 2003).

After selecting the items, it is necessary to determine the dimensionality, usually verified by the internal structure (Muñiz & Fonseca-Pedrero, 2019). Ten out of eleven instruments performed some type of factor analysis. In three of them, exploratory factor analysis was used, due to the lack of a consistent theoretical basis or empirical evidence that indicated how the items should be retained and evaluated (Damásio, 2012). In other three instruments, confirmatory factor analysis was performed, in which the items were tested in a theoretical model already defined. In addition, the other four instruments used both methods. According to DeVellis (2003), the combination of these two types of factor analysis generates more robust psychometric results. The results found diverged from the analysis by Morgado et al. (2017) in stating the predominance of exploratory factor analysis in development of instruments.

Regarding validity evidence based on the relation to conceptually related constructs, only two questionnaires, the Brief Aggression (Vitoratou et al., 2009) and The Reactive-Proactive (Raine et al., 2006), reached the total score. They corroborated the findings of Muñiz and Fonseca-Pedrero (2019), highlighting that the proper selection of external variables can aggregate different types of evidence and provide more appropriate interpretations of the scores when using the instrument. However, six instruments (BDHI; Buss & Durkee, 1957, HCF; Hu & Cui, 2020, FBM; Rocca & Vogl‐Bauer, 1999, HIASQ; Wann, Carlson et al., 1999, TSCAS; Shuv-Ami & Toder-Alon, 2021, VTS; Yalcin et al., 2021) did not score any point in this category. This lack of punctuation suggests limitations, as pointed out by Tay and Jebb (2017), a measure that relates empirically to others confirms theoretical assumptions and provides important evidence of validity.

In the validity based on the relationship with criterion, only the Hostile and Instrumental Aggression (Wann, Carlson et al., 1999) and Reactive-Proactive (Raine et al., 2006) questionnaires, together with the Competitive Aggressiveness and Anger Scale (Maxwell & Moores, 2007), reached the total score. These instruments included comparative analyses with the type of sport, the severity of delinquency, and variables such as gender, age, socioeconomic status, and ethnicity. The careful choice of groups/variables to be used to assess the validity of the criterion is crucial and provides greater confidence in the interpretation of the instrument (Muñiz & Fonseca-Pedrero, 2019).

Regarding the evidence of reliability, only four instruments obtained a score in temporal stability. The other seven ones did not analyze this type of evidence, contradicting the authors in the area (DeVellis, 2003; Morgado et al., 2017), who state this instrument is one of the most investigated types of evidence in the development of others. From these seven instruments, except for The Reactive-Proactive Questionnaire (Raine et al., 2006), all obtained a low total score (less than half, i.e., 12 points). Regarding internal consistency, most of the instruments (nine of them) performed the analysis using Cronbach's alpha coefficient. According to Hair et al. (2009), this is the most used statistical procedure in this type of evidence.

The main aim of this review was to examine the instruments that measure aggression and to analyze their theoretical basis, operational characteristics, and psychometric evidence. However, there are some limitations. The first one is related to the non-inclusion of instruments that may have been published only in the gray literature (theses, dissertations, conferences, etc.). The second limitation concerns the descriptors; it is believed that the theme is not completely represented by these terms. To mitigate this limitation, the descriptors were selected based on keyword index search and related articles, as suggested by Clark and Watson (1995). The third one is the absence of an impact factor evaluation on the journals where the studies were published, as well as the lack of evaluation of the methodological quality of the articles. The fourth limitation, the choice of languages (Portuguese, English, Spanish and French) may not have covered relevant tools for the analyses and discussions.

Based on the findings of this review, the scarcity of specific instruments to assess aggressiveness by fans is remarkable, corroborating the report by Zeferino et al. (2021). This highlights the urgency of greater investment in research focused on construction and validation of instruments for this purpose. Despite the limitations, it is believed that this review offers relevant contributions to a broader understanding of the psychometric instruments used to assess aggressiveness by sports fans and, mainly, it presents the theoretical and psychometric bases for the development of a specific instrument that can fill this gap. However, it is important to emphasize that the assessment of aggressiveness is not restricted only to these aspects, as other variables can influence the manifestation of this construct.

Final considerations

The assessment of aggression by fans is of utmost importance for sports administrators, researchers, government officials/politicians, individuals directly involved in sports (staff, coaches, athletes, fans, among others), and society in general. There is a clear need for creation and validation of instruments grounded in theoretical and methodological foundations to assess the aggression by this population across various sports and groups.

It is believed that an instrument with this purpose could help to map aggression, facilitate the identification of aggressive profiles, and, most importantly, understand the levels of aggression by these fans. Thus, it would be possible to develop and coordinate measures in fan environments, competition venues, public health, and safety, as well as the performance and mental health of athletes, to benefit sports, investors, and society at large (Murad, 2017; Zeferino et al., 2021).

Based on this review, it is believed that a multidimensional instrument would be the most suitable to address the lack of instruments, given that aggression has multiple facets. Moreover, there is a noticeable need for specific instruments to assess aggression by fans with adequate psychometric evidence to address the methodological shortcomings identified in the instruments analyzed in this review. Furthermore, there is encouragement for further studies to understand other variables (e.g., fanaticism, anonymity within crowds, personality traits, social desirability, education, gender, age, marital status, among others) capable of affecting the aggressive behavior of this population (Bandeira & Ramos, 2020; Fanti et al., 2019; Hennessy & Schwartz, 2007; Zeferino et al., 2021).

REFERENCES

Albino, I., & Conde, E. (2019). Revisão sistemática: instrumentos para avaliação do estresse e ansiedade em jogadores de futebol. Revista Brasileira de Psicologia do Esporte, 9(1), 12-31. http://doi.org/10.31501/rbpe.v9i1.9706 [ Links ]

Allen, J. J., & Anderson, C. A. (2017). General Aggression Model. The International Encyclopedia of Media Effects, 1-15. https://doi.org/10.1002/9781118783764.wbieme0078 [ Links ]

American Educational Research Association, American Psychological Association & National Council for Measurement in Education. (2014). The Standards for Educational and Psychological Testing. [ Links ]

Anderson, C. A., & Bushman, B. J. (2002). Human Aggression. Annual Review of Psychology, 53(1), 27-51. https://doi.org/10.1146/annurev.psych.53.100901.135231 [ Links ]

Anderson, C. A., & Huesmann, L. R. (2003). Human aggression: A social-cognitive view. Em M. A. Hogg & J. Cooper (Eds.), The Sage Handbook of Social Psychology (pp. 296-323). Sage. [ Links ]

Bandeira, V. de F., & Ramos, D. G. (2020). Aspectos sociodemográficos relacionados à agressividade e ao fanatismo em uma torcida de futebol. Psicologia Revista, 29(1), 246-272. https://doi.org/10.23925/2594-3871.2020v29i1p246-272 [ Links ]

Bensimon, M., & Bodner, E. (2011). Playing With Fire: The Impact of Football Game Chanting on Level of Aggression. Journal of Applied Social Psychology, 41(10), 2421-2433. https://doi.org/10.1111/j.1559-1816.2011.00819.x [ Links ]

Bernache‐Assollant, I., & Chantal, Y. (2009). Perceptions of sport fans: An exploratory investigation based on aggressive and cheating propensities. International Journal of Sport and Exercise Psychology, 7(1), 32-45. https://doi.org/10.1080/1612197X.2009.9671891 [ Links ]

Brandão, T., Murad, M., Belmont, R., & Santos, R. F. D. (2020). Álcool e violência: Torcidas organizadas de futebol no Brasil. Movimento, 26. https://doi.org/10.22456/1982-8918.90431 [ Links ]

Budevici-Puiu, L., Manolachi, V., & Manolachi, V. (2020). Specific Elements of Good Governance in Sport, as Important Factors in Ensuring the Management. Revista Romaneasca pentru Educatie Multidimensionala, 12(4), 328-337. https://doi.org/10.18662/rrem/12.4/348 [ Links ]

Bushman, B. J., & Anderson, C. A. (2020). General Aggression Model. The International Encyclopedia of Media Psychology, 1-9. https://doi.org/10.1002/9781119011071.iemp0154 [ Links ]

Buss, A. H., & Durkee, A. (1957). An inventory for assessing different kinds of hostility. Journal of Consulting Psychology, 21(4), 343-349. https://doi.org/10.1037/h0046900 [ Links ]

Buss, A. H., & Perry, M. (1992). The Aggression Questionnaire. Journal of Personality and Social Psychology, 63(3), 452-459. https://doi.org/10.1037/0022-3514.63.3.452 [ Links ]

Carriedo, A., Cecchini, J. A., & González, C. (2020). Soccer spectators’ moral functioning and aggressive tendencies in life and when watching soccer matches. Ethics & Behavior, 31(2), 136-150. https://doi.org/10.1080/10508422.2020.1715801 [ Links ]

Clark, L. A., & Watson, D. (1995). Constructing Validity: Basic Issues in Objective Scale Development. Psychological Assessment, 7(3), 309-319. https://doi.org/10.1037/1040-3590.7.3.309 [ Links ]

Damásio, B. F. (2012). Uso da análise fatorial exploratória em psicologia. Avaliação Psicológica: Interamerican Journal of Psychological Assessment, 11(2), 213-228. https://www.redalyc.org/pdf/3350/335027501007.pdfLinks ]

DeVellis, R. F. (2003). Scale development: Theory and applications (2a ed.). Sage. [ Links ]

End, C. M., & Foster, N. J. (2010). The Effects of Seat Location, Ticket Cost, and Team Identification on Sport Fans' Instrumental and Hostile Aggression. North American Journal of Psychology, 12(3). [ Links ]

Fanti, K. A., Phylactou, E., & Georgiou, G. (2019). Who is the hooligan? The role of psychopathic traits. Deviant Behavior, 42(4), 492-502. https://doi.org/10.1080/01639625.2019.1695466 [ Links ]

Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2009). Análise multivariada de dados (6a ed.). Bookman. [ Links ]

Harrell, W. A. (1981). Verbal aggressiveness in spectators at professional hockey games: The effects of tolerance of violence and amount of exposure to hockey. Human Relations, 34(8), 643-655. https://doi.org/10.1177/001872678103400 [ Links ]

Hennessy, D. A., & Schwartz, S. (2007). Personal Predictors of Spectator Aggression at Little League Baseball Games. Violence and Victims, 22(2), 205-215. https://doi.org/10.1891/088667007780477384 [ Links ]

Hu, J., & Cui, G. (2020). Elements of the Habitus of Chinese Football Hooli-fans and Countermeasures to Address Inappropriate Behaviour. The International Journal of the History of Sport, 37(sup1), 41-59. https://doi.org/10.1080/09523367.2020.1742701 [ Links ]

Infante, D. A., & Wigley III, C. J. (1986). Verbal aggressiveness: An interpersonal model and measure. Communications Monographs, 53(1), 61-69. https://doi.org/10.1080/03637758609376126 [ Links ]

Lake, Y. A. (2020). Political causes of sport related violence and aggression: A systematic review. Iconic Research and Engineering Journals, 3(7), 50-53. [ Links ]

Maciel, P. R., Souza, M. R. D., & Rosso, C. F. W. (2016). Estudo descritivo do perfil das vítimas com ferimentos por projéteis de arma de fogo e dos custos assistenciais em um hospital da Rede Viva Sentinela. Epidemiologia e Serviços de Saúde, 25, 607-616.https://doi.org/10.5123/S1679-49742016000300016 [ Links ]

Maxwell, J. P., & Moores, E. (2007). The development of a short scale measuring aggressiveness and anger in competitive athletes. Psychology of Sport and Exercise, 8(2), 179-193. https://doi.org/10.1016/j.psychsport.2006.03.002 [ Links ]

Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015). Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Systematic Reviews, 4(1), 1-9. https://doi.org/10.1186/2046-4053-4-1 [ Links ]

Moore, S. C., Shepherd, J. P., Eden, S., & Sivarajasingam, V. (2007). The effect of rugby match outcome on spectator aggression and intention to drink alcohol. Criminal Behaviour and Mental Health, 17(2), 118-127. https://doi.org/10.1002/cbm.647 [ Links ]

Morgado, F., Meireles, J., Neves, C., Amaral, A., & Ferreira, M. (2017). Scale development: Ten main limitations and recommendations to improve future research practices. Psicologia: Reflexão e Crítica, 30(1), 1-20. https://doi.org/10.1186/s41155-016-0057-1 [ Links ]

Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test. Psicothema, 31(1), 7-16. https://doi.org/10.7334/psicothema2018.291 [ Links ]

Murad, M. (2017). A violência no futebol: novas pesquisas, novas ideias novas propostas (2a ed.). Benvirá. [ Links ]

Pasquali, L. (2010). Testes referentes a Construto: teoria e modelo de construção. Em L. Pasquali (Ed.), Instrumentação Psicológica: fundamentos e práticas (pp. 37-71). Artmed. [ Links ]

Raine, A., Dodge, K., Loeber, R., Gatzke‐Kopp, L., Lynam, D., Reynolds, C., Stouthamer-Loeber, M., & Liu, J. (2006). The reactive-proactive aggression questionnaire: Differential correlates of reactive and proactive aggression in adolescent boys. Aggressive Behavior: Official Journal of the International Society for Research on Aggression, 32(2), 159-171. https://doi.org/10.1002/ab.20115 [ Links ]

Ranjan, M., & Muraleedharan, S. (2021). Sports Fandom in Fredrerick Exley’sa Fans Notes. Annals of the Romanian Society for Cell Biology, 25(6), 9074-9078. https://www.annalsofrscb.ro/index.php/journal/article/view/7152/5348 Links ]

Rocca, K. A., & Vogl‐Bauer, S. (1999). Trait Verbal Aggression, Sports Fan Identification, and Perceptions of Appropriate Sports Fan Communication. Communication Research Reports, 16(3), 239-248. https://doi.org/10.1080/08824099909388723 [ Links ]

Rubio, K., & Camilo, J. D. O. (2019). Por quê uma Psicologia Social do Esporte. Em K. Rubio e J. A. de O. Camilo (Orgs.), Psicologia Social do Esporte (pp. 9-18). Képos. [ Links ]

Russell, G. W., & Arms, R. L. (1995). False consensus effect, physical aggression, anger, and a willingness to escalate a disturbance. Aggressive Behavior, 21(5), 381-386. https://doi.org/10.1002/1098-2337(1995)21:5<381::AID-AB2480210507>3.0.CO;2-L [ Links ]

Shuv-Ami, A., & Toder-Alon, A. (2021). A new team sport club aggression scale and its relationship with fans’ hatred, depression, self-reported aggression, and acceptance of aggression. International Journal of Sport and Exercise Psychology, 1-21. https://doi.org/10.1080/1612197X.2021.1979076 [ Links ]

Silva, M. A., Mendonça Filho, E. J., Mônego, B. G., & Bandeira, D. R. (2018). Instruments for multidimensional assessment of child development: a systematic review. Early Child Development and Care, 1-15. https://doi.org/10.1080/03004430.2018.1528243 [ Links ]

Tay, L., & Jebb, A. (2017). Scale Development. In S. Rogelberg (Ed.), The SAGE Encyclopedia of Industrial and Organizational Psychology (2a ed., pp. 1381-1384). Sage. [ Links ]

Tertuliano, I. W., & Machado, A. A. (2019). Psicologia do Esporte no Brasil: conceituação e o estado da arte. Pensar a Prática, 22. https://doi.org/10.5216/rpp.v22.53382 [ Links ]

Toder-Alon, A., Icekson, T., & Shuv-Ami, A. (2018). Team identification and sports fandom as predictors of fan aggression: The moderating role of ageing. Sport Management Review, 22(2), 194-208. https://doi.org/10.1016/j.smr.2018.02.002 [ Links ]

Turğut, M., Yaşar, O. M., Sunay, H., Özgen, C., & Beşler, H. K. (2018). Evaluating Aggression Levels of Sport Spectators. European Journal of Physical Education and Sport Science, 4(3). https://doi.org/10.5281/zenodo.1196665 [ Links ]

Vitoratou, S., Ntzoufras, I., Smyrnis, N., & Stefanis, N. C. (2009). Factorial composition of the Aggression Questionnaire: a multi-sample study in Greek adults. Psychiatry Research, 168(1), 32-39. https://doi.org/10.1016/j.psychres.2008.01.016 [ Links ]

Wachelke, J. F., Andrade, A. L. D., Tavares, L., & Neves, J. R. (2008). Mensuração da identificação com times de futebol: evidências de validade fatorial e consistência interna de duas escalas. Arquivos Brasileiros de Psicologia, 60(1), 96-111. http://www.redalyc.org/pdf/2290/229017544009.pdfLinks ]

Wann, D. L. (1994). The “noble” sports fan: The relationships between team identification, self-esteem, and aggression. Perceptual and Motor Skills, 78(3), 864-866. https://doi.org/10.1177/00315125940780 [ Links ]

Wann, D. L., Carlson, J. D., & Schrader, M. P. (1999). The impact of team identification on the hostile and instrumental verbal aggression of sport spectators. Journal of Social Behavior and Personality, 14(2), 279-286. https://home.csulb.edu/~jmiles/psy100/wann.pdfLinks ]

Wann, D. L., Fahl, C. L., Erdmann, J. B., & Littleton, J. D. (1999). Relationship between identification with the role of sport fan and trait aggression. Perceptual and Motor Skills, 88(3_suppl), 1296-1298. https://doi.org/10.2466/pms.1999.88.3c.1296 [ Links ]

Yalcin, İ., Ekinci, N. E., & Ayhan, C. (2021). The effect of hopelessness on violence tendency: Turkish football fans. Physical Culture and Sport Studies and Research, 91(1), 13-20. https://doi.org/10.2478/pcssr-2021-0015 [ Links ]

Zeferino, G. G., Silva, M. A., & Alvarenga, M. A. S. (2021). Associations between sociodemographic and behavioural variables, fanaticism and aggressiveness of soccer fans. Ciencias Psicológicas, 15(2), e-2390. https://doi.org/10.22235/cp.v15i2.2390 [ Links ]

Data availability: The dataset supporting the results of this study is not available

How to cite: Zeferino, G. G., Santos da Silva, A. C., & Mansur-Alves, M. (2024). Instruments for Evaluation of Aggressiveness in Sports Fans: A Systematic Review. Ciencias Psicológicas, 18(1), e-3178. https://doi.org/10.22235/cp.v18i1.3178

Authors’ contribution (CRediT Taxonomy): 1. Conceptualization; 2. Data curation; 3. Formal Analysis; 4. Funding acquisition; 5. Investigation; 6. Methodology; 7. Project administration; 8. Resources; 9. Software; 10. Supervision; 11. Validation; 12. Visualization; 13. Writing: original draft; 14. Writing: review & editing. G.G.Z. has contribuited in 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14; A. C. S. S. in 2, 3, 4, 5, 6, 8, 9, 12, 13; M. M. A. in 3, 4, 5, 6, 7, 10, 11, 12, 13, 14.

Scientific editor in-charge: Dra. Cecilia Cracco

Received: January 12, 2023; Accepted: December 21, 2023

Creative Commons License Este é um artigo publicado em acesso aberto sob uma licença Creative Commons