Study 1 - Validity Evidence based on Test Content

Participants

Three psychologists, of both sexes, with undergraduate, master's and doctoral degrees in psychology, participated in the analysis of the judges. Participants had an average of 18 years of professional experience, with knowledge and experience in research in the areas of school psychology and psychological assessment, as well as in the development of instruments. Twenty adolescents, both male and female, with ages ranging from 12 to 13 years (M= 12.15; SD= 0.36) participated in the pilot study. They were students of the 7th grade of a public school in the State of Ceará, Brazil. Students who had severe cognitive impairments and /or some disorder that prevented them from participating in the research were excluded from the sample. Information on which students might fit the sample exclusion criteria was obtained from the school coordinator. Upon consultation, it was found that the school did not have any students who met the research exclusion criteria.

Instruments

Assessment Form for Independent Judges. The form sent to the three expert judges included personal identification and education background questions. In addition, there was a brief description of the research and its objectives, as well as the construct - School SWB. Then, there were instructions on how to proceed with the assessment of the items, which were analyzed for language clarity, theoretical relevance, practical pertinence, and theoretical dimension.

School Subjective Well-Being Scale - initial version. The aim of the instrument is to assess student satisfaction with school and the frequency of positive and negative affect experienced by students in the school context. It consisted of two subscales. The first subscale contained 10 initial items aimed at evaluating the School Satisfaction factor (cognitive aspect of School SWB), assessing the positive and the negative pole of the construct, with a five-point Likert scale answer key, ranging from 1 (strongly disagree) to 5 (strongly agree). Item examples: "I am satisfied with my school"; "My school is excellent". The second subscale, Positive and Negative Affect experienced by students in the school environment, was composed of 60 initial items, 30 regarding positive affect and 30 regarding negative affect. The items consisted of adjectives, with a five-point Likert scale answer key, ranging from 1 (never) to 5 (always). Item examples: "Confident", "Motivated", "Tired" and "Upset". In order to assess item understanding by the target population, the following questions were inserted: "Was there any word you didn't know?"; "Was any question (item) unclear to you, and if so, which one was it (Indicate question number)?"; "Were the instructions for completing the test clear to you?"; "Do you have any suggestions regarding the questions or the test in general?".

Procedures

Initially, authorization to conduct the instrument was requested to the board of a public school in the State of Ceará, Brazil. Then, the research project was submitted for consideration by a Research Ethics Committee of a Brazilian university, and approved under CAAE number: 97953218.2.0000.5514. After approval, an email was sent to the expert judges inviting them to participate in the research and assess the instrument's items. Their consent to participate in the study was given by signing an Informed Consent Form. (Termo de Consentimento Livre e Esclarecido - TCLE) and an access link to the item assessment protocol was sent to the expert researchers.

After the analysis of the judges, in the pilot study stage, a Free and Informed Consent (TCLE) was signed by one of participant's parents or legal guardians, in addition to a specific Free and Informed Consent Form for minors (Termo de Assentimento Livre e Esclarecido - TALE) signed by students themselves, in accordance with the ethical aspects required for research involving human beings. The instrument was conducted in person and collectively, at a time previously scheduled with the school, and included one student group. The SSWBS version used in the pilot study was obtained from the analysis of the judges, in which items were excluded for not meeting the established criteria of content validity. During the session, students were given the necessary guidance as to how the instrument should be completed. The time allowed to conduct the pilot version of the scale was 30 minutes. After participants completed the instrument, the researcher conducted a survey on potential doubts arising from the response process.

Data Analysis

Judges' scores and responses to the Assessment Form for Independent Judges were stored on an electronic spreadsheet and analyzed according to the Content Validity Coefficient (CVC; ^{Hernández-Nieto, 2002}). Items with a CVC equal to or greater than 0.80 were kept.

In the pilot study, descriptive statistics were used to characterize the sample using the statistical software Statistical Package for the Social Sciences (SPSS), v.25. The responses of the 20 students to the SSWBS assessment questionnaire were fully read and qualitatively analyzed, grouped according to content similarity, using the content analysis methodology proposed by ^{Bardin (2011}). The doubts and suggestions mentioned by students while responding the instrument were noted for later assessment.

Results of Study 1

The 70 items developed for the initial version of the SSWBS were analyzed by the judges, and items with a CVC equal to or greater than .80 were kept. For the School Satisfaction factor, three items were excluded: "When my friends talk about their schools, I think their school is better than mine" (.76); "My school is close to an ideal school model" (.36) and "In most respects, I am satisfied with my school" (.78). For Positive Affect in School, seven items were excluded: "accepted" (.76), "relieved" (.63), "well" (.76), "comfortable" (.76), "hopeful" (.76), "strong" (.63) and "optimistic" (.69). For Negative Affect in School, eleven items were excluded: "threatened" (.76), "scared" (.76), "guilty" (.69), "depressed" (.76), "bored" (.76), "humiliated " (.76), "misunderstood" (.76), "hurt" (.69), "frightened" (.69), "disturbed" (.63) and "ashamed" (.76). Therefore, 21 items were excluded from the scale.

The pilot study was conducted with a 49-item version of the SSWBS. Participants did not have any doubts regarding instructions, response format and item comprehension. Only the “enthusiastic” item was marked as difficult to understand, which caused it to be removed from the scale. Thus, after the analysis of the judges and the pilot study, 22 items were removed, resulting in a 48-item version of the SSWBS.

Study 2 - Validity Evidence based on Internal Structure

Participants

The study sample was composed of 434 students from the 7th year of Elementary School (7th grade) to the 2nd year of High School (11th grade), with a predominance of females (n= 240; 55.30%), and ages between 12 and 19 years (M= 14.88; SD= 1.70), regularly enrolled in a public school in the State of Ceará, Brazil. Students who had severe cognitive impairments and/or some disorder that prevented them from participating in the research were excluded from the sample. Upon consultation with the school coordination, it was found that there were no students who met the exclusion criteria of this research in the school.

Instruments

Sociodemographic and School Identification Questionnaire. Instrument developed by the proponents of this research. Composed of questions regarding age, gender, school and grade.

School Subjective Well-Being Scale - initial version. Instrument designed to assess student well-being in the school context. The version used in this stage results from the test content validation study (Study 1). Thus, a version of the SSWBS composed of 48 items distributed in three factors: School Satisfaction (7 items), Negative Affect in School (22 items) and Positive Affect in School (19 items).

Procedure

The participation of students under 18 years of age in the research occurred by submitting a Free and Informed Consent Form signed by a parent or legal guardian and a specific Free and Informed Consent Form for minors signed by the students themselves. Students aged 18 or more participated in the survey by signing an informed consent form. The instrument was conducted in person and collectively, at a time previously scheduled with the school.

Data Analysis

Initially, the SSWBS data matrix was tested for factoring, by calculating the Kaiser-Meyer-Olkin index (KMO) and by Bartlett's Test of Sphericity. KMO values equal to or greater than .70 with sphericity test significance levels p< 0.05 indicate the matrix is factorable, making it possible to use Exploratory Factor Analysis (^{Pasquali, 1999}). Then, parallel analysis, Minimum Average Partial (MAP), and the Hull Method were used to determine the number of factors of the SSWBS (^{Lorenzo-Seva, Timmerman, & Kaers, 2011}; ^{Velicer, 1976}).

After that, the parameters of the factor solution were estimated by Oblimin rotation and the Weighted Least Squares Mean and Variance Adjusted - WLSMV estimator, from the matrix of polychoric correlations between variables. This estimator is considered robust and recommended for categorical and ordinal items, such as Likert scales. Since the SSWBS is intended to be a brief instrument, items of moderate and high discrimination with factor loadings equal to or greater than .40 in at least one of the factors were kept. Then, Exploratory Factor Analysis was performed and the following fit indices were considered: Root-Mean-Square Error of Approximation (RMSEA; reference value < .08), Comparative Fit Index (CFI; reference value > .90) and Tucker-Lewis Index (TLI; reference value > .90; ^{Hair, Black, Babin, Anderson, & Tatham, 2009})

The internal consistency indices of the general scale and for each of its factors were calculated using Cronbach's alpha and McDonald's omega. Internal consistency coefficient values equal to or greater than .70 are considered adequate (^{Campo-Arias & Oviedo, 2008}). Test information curves were also used, as they are more specific than traditional reliability coefficients, allowing the identification of the instrument's reliability for each part of the latent trait. The internal structure of the SSWBS was also analyzed through the Item Response Theory, using the Rasch model (^{Bond & Fox, 2015}) to investigate the difficulty parameters of the items and people. Items with item-person correlations above .30 are considered adequate. Item goodness of fit was also analyzed through Infit and Outfit indices, taking values between .50 and 1.50 as a reference (^{Linacre, 2014}). The analyses of this study were performed using the software Factor Analysis 10.8, Mplus 7.11, RStudio and Winsteps 3.74.

Results of Study 2

Initially, a Kaiser-Meyer-Olkin (KMO) coefficient of .94 was found, which suggests a sufficient level of common variance between items to perform factor analysis. Bartlett's Test of Sphericity, which assesses the general significance of all correlations in a data matrix, showed statistical significance (p< .001). These results indicate the possibility of data factoring. As for the SSWBS factor solution, retention methods were not unanimous, recommending solutions of one (Hull method), three (MAP) and even four factors (Parallel Analysis).

When analyzing these solutions, it was found that 19 items had cross loadings above .30 and one item did not load for any of the factors. Thus, these items were excluded, considering the goal was to develop a brief and high quality psychometric measure. Items with factor loadings equal to or greater than .40 in the expected theoretical dimension were considered (^{Hair et al., 2009}). Therefore, 20 items were excluded, and a 28-item version of the SSWBS was obtained.

Exploratory Factor Analysis was performed on this version to investigate the model's fit indices with one, three and four factors, using RMSEA, CFI and TLI. The one-factor solution showed the poorest fit indices (RMSEA = .13; CFI = .74; TLI = .72). The data showed that the four-factor solution had better indices (RMSEA= .05; CF = .97; TLI= .96) than the three-factor solution (RMSEA= .06; CFI= .95; TLI= .94). However, the interpretability of the values assumed by factor loadings did not allow the identification of a theoretical distribution pattern justifying a fourth factor. Thus, results favored a three-factor model, whose items, loadings and internal consistency coefficients are shown in Table 1.

Table 1: Results of Exploratory Factor Analysis (Oblimin Rotation) and Internal Consistency Coefficients 

As shown in Table 1, the three-factor solution presented low cross loadings (≤|.29|). The interpretability of factor loadings and the distribution pattern of the items in the factor corroborated the tripartite theoretical model of subjective well-being. The first factor explained items related to students' satisfaction with school, i.e., students' cognitive assessment of their school. Factors 2 and 3 explained items related to the positive and negative affect experienced by students in the school environment. Correlations between factors were moderate (r _F1xF2 = (.39; r _F1xF3 = .43 e r _F2xF3 = (.34_), indicating an oblique relationship between variables (^{Damásio, 2012}).

The calculation of internal consistency indices showed Cronbach's alpha coefficients ranging from .82 to .88, and omegas from .87 to .90. Then, the consistency indices for the 28-item version of the SSWBS were calculated, resulting in an alpha of .91 and a McDonald's omega of .93, which indicates good reliability indices for the instrument.

The next step was to investigate more accurately the reliability of the three factors through information curves, using IRT. These analyses showed that the first factor, school satisfaction, presented a higher level of reliability for the interval between (2 and 0 of the theta scale. Negative affect in school, the second SSWBS factor, showed higher levels of reliability for the interval between (1 and (3, with a higher level of information on a broad spectrum of the latent trait, with greater reliability for the population above the population average (standardized as 0). It was found that factor 3, positive affect in school, had greater reliability in the interval between (2 and (1, showing a better functioning of the instrument at the lower pole of the latent trait. The analysis of the information curves shows that the instrument has greater reliability when measuring students with lower levels of school satisfaction and positive affect, and higher levels of negative affect (Figure 1).

Figure 1: nformation Curves of SSWBS Factors - 28 items. 

Then, the Item Response Theory (IRT) model was used through a Rating Scale analysis (see Table 2). All factors were considered one-dimensional, according to the criteria of ^{Linacre (2014}), in which the variance of the first contrast must be lower than the second. The analysis in Table 2 shows that most items are within the expected fit, with infit and outfit values between .50 and 1.50 (Linacre, 2014). Only item 9 (“agitated”) had fit indices outside the expected range.

Table 2: Item Analysis by the Rasch Model for the Three Factors of the SSWBS 

In regards to item-total correlations, item 9 (“agitated”) showed the lowest coefficient (r = .35) and infit and outfit indices outside the reference range, indicating that this item does not represent the assessed construct very well. A new exploratory factor analysis was performed with the 27 remaining items and the scale fit indices improved, with CFI= .96; TLI= .95; RMSEA= .06. As for the other items, correlations ranged from .50 to .81, interpreted as moderate to high. The internal consistency of the items was considered adequate, with a lower value of .96 (F3) and a higher value of .99 (F1 and F2). In regards to people, coefficients were .78 (F1), .84 (F2) and .83 (F3), which were also considered adequate. With regard to item difficulty, item 6 of school satisfaction (“My school is excellent”), item 13 of negative affect in school (“furious”), and item 22 of positive affect in school (“understood”) were the least endorsed. Item 2 of SS ("my school is bad"), item 11 of NAS ("tired"), and item 26 of PAS ("integrated") were the most easily endorsed.