Intraclass Correlation Coefficient (ICC)

With regard to evidence of reliability, one of the most widely used methods is internal consistency (^{Cascaes Da Silva et al., 2015}; ^{Ledesma, Molina, & Valero Mora, 2002}). Among the coefficients with which this method works, the use of the Alpha coefficient stands out (^{Livia & Ortiz, 2014}; ^{Muñiz, 2010}). This has received criticism due to non-compliance with the assumptions required for its application (^{Domínguez & Merino, 2015}; ^{Ventura-León, 2018}), such as the tau-equivalent assumption which is required to estimate alpha coefficients by dimensions (^{Raykov, 1997}). Therefore, specialized bibliography suggests the use of other coefficients such as the Omega (^{Ventura-León, 2017}; ^{Viladrich, Angulo-Brunet & Doval, 2017}) or the composite reliability coefficient (^{Hair, Anderson, Tatham & Black, 2010}) which yield less biased estimates.

However, there are other procedures for demonstrating the reliability of an instrument. For example, temporal stability-less popular than internal consistency, but no less important. This method refers to the agreement of the score at two different points in time (^{Muñiz, 2010}; ²⁰¹⁸). This procedure is also known as test-retest. Applications of the procedure usually resort to the calculation of Pearson's product-moment correlation coefficient (r), with which it is possible to verify the relation between the two measurements; however, this value is generally overestimated (^{Spence-Laschinger, 1992}) due to the linear nature of the coefficient (^{Shrout & Fleiss, 1979}).

The use of this coefficient implies an important limitation because, if an instrument systematically measures moments that are different from one another, the correlation may be perfect even though the agreement is null (^{Pita & Pértegas, 2004}). For this reason, the use of Pearson's coefficient can constitute a source of error in measurement, since it omits the intra and intersubject variability in the calculation (^{Shrout & Fleiss, 1979}), therefore, exposing researchers to systematic errors in their interpretations (^{Bartko, 1994}; ^{Ledesma et al., 2002}).

To resolve this, the generalizability theory (GT) offers a profound theoretical development about reliability, defining it as the proportion of the variance of an observed score that is not attributable to errors in measurement (^{Spence-Laschinger, 1992}). This encourages the specification and estimation of the true score variance, error score variance and observed score variance components, and the calculation of coefficients based on these estimates (^{Mandeville, 2005}; ^{Pita & Pértegas, 2004}). From this approach, it is suggested to consider the use of the ICC to determine the agreement between two measurements taken in a time interval (^{Esquivel et al., 2006}; ^{Koo & Li, 2016}; ^{Mandeville, 2005}; ^{Shrout & Fleiss, 1979}; ^{Weir, 2005}). Unlike other coefficients, the ICC allows for the detection of systematic measurement bias (^{Esquivel et al., 2006}), in addition to verifying the temporal stability of the scores (^{Martínez, 2005}; ^{Muñiz, 2018}).

At this point, it is necessary to review the complexity of the definition of reliability, since it contemplates the variance ratio between the true score with respect to the total score variance (^{AERA, APA, NCME, 2018}). This definition is important when the objective of the study has to do with determining internal consistency (^{Vargha, 1997}). However, when the aim is to measure the agreement of the scores of a measurement instrument at two moments in time on an unaltered sample, scientific literature does not suggest a specific procedure (^{Muñiz, 2018}), and the main reason involves the measurement scale, with regard to the temporal stability of continuous measures (^{Benavente, 2009}; ^{Mandeville, 2005}).

In this framework, the calculation of reliability through temporal stability (test-retest) is not a commonly used procedure (^{Camacho-Sandoval, 2008}; ^{Pita & Pértegas, 2004}; ^{Prieto, Lamarca & Casado, 1998}), but this does not mean that its estimation is irrelevant. It responds rather to aspects of convenience, given that the test-retest method aims at verifying that the variability of the scores does not differ significantly from one another (^{Weir, 2005}). However, when the assigned scores differ consistently between each observation, it is necessary to resort to more sophisticated calculation methods that allow reducing the measurement error. One of the suggested procedures is the calculation of correlation coefficients resulting from the residuals of a repeated measures ANOVA (^{Cerda & Villarroel, 2008}; ^{Koo & Li, 2016}; ^{Shieh, 2016}).

The ICC was originally developed by ^{Fisher (1954}) as a modification of Pearson's correlation coefficient. Thus, the ICC applies now is calculated from the mean squares, resulting from a repeated measures analysis of variance, and is widely used in other disciplines (^{Cortés, Rubio & Gaitán, 2010}; ^{Koo & Li, 2016}) to assess the validity and reliability of measurement instruments. f4

Where:

The most important aspects to recommend the use of the ICC in psychology research involves the fact that it takes into account the measurement error, which is necessary to be able to control bias (^{Camacho-Sandoval, 2008}), and intra and intersubject variability (^{Hazra & Gogtay, 2016}). This shows its benefits compared to coefficients such as Pearson’s or Spearman’s (^{Esquivel et al., 2006}). To this end, ^{Abad Olea, Ponsoda and García (2011}) point out that by breaking down the variability of the data, according to the sources of error, the corresponding variance components are estimated. These elements refer to an estimate of the variability attributed to the subjects, items and the residual. Therefore, the calculation of the ICC constitutes a more accurate and less biased estimate. Likewise, in terms of variance components, the ICC is obtained as follows: f5

According to ^{Shrout and Fleiss (1979}), the ICC expresses single quantities of the relative magnitude of the two variance components of a score. As the proportion of error variance of total variance in a set of scores decreases, the possible values of the ICC range from 0 to 1 (^{Manterola et al., 2018}; ^{Müller & Büttner, 1994}), wherein a large proportion of error variance in a set of scores produces a low ICC coefficient and indicates low reliability (^{Turner & Carlson, 2003}). They also point out that the minimum acceptable value for the ICC is 0.75 (^{Haggard, 1958}; ^{Shrout & Fleiss, 1979}). In this regard, ^{Prieto et al. (1998}) modified the calculation of the ICC based on the variability of the observed scores: The more homogeneous the study sample, the lower the ICC tends to be.

According to the GT, an approximation to the measurement of the error variance can be obtained by breaking down the variability of the data from each source of variation. This way, the elements of the variance (variability attributed to the subject, to the items and to the measurement error) are estimated. The application of ANOVA allows to make these estimates. To do so, it is necessary to define the number of levels of the intrasubject variable (number of measurements carried out in a period of time). Among the results, we select the sums of squares (SS), degrees of freedom (df) and quadratic means (QM), with which it is possible to calculate the ICC. f6

Where:

Accordingly, the convenience and advantages of the ICC in relation to other correlation coefficients (concordance) has been shown. Next, an application of the ICC shall be presented with the objective of determining the temporal stability of the scores of the Interpersonal Reactivity Index (IRI) in a sample of university students from Lima. The traditional procedure that measures the concordance of the measures, using Pearson’s correlation coefficient is compared with the procedure suggested through the ICC that comes from a repeated measures ANOVA.

Method

Results

Table 1 shows the descriptive measures for PT, FS, EC, and PD for two measurements reported with a three-week margin. The results show that the PT averages show little variation (M1 = 20.710 and M2 = 20.120), the FS averages show a similar behavior (M1 = 18.900 and M2 = 17.760); as for EC the measures are quite similar (M1 = 25.370 and M2 = 23.220). Likewise, the PD averages show the same state (M1 = 15.020 and M2 = 16.170). Finally, the skewness and kurtosis coefficients are below 1.5 which suggests that the variables present univariate normality.

Table 1: Descriptive statistics 

^Notes: M: Mean; SD: Standard deviation; g1: Skewness coefficient; g2: Kurtosis coefficient.

Analysis of variances

Table 2 shows the results of the repeated measures ANOVA for two factors. The results of the FS dimension show that, at the intrasubject level, no statistically significant differences were found and that the effect size is non-existent (F = 0.531; p > .05; ηp ² =0.013). However, in the intersubject effect test, the variations are statistically significant and the magnitude of the differences is large (F = 1327.275; p < .001; ηp ² = 0.971). Regarding the FS dimension, the intrasubject effect test shows that there are no statistically significant differences and the effect size is not significant (F = 2.832; p > .05; ηp ² = 0.066). Whereas the intersubject effect test indicates that the individual-group variations are statistically significant and the magnitude of these is large (F = 928.659; p < .001; ηp ² = 0.959). The results in EC indicate that there are no statistically significant intrasubject differences, reaching a very small effect size (F = 9.156; p > .05; ηp ² = 0.186). Meanwhile, statistically significant differences were found at the intersubject level, being the magnitude of these differences large (F = 1327.275; p < .001; ηp ² = 0.973). Finally, in the PD dimension, the results at the intrasubject level reflect that there are no statistically significant differences and that the effect size is not significant (F = 3.800; p > .05; ηp ² = 0.087). Whereas the intersubject effect test indicates that the variations are statistically significant and that the magnitude of these is large (F = 729.928; p < .001; ηp ² = 0.948).

Table 2: Intra and intersubject effect test 

^Notes: QM: Quadratic Mean; F: Repeated Measures ANOVA Test Statistic; p: Statistical Significance; ηp²: Partial Eta Squared. Intrasubject Effect Test: It assesses the variability of the same measures among people. Intersubject Effect Test: It assesses the variability between the same measures among people.

Temporal Stability of the Measure

From the repeated measures ANOVA procedure, the QM: Quadratic mean of the scores and the MSE: Sum of one-way mean squared errors, which are necessary elements for the calculation of the ICC, with their respective confidence intervals at 95 %, were obtained . Likewise, the Pearson's product-moment correlation coefficients (r) are presented with the respective statistical significance (Table 3). The ICC - r coefficients are compared. From them, the delta between these coefficients was calculated, obtaining changes above 0.001.

Table 3: Comparison between Pearson's product-moment and ICC Coefficients 

^Notes: Δ ICC - r: Change between the coefficients; *p < .05; **p < .001.

Discussion

The ICC is an agreement index for continuous data; it assesses the size of the variance components between and within groups (^{Davis & Joseph, 2016}; ^{Shoukri, 2004}). It also describes the proportion of the total variance which is explained by differences between scores and instruments (^{Mandeville, 2005}). According to ^{Hazra & Gogtay (2016}), the ICC is developed within the analysis of variance and its calculation is based on the true (between subjects) variance and the measurement error variance, produced during the repeated measurement (^{Hazra & Gogtay, 2016}; ^{Manterola et al., 2018}).

In this sense, the purpose of this research was to carry out a theoretical review of the applicability of the ICC to estimate the temporal stability of the scores of the measurement instruments. For this purpose, a longitudinal study of two measurements of IRI scores was conducted. These were then analyzed from a traditional perspective by means of a bivariate analysis with Pearson's correlation coefficient. Meanwhile, in the second approach, the analysis comprises a repeated measures analysis of variance (ANOVA).

It is worth mentioning that the evidence of reliability by the temporal stability method (test-retest) has already been used in the psychometric analysis of the IRI. For example, it has been used in studies in Spanish (^{Carrasco, Delgado, Barbero, Holgado & Del Barrio, 2011}), Belgian (^{De Corte et al., 2007}) and Chilean (^{Fernández, Dufey & Kramp, 2011}) populations in which cases a test-retest correlation between moderate and high was found.

On the other hand, the Pearson product-moment correlation coefficients indicate that there is a relationship between these scores. However, this does not indicate that there is concordance between the measures, which has already been quite discussed in the bibliography (^{Davis & Joseph, 2016}; ^{Koo & Li, 2016}; ^{Shoukri, 2004}). Moreover, as it is a linear calculation procedure, the interpretations are biased and there is a risk of overestimation (^{Hazra & Gogtay, 2016}; ^{Manterola et al., 2018}). In turn, the repeated measures ANOVA provides the inputs for the calculation of the ICC which, due to its non-linear nature, constitutes an adjusted measure of concordance between measurements. As a result, it was identified that the four dimensions of the IRI (PT, EC, FS and PD) do not present a major difference in the scores within the group (intrasubject), showing non-significant differences with non-existent effect magnitudes. However, when analyzing the variations between groups, it could be seen that there were statistically significant differences, with large effect sizes.

Thus, it was possible to corroborate the practical usefulness of the calculation of the ICC because it not only provides information about the relationship between the two measures, but also provides information about the fulfillment of the assumptions of no intra and intergroup variations. These allow the estimation of the measurement error (^{Pita & Pértegas, 2004}).

Likewise, when comparing the Pearson and ICC coefficients, it could be seen that the former are slightly higher. Furthermore, they are interpreted as significant and very significant correlations, but this does not imply that the variances have been analyzed, and, therefore, the concordance itself is not being assessed. What this coefficient expresses is the product-moment relationship between two measurements, disregarding the inter and intrasubject variation (^{Shoukri, 2004}; ^{Shrout & Fleiss, 1979}).

Additionally, to evaluate whether the changes between the correlation coefficients were significant, the differentials (Δ) were calculated and ^{Byrne’s criterion (2008}) was considered to determine the measurement invariance. It can be seen that, with the exception of the PT dimension, these differences are significant in the remaining dimensions, which is evidence of the overestimation that usually occurs when using Pearson's correlation coefficient as a concordance statistic.

Regarding the estimation method used, it is important to emphasize that the test-retest procedure has been previously used in other studies. Such is the case of the research by ^{Carrasco et al. (2011}) where the temporal stability of the IRI was analyzed in a sample of Spanish adolescents in which Pearson’s product-moment correlations, which range from 0.44 to 0.65, after a one-year interval, were reported. Similarly, ^{Fernández et al. (2011}) reported that Pearson’s product-moment correlations higher than 0.70 were found after a 60-day interval in Chilean university students. These studies indicate that the examined construct is not subject to random fluctuations (^{Reidl-Martinez, 2013}); on the contrary, it seems to be quite stable over time. Nevertheless, despite the fact that the time intervals used in these precedents are different from those of this research, it is necessary to emphasize that these have been established, in accordance with the criteria suggested by the bibliography (^{Martínez, 2005}). The foregoing is indicated as a reference to highlight that the findings of the research do not respond to an anomalous behavior of the construct, nor to some other resulting aspect, typical of the task performed (^{Medrano & Pérez, 2019}).

An important aspect is related to the applicability of the procedure for calculating the ICC since this is not only limited to the estimation of the temporal stability of the scores of an instrument, therefore, being possible to use it in quasi-experimental studies (more than one measurement). In these designs, the related t or the Wilcoxon rank-sum are commonly used. These are estimates that only express the specific difference between before-after and not the intra and intersubject variation as a product of the effect of a factor (intervention program) (^{Abad et al., 2011}).

An important limitation has to do with the sample size and the type of sampling which restricts the generalizability of the results. However, given that in this case the aim is to expose the analysis technique, the sample size does not affect this. It is also necessary to demonstrate the applicability of the ICC in other procedures such as validation by expert judgement in which case the demonstration to yield more precise estimates than other coefficients would be expected, such as Aiken’s V.

Finally, it is important to emphasize that recent psychometric studies include within their reliability measures the test-retest procedure or the temporal stability of the measure (^{Correa-Rojas, Grimaldo & Del Rosario-Gontaruk, 2020}; ^{Lascurain, Lavandera & Manzanares, 2017}). This as a complement to internal consistency which is necessary, especially, if it is intended to use these measures in longitudinal studies (^{Abad et al., 2011}; ^{Muñiz, 2018}) to ensure that they do not constitute a source of systematic error.