SciELO - Scientific Electronic Library Online

 
vol.7 número1PresentaciónFactores explicativos de la eficiencia educativa en Uruguay: evidencia a partir de PISA 2012 índice de autoresíndice de materiabúsqueda de artículos
Home Pagelista alfabética de revistas  

Servicios Personalizados

Revista

Articulo

Links relacionados

Compartir


Páginas de Educación

versión On-line ISSN 1688-7468

Pág. Educ. vol.7 no.1 Montevideo mayo 2014

 

 

ADVANCING ACADEMIC OPPORTUNITIES FOR DISADVANTAGED YOUTH: THIRD YEAR IMPACT EVALUATION OF A PRIVATELY-MANAGED SCHOOL IN A POOR NEIGHBORHOOD IN MONTEVIDEO

 

Oportunidades académicas de calidad para jóvenes en desventaja: tres años de evaluación de impacto de un centro educativo autogestionado en un barrio de Montevideo de contexto crítico

 

Ana Inés Balsa* y Alejandro Cid**

 

ABSTRACT. We study the three-year impact of a private tuition-free middle school on the academic outcomes of poor students. Several features of the treatment school fit with innovative paradigms that have delivered successful outcomes in poor urban areas. Our research design exploits the excess of applicants over the school capacity and the fact that participants were selected randomly. Specifically, we follow a cohort of students that entered middle school in 2010 and that were randomly assigned to attend the treatment school or public school as usual. We find that the treatment school impacted favorably on students’ academic advancement and math competencies. Also, the treatment school had a positive —and quite robust over time— impact on students’ and their parents’ academic expectations. This culture of high expectations has been previously identified in the literature as a key input for school success.

 

Keywords: Randomized design; Private school; Low-income population; High Expectations

 

Resumen. Durante tres años evaluamos el impacto, sobre los resultados académicos, -de un centro de educación secundaria, autogestionado y gratuito, dirigido a adolescentes sin recursos económicos. Varias características de este centro educativo se ajustan a paradigmas innovadores que han mostrado resultados exitosos en zonas urbanas pobres de otras regiones. Hemos diseñado nuestra investigación de modo de aprovechar la existencia de exceso de solicitudes de admisión respecto a la capacidad del centro educativo en cuestión y al hecho de que los admitidos fueron seleccionados al azar. En concreto, seguimos una cohorte de estudiantes que ingresaron a la educación secundaria en 2010 y que fueron asignados aleatoriamente para asistir al centro educativo en cuestión (grupo de tratamiento) o a los liceos públicos de la zona (grupo de control). Encontramos que el centro educativo objeto de estudio impactó favorablemente en indicadores de avance en educación secundaria y en las competencias académicas (matemáticas) de los estudiantes. Además, el centro educativo de tratamiento tuvo un efecto positivo y robusto sobre las expectativas académicas de los jóvenes y de sus padres, en comparación al grupo de control. Esta cultura de altas expectativas que se observó en el centro educativo ha sido identificada en la literatura científica previa como un insumo clave para el éxito académico.

 

Palabras clave: diseño experimental, evaluación mediante aleatorización, centros educativos autogestionados, población de bajos ingresos expectativas académicas

 

Recibido el 1 de julio de 2014

Aceptado el 20 de agosto de 2014

 

INTRODUCTION

 

In the past few years, new schooling initiatives have been appearing in Uruguay in response to poor levels of academic achievement, and high repetition and drop-out rates among socioeconomically vulnerable students. These new modalities have taken the form of privately managed middle schools that offer full-time, free-of-charge formal education to poor adolescents, financially supported by corporate and individual donations. Most of them rely on extended academic time, strict discipline, a sense of school belonging, high academic expectations, and the involvement of the family and the community.

Using a randomized trial, we evaluate the impact of one of these innovative privately-managed schools, Liceo Jubilar, on students’ academic expectations and educational outcomes. The research exploits the excess of applicants over the school’s capacity and the fact that participants are selected randomly. The study tracks and compares the three-year trajectories of adolescents selected to enter the school in 2010 against those of students not drawn in the lottery, who entered traditional public schools.

The treatment school has limited independence to innovate over academic contents, and does not differ with public schools in the observable quality or remuneration of the teachers. However, it operates over an extended academic schedule, has freedom to selectively hire personnel, shows a strong involvement of the family, and offers a climate of discipline and belonging. In Ana Inés Balsa and Alejandro Cid we found positive effects of the treatment school on academic expectations and rates of academic promotion one year after the initiation of the intervention. In this paper, we assess the school’s middle-run impact by studying students’ outcomes after the third and final year of the intervention. As before, we find positive impacts on students’ promotion levels. We also find suggestive evidence that students in the treatment school achieved better math outcomes than comparative students. Furthermore, results confirm the positive and sustained impact of the intervention on students’ and parents’ academic expectations, even several months after having left the treatment school. Our findings underscore new approaches to education that may contribute to foster a culture of high expectations and improved opportunities for disadvantaged adolescents.

 

Background and significance

 

Among a variety of school strategies and outcomes, Stewart Purkey and Marshall S. Smith (1983), and Pam Sammons, Josh Hillman, and Peter Mortimore (1995), go beyond the traditionally collected input measures — class size, per pupil expenditure, the fraction of teachers with no certification, and the fraction of teachers with an advanced degree — and argue that successful schools have organizational structures that empower school leaders, develop human capital, reach out to parents, create a positive school culture, and maximize learning time.

The capacity for innovation and exploration of new pedagogical approaches, a greater involvement of parents and families, community participation through financial support and volunteerism, and stronger pressure to achieve goals and be accountable to the community have also been identified as major drivers of success and satisfaction with the school (Berends, Cannata, Goldring, and Preston; Bifulco and Ladd; Bierlein, Finn, Manno, and Vanourek). In the same lines, Will Dobbie and Roland G. Fryer highlight five effective schooling policies suggested to promote academic success by over forty years of qualitative research. These policies are frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and a culture of high expectations.

The school under analysis has introduced some of these effective policies in its program, representing a huge change in the traditional educational paradigm found in most Uruguayan public schools. This study provides evidence on the school’s academic effectiveness and seeks to offer new insights regarding the strategies employed by the school to boost students’ academic achievements.

Liceo Jubilar is one of the few tuition-free privately managed schools in Uruguay. It is located in Casavalle, one of the poorest neighborhoods in Montevideo (a neighborhood that showed a repetition rate of 26% and a school dropout rate of 60% at the start of the impact evaluation in 2010). Liceo Jubilar offers middle school education (1st, 2nd, and 3rd grades of secondary education) to 210 students. Unlike traditional middle schools in the country, Liceo Jubilar is a full time school. Students are taught the national school curriculum in the mornings, and are required to take courses beyond the national curriculum and to choose among several educational and recreational workshops in the afternoons. Students spend an average of 9 hours per day at school and the school-year is 44 weeks long, 6 weeks longer than the traditional-school year. The teaching-learning approach is highly personalized, based on a close interaction with families and on a strict discipline. Students are followed by a team of psychologist and specialized teachers, and are provided tutoring or learning difficulties support on an individual or group basis as needed. Outings, camps and weekly encounters held by the pastoral department contribute to the understanding of each student’s environment and to the creation of personal bonds with the students. In addition, the school holds frequent interviews, meetings and workshops with parents, trying to involve the family in the student’s learning process.

The school cannot choose its own academic curriculum; it has to abide by the rules and contents of the national curriculum designed by the Uruguayan Education Authorities. However, and unlike public schools in the country, it can selectively hire and dismiss teachers, and can assign teachers’ workload flexibly to teaching, coordination, and training.

The school shows many of the features identified in previous literature as key inputs for education: increased instructional time, high-dosage tutoring, a positive school culture, families’ involvement, and a culture of high expectations (for a more extensive description of the school (Balsa and Cid).

 

METHODOLOGY

 

Average dropout and repetition rates are lower in Liceo Jubilar than in the neighborhood's traditional school system. This simple comparison of means captures not only Liceo Jubilar's treatment effect, but also differences in the baseline characteristics of the populations compared (selection bias). For example, students who apply to Liceo Jubilar are probably better than other youth in terms of their motivation, perception of the value of education, and family support. These latter features could bias the impact estimates upwards if selection bias were not adequately addressed. While some of the variables that characterize each group can be observed with relative ease (i.e. socioeconomic background, family structure, family education and occupation), other characteristics such as parental commitment towards education or student's motivation are more difficult to observe. In this sense, the adjusted comparison of means based on regression or propensity score analysis does not completely solve the problem of selection bias.

To avoid this problem, our impact assessment is based on the randomization of a cohort of children who applied to enter Liceo Jubilar by the end of sixth grade in 2009. The research exploits the excess of applicants over the school capacity and the fact that participants were selected randomly. This allocation rule ensures that the group of students entering Liceo Jubilar —the treatment group— is similar at baseline to the group of adolescents who are not drawn in the lottery —control group. The cohort under evaluation (N = 101) was interviewed in October 2009 and randomized in December, three months before starting the school year. The current paper reports the third year follow-up results for this cohort.

 

Data collection

 

In September 2009 Liceo Jubilar opened an enrollment window inviting families of children in the last year of primary school to apply for a placement at the middle-school. The school had 70 places available (corresponding to two classes of 35 students each). Applications were received from 172 students, of whom 43 were rejected because they exceeded the grade-appropriate age by 2 years or more, did not live in the neighborhood, or had a household income above the poverty threshold. Out of the remaining 129 applications, 28 students were automatically chosen to enter the school, majorly because they were siblings of current or former students. This left a waiting list of 101 candidates who were randomly assigned to meet the remaining quota of 42 places in December 2009.

Before drawing the lottery, the research team at Universidad of Montevideo surveyed the applicants. These baseline surveys were administered at Liceo Jubilar in November 2009. The questionnaire inquired about demographics, academic performance, academic expectations, risky behaviors, and habits. An additional survey was administered by the school staff to parents or family referents with questions about family structure, education, income, and occupation, among other socioeconomic characteristics.

Randomization was executed to achieve balance in gender, two categories of household income (high and low), and two categories of achievement in Liceo Jubilar’s baseline placement test. Most of the students not selected to enter the treatment school ended up attending a public school. We found no statistically significant differences in base line characteristics between subjects selected by lottery to enter Liceo Jubilar in March 2010 (treatment group) and applicants who were not drafted (control group), confirming that the selection process had been in effect random (Balsa and Cid).

A third-year follow-up was conducted between December 2012 and August 2013. In December 2012, treatment and control subjects were asked to take a math standardized test. The test was developed, administered, and graded by the Institute on Educational Assessment of the Uruguayan Catholic University. It was designed to assess the content of the formal education curriculum in Uruguay and to evaluate three key competencies underscored in the PISA1 approach: reflectiveness, reproduction, and connection (PISA). The content of the math test was unknown to school teachers and staff. It was administered by external applicators in the school premises in the case of treatment subjects, and at University of Montevideo in the case of the control individuals. In addition, during July-August 2013, research subjects were interviewed at home. The survey included interviewer-administered questions about academic achievement, perceptions about school, use of time, values, life satisfaction, expectations, and health status, plus a self-administered questionnaire with sensitive questions on crime and delinquency, substance use, and sexual behavior. In addition, parents were asked to respond a questionnaire regarding their socio-demographic characteristics and their beliefs about their child’s school, and to fill-in a psychometric scale that inquired about the child’s behavior.

This paper focuses on the school’s three year impact on the following academic outcomes: dropout and promotion rates, academic expectations, and standardized tests results. The simplest way of estimating the average treatment effect is by conducting a regression of each outcome on the coefficient of the treatment dummy, i.e. a dichotomous variable that takes the value of 1 if the adolescent attended Liceo Jubilar and 0 otherwise. However, at the moment of the third-year-follow-up, two of the participants initially selected to enter the treatment school were not attending the school and three subjects from the control group had managed to enter the school. Thus, the group of those that were finally treated differs slightly from those initially selected to be treated (the intention to treat group). In this context, a simple Ordinary Least Squares (OLS) regression like the one specified above may introduce bias in the impact estimates if selection into and out of the treatment group is not random. To avoid this problem, we use the initial status that resulted from the randomization, which we refer to as the intention to treat status (ITT), as the relevant explanatory variable. For robustness, we also use the ITT as an instrument for effective participation and estimate the effects using instrumental variables.

 

Sample size and attrition

 

The initial cohort of students selected for this study consisted of 100 participants, 42 in the group randomly selected to enter Liceo Jubilar in 2010 and 58 in the control group.

2 In the third-year follow up, we were able to obtain data on drop-out rates, promotion rates, and academic expectations for 40 students in the treatment group and 48 students in the control group. Our identification strategy remains valid as long as this attrition is unbiased. We assess this assumption by comparing pre-enrollment characteristics by ITT status in the subsample responding to the 3-year follow-up survey. This comparison is depicted in Table 1.

 

Table 1 shows no statistically significant differences in baseline characteristics by ITT status for most pre-treatment measures analyzed. The only exception is a slightly higher probability of having more than 10 books at home for subjects in the ITT group. To dismiss any concerns about selection, we conduct robustness tests that control for this characteristic in the outcomes regressions.

Unfortunately, take up rates in the math standardized tests were lower than in the home interview. The response rate was 62% in the treatment group and 48% in the control group. To identify potential biases in attrition, we again compared pre-treatment characteristics for examined students who had been randomly selected to participate in the treatment school and examined students who had not been drafted for the treatment school (see Appendix Table A1). Although the majority of baseline variables showed no statistical difference across both groups, there is some evidence of unbalanced attrition in favor of the treatment group. Those taking the test in the treatment group were more likely to show good or excellent grades at baseline, were less likely to have repeated a grade in primary school, were more likely to be catholic, and were more likely to have more than 10 books at home. We come back to this issue in the next subsection.

 

Impact evaluation

 

The analysis in this paper compares third year academic outcomes across treated and control subjects using ordinary least squares regression (OLS). We are interested in the academic standing of students three years after having initiated middle school, which coincides with the end of treatment in the school under analysis. The cohort being analyzed enrolled in middle school (1st year of secondary school in Uruguay) in March 2010 and was expected to graduate from 3rd grade of secondary school in December 2012. We measure academic standing during the first months of the 2013 academic year, when treatment students had already left the Liceo Jubilar. The outcomes of interest are: (1) the likelihood of attending the age-corresponding grade in 2013 (4th grade of secondary school), (2) the likelihood of having repeated at least one grade between 2010 and 2012, (3) the likelihood of having dropped-out of school by the beginning of the 2013 academic year, (4) expectations about college completion as of 2013, and (5) the results of the standardized math test administered in 2012.

Due to the existence of non-compliers, we employ the indicator of random selection into treatment (ITT) as the relevant explanatory variable. In a robustness check, we use the ITT indicator as an instrument for effective participation and analyze the data using two stages least squares estimation. Because random assignment balances characteristics across treatment types, and attrition does not appear to have affected this balance when assessing outcomes (1)-(4), the core regressions explaining these outcomes do not control for other covariates.3 Using controls could help reduce the residual variance and improve the precision of the estimates if these controls are predictive of the outcomes under analysis. Unfortunately, potentially relevant controls had one or more missing observations, so we chose to run uncontrolled regressions and avoid further sample loss rather than improve precision. To account for the multiplicity of outcomes, we used the family-wise Holm-Bonferroni adjustment of p-values. All standard errors were estimated using hetero scedasticity-robust specifications.

In the case of the math results, we mentioned that attrition resulted in an unbalance of pre-treatment characteristics across test takers in the treatment and control groups. We accounted for this unbalance in two ways. First, we regressed the math scores on the ITT indicator controlling for unbalanced pre-treatment characteristics. Second, we estimated the ITT effect exclusively for students who were attending the age-appropriate grade in 2013.4 This approach allows us to dismiss the hypothesis that the treatment school had lower thresholds for passing than other schools. If this were the case, ITT subjects attending the age-corresponding grade would be expected to show worse math results than non-ITT subjects in the same grade. In all cases, we analyzed standardized test scores (the ratio of the individual’s test score minus the test sample mean and the test’s standard deviation).

 

 

 

RESULTS

 

Table 2 reports OLS estimates of the effect of ITT status on students’ likelihood of attending the age-appropriate grade, having repeated a grade, and having dropped-out from school six months after treatment completion (July-August 2013), as well as on their expectations of completing college.

 

 

The first column in Table 2 shows that ITT status increased the likelihood of attending the age-appropriate grade in 2013 by 41 percentage points,a 100% increase relative to the observed likelihood in the non-drafted group (p<0.01). Most of this differential is explained by grade retention (see Column (2)). Almost half of the students in the control group (49%) repeated at least one grade between 2010 and 2013, whereas the likelihood of grade retention was of 13% (36 percentage points smaller) for ITT subjects (p<0.01). Results from the 1st follow-up wave, shown in Appendix Table A2, reveal a high rate of repetition for control subjects (21%) ever since the first year. The likelihoodof an ITT student not being promoted to the next grade was much smaller in the first year of treatment (2.4%) but increased in the following two years.

The sign of the coefficient in Column (3) suggests that ITT status may have also decreased the likelihood of dropping out from school. The coefficient, however, is not statistically significant. We are unable to say whether this non- significance reflects no differences in quit rates, or just the lack of statistical power.

Column (4) shows the incidence of ITT on students’ educational expectations. ITT status is associated with a 24 percentage point increase in a student’s expectations of completing college after having spent three years in the intervention school, double the expectations of the control group (p<0.04). The comparison with 1st year results (see Appendix Tables A2 and A3) indicates that the surge in academic expectations happened entirely during the 1st year. In effect, by the end of the first year, about 59% of the subjects within the ITT group and 30% of those not in the ITT sample reported they expected to complete college. While the rates decreased slightly for both groups in the following two years (to 49% and 24% respectively), the difference remained proportionally stable over time. There is also evidence (results can be shown upon request) that the treatment increased and sustained parents’ expectations about their children’s likelihood of completing college: by the end of the 3rd year, ITT parents’ expectations almost doubled the expectations of parents in the control group.5

Table 3 depicts the ITT effect on the scores of the math test administered at the end of 2012. The first column shows an unadjusted regression of the standardized math score on ITT status. Column (2) shows the estimates of the regression controlling for unbalanced covariates at baseline. In Column (3) we take a more conservative approach and only compare students that took the math test and attended the age-appropriate grade in 2012.

 

 

The unadjusted regression in Column (1) suggests that ITT status is associated with a 0.91 standard deviation increase in the score of the math standardized test. Once we adjust for unbalanced pre-treatment characteristics, the effect decreases to 0.67 standard deviations (p<0.05), which is still an economically significant impact. The estimate falls to half a standard deviation (significant at a statistical level of 10%) when comparing math scores only across students attending the age appropriate grade in 2013. This result is quite important as it shows that the intervention went beyond encouraging students to keep in track with the education system: it contributed differentially to the improvement of learning outcomes. Furthermore, the fact that treatment-school-students attending the age-corresponding grade in 2014 had higher academic achievement than the corresponding students in public schools dismisses any concern about a lower passing threshold in the treatment school.

The results above were robust to the inclusion of several covariates as controls in the OLS regressions. In particular, they were robust to the adjustment for the number of books at home during baseline and for randomization strata fixed effects. Results were also robust when using ITT status as an instrument of treatment in an instrumental variables regression. Results of the different robustness exercises are depicted in Appendix Tables A4 and A5.

 

DISCUSSION

 

The results from the first-year impact evaluation showed large positive effects of the treatment school on rates of academic promotion and academic expectations of disadvantaged adolescents (Balsa and Cid). In this third-year follow-up, we show that the first year effects were sustained and even strengthened over time. By the end of the intervention, treatment subjects had doubled the probability of attending the corresponding grade for their age than control individuals. Most of this effect was due to a lower likelihood of repeating a grade among treatment individuals. Moreover, the treatment school contributed substantially to improving learning, as revealed by treatment-control differences of more than half a standard deviation in math standardized scores.

In addition, the treatment boosted impressively students’ expectations of completing college. This effect appeared at the beginning of the treatment and persisted over time: even several months after leaving Liceo Jubilar, the academic expectations of the treatment students remained substantially higher than those of the control group. High expectations could be a consequence of students’ progressive realization that higher aims can be pursued and reached. If this were the case, one would expect high expectations to increase and strengthen over time as the student advances academically. Alternatively, the school may impose a culture of high expectations from day one, encouraging students to aim beyond what they are normally expected to deliver. While both explanations are feasible, we tend to think the latter fits well with the treatment school, particularly because the most important increase in expectations appeared at the beginning of the intervention, and suffered some decline for both treatment and control subjects in the following years.

According to Sulimani-Aidan and Benbenishty (2011), expectations are especially important in times of transitions and crises, which define the adolescence stage. In a review of the literature, they show that adolescents’ plans, aspiration and fears concerning probable events in various life domains in the near and distant future have a significant impact on their psychological status and on their motivation to engage in programs that prepare them for adult life. Positive expectations about the future have been identified as protective factors for urban children under stress, and have been related to resilience, social adjustment and well-being in general (Wyman, Cowen, Work, and Kerley). The ways in which adolescents see their future also play an important part in their identity formation, often defined in terms of exploration and commitments concerning future interests. In a nine month prospective study on expectations, Eric Dubow, Mitzi Arnett, Karen Smith, and Maria Ippolito found that higher level of positive future expectations were associated with lower levels of problem behaviors and negative peer influence, and to higher levels of school involvement, internal resources and social support. Other studies found that positive future expectations were associated with academic achievements (Arbona; Zimbardo & Boyd). Richard Catalano, Lisa Berglund, Jean Ryan, Heather Lonczak, and Davis Hawkins examined youth development program outcomes and found that positive beliefs about the future were linked to long-term goal setting, more positive beliefs about the value of higher education and work, better social and emotional adjustment in school, and improved self-competency. They concluded that belief in the future is an important component of intervention programs that produces positive outcomes among youth.

Aside from encouraging high expectations, there are other features in the treatment school that could help explain the differences in academic outcomes. Because we are dealing with a single school, we are unable to isolate the particular causal mechanisms behind these differentials. However, we can still identify relevant characteristics of the treatment and control schools that can help us speculate on potential mediators and construct hypotheses for future research.

An important difference between treatment and control schools has to do with students’ perceptions of the school’s environment. Table 4 depicts differences by ITT status in students’ perceptions of school climate, obtained from third year follow up home interview data. As before, we account for the multiplicity of measures being analyzed by using the Holm Bonferroni family-wise adjusted p-values. In what follows, we report statistical significance using these adjusted p-values.

Most of the reported measures reveal that ITT students have in average better perceptions of the school environment than non-drafted students. Students feel happier and safer in the treatment school (p<0.10), are more likely to feel proud of their school (p<0.10), and are more likely to report that their school is like a family (p<0.05). They are more likely to see commitment in their teachers (p<0.01) and to feel gratitude for the teachers’ work (p<0.10). These features suggest a stronger sense of belonging in the treatment school and a better social support network. Social support networks are defined as communities that provide psychological and tangible resources that can help individuals cope with multiple sources of stress.6

 



 

 

Resiliency studies suggest that social support networks play an important role in adolescents’ lives, shaping their future expectations (Luthar, Cicchetti, & Baker; Newman & Blackburn). High levels of social support have also been associated with higher self esteem, while lower levels of social support have been associated with depression and anxiety (Cohen).

Students’ responses about school climate reveal also striking differences in their perceptions of discipline. Students in the ITT category are 29 percentage points more likely than subjects in the control group to believe that “students in their school respect the teachers and staff, and that there is a disciplined environment” (p<0.10). Furthermore, only 35% of control subjects believe that “students in their school can resolve conflicts without fights, offenses, or threats”. The rate among ITT youths is 67% (p<0.05). ITT subjects also show a higher likelihood of considering that “the school imposed too many boundaries, students were not free enough”, suggesting lower tolerance levels in the treatment schools (p<0.05).

On the other hand, we find no statistically significant differences in students’ perceptions about the academic difficulty or usefulness of the school, or in the availability of schooling materials.

In Balsa and Cid, we used administrative data from the treatment and public schools (ANEP - CES) and self-reported data from the household interview to identify other differences in school characteristics by ITT status that could shed some light on school mechanisms. Regarding traditional school inputs, we showed that the treatment school was smaller than the average public school attended by control subjects and had a longer school day and academic year. The size of a cohort was 70 in the treatment school versus 382 in the average public school, and students in the treatment school spent 2.6 additional hours per day and about 40 days more at school per year compared to control subjects. In addition, treatment school students were less likely to exceed the grade appropriate age, reflecting a better peer academic quality. Specifically, the likelihood of having a peer exceeding the grade appropriate age was 60% in control public schools, versus 13% in the treatment school. This peer composition was both a result of the treatment school’s selection criterion7 and of the academic trajectories during the intervention phase. Other differences in favor of the treatment school included more parental involvement and more extracurricular activities. According to survey data, parents in the ITT group were more likely than parents of control subjects to turn to the school as a source of support and get involved in school activities. This involvement is directly related to the school’s policy of coordinating frequent interviews, and organizing meetings and workshops with parents. In addition, students in the treatment group were more likely to participate in religion and job training workshops, community service activities, and tutoring. Average class sizes, on the other hand, were larger in the treatment school.

This paper presents potential limitations. First, the rates of attrition in the math test were large. The rate of attrition was 33% in the treatment sample and 55% in the control sample. The mean comparison of baseline characteristics between the treatment and control adolescents taking the test showed a bias in favor of treated adolescents. For example, treated students taking the math test in Wave 3 were less likely to have repeated a grade than control students sitting for the test. To overcome this problem, we adjusted the raw test differentials for differences in baseline characteristics and compared test results only across students that had not repeated a grade. The estimates remain large and significant after these adjustments. We still believe we should place some caution when interpreting these results.

The overall attrition rate, on the other hand, was not bad for a third year follow up (7% and 16% in the treatment and control groups respectively). Furthermore, the comparison of observable characteristics at baseline did not show evidence of differential attrition for treatment and control youths. We could still be concerned that this result is due to poor statistical power. If there are non-observed differences between the remaining subjects, our estimates could be biased. We would be overestimating the school’s impact if, for example, non-respondents in the treatment group were in average lower achievers than non-respondents in the control group. Alternatively, we would be underestimating the effect if non-respondents in the control group happened to be students with lower ability than non-respondents in the treatment group. This second scenario will be more likely if bad students happen to leave the sample first.

Finally, the external validity of our conclusions is limited in principle to families that are similar to those that signed up for a placement in the treatment school and that satisfied the treatment school’s inclusion criteria. In a strict sense, our results can only be extrapolated to adolescents that do not exceed the grade-appropriate age in more than a year, and that come from poor families with enough motivation to seek for alternative educational offers. We believe, however, that our findings can shed new light on schooling innovations with the potential of delivering successful outcomes in broader contexts.

 

CONCLUSIONS

 

In this paper we study the middle-run academic impact of a privately managed middle school offering free-of-charge full-time education to socioeconomically disadvantaged youth. We find that, in a span of three years, the intervention increased the likelihood that an adolescent attends an age-appropriate grade by 41percentage points (mainly due to a lower likelihood of repetition) and had a strong impact on students’ and their parents’ expectations about college completion. We also present suggestive evidence that the school increased math scores by at least half a standard deviation.

Despite being unable to identify the causal mechanisms behind the observed treatment-control differences in outcomes, we speculate about potential channels that could explain the positive school’s impact. We find that the treatment school differs from schools attended by control subjects in several dimensions, namely a smaller size, more exposition of students to instructional time, a higher average academic quality of the student body, more parental involvement with the school, higher participation in extracurricular activities (including tutoring), a climate of discipline and belonging, and a culture of high expectations. The treatment school also differs from public schools in its ability to selectively higher and dismiss teachers, and in its freedom to assign teachers’ workload flexibly to teaching, coordination, and training. Future research should explore the impact of each of these features, and in particular the role of high academic expectations in fostering young peoples' academic progress.

ACKNOWLEDGEMENTS

 

We thank María Noel Domínguez for her technical support and field coordination, Florencia Saravia for her excellent research assistance, the treatment school’s management team for agreeing to participate in the project and for their collaboration in the process. We acknowledge the National Agency for Research and Innovation (ANII, Agencia Nacional de Investigación eInnovación) and the Center for Applied Research on Economics at the University of Montevideo for supporting the project financially. The authors are solely responsible for the contents and opinions in this paper.

 

REFERENCES

 

Arbona, Consuelo. “The development of academic achievement in school aged children: Precursors to career development” en Lent, Robert W. y Steven D. Brown (eds.) Handbook of counseling psychology. New York: John Wiley, 2000, pp.270-309.

Balsa, Ana y Alejandro Cid. “Impact Evaluation of a Privately Managed Tuition-Free Middle School in a Poor Neighborhood in Montevideo”. Working paper UM_CEE_2012-02, 2012. Disponible en http://fcee.um.edu.uy/investigacion/areas/ [Accedido en agosto de 2014].

Berends, Mark, Marisa Cannata, Ellen Goldring y Courtney Preston. School Innovation in District Context: Comparing Traditional Public Schools and Charter Schools” en Economics of Education Review, vol.31, 2012, pp.318-330.

Bierlein, Louann A., Chester E. Finn Jr., Bruno V. Manno y Gregg Vanourek. “How Charter Schools Are Different: Lessons and Implications from a National Study” en Phi Delta Kappan, vol.79, 1998, pp.488-98.

Bifulco, Robert y Helen F. Ladd, Helen. “Institutional Change and Coproduction of Public Services: The Effect of Charter Schools on Parental Involvement” en Journal of Public Administration Research and Theory, vol.16, 2005, pp.553-76.

Catalano, Richard F., M. Lisa Berglund, Jean A.M. Ryan, Heather S. Lonczak y J. Davis Hawkins. “Positive youth development in the United States: Research findings on evaluations of positive youth development programs” enThe Annals of the American Academy of Political and Social Science, vol.591, 2004, pp. 98-124.

Cohen, Sheldon. “Social relationships and health” en The American Psychologist, vol.59, 2004, pp.676-84.

Dobbie, Will y Roland G. Fryer Jr. “Are High Quality Schools Enough to Close the Achievement Gap? Evidence from the Harlem Children’s Zone” en American Economic Journal: Applied Economics, vol.3, n.3, 2011, pp.158-87.

---. “Getting Beneath the Veil of Effective Schools” en American Economic Journal: Applied Economics, vol.5, n.4, 2013, pp.28-60.

Dubow, Eric F., Mitzi Arnett, Katherine Smith y Maria F. Ippolito. “Predictors of future expectations of inner-city children: A 9-month prospective study” en The Journal of Early Adolescence, vol. 21, 2001, pp.5−28.

Luthar, Suniya S., Dante Cicchetti y Bronwyn Baker. “The construct of resilience: A critical evaluation and guidelines for future work” en Child Development, vol.71, 2000, pp. 543-62.

Newman, Tony y Sarah Blackburn. Transitions in the lives of children and young people: Resilience factors. Edinburgh: Scottish Executive Education Department, 2002.

PISA. Assessment Framework Key Competencies in Reading, Mathematics and Science: Key Competencies in Reading, Mathematics and Science 2009. OECD, 2009. Disponible en http://www.oecd.org/pisa [Accedido en agosto de 2014]

Purkey, Stewart C. y Smith, Marshall S. “Effective Schools: A Review” en The Elementary School Journal, Vol.83, 1983, pp. 426-52.

Sammons, Pam, Josh Hillman y Peter Mortimore. Key Characteristics of Effective Schools: A Review of School Effectiveness Research. England: Office for Standards in Education, London, 1995.

Sulimani-Aidan, Yafit y Rami Benbenishty. “Future Expectations of Adolescents in Residential Care in Israel” en Children and Youth Services Review, vol.33, 2011, pp.1134-41.

Wyman, Peter A., Emory L. Cowen, William C. Work y Judy H. Kerley. “The role of children future expectations in self esteem functioning and adjustment to life stress: A prospective study of urban at risk children” en Development and Psychopathology, Vol. 5,1993, pp. 645-61.

Zimbardo, Philip G. y John N. Boyd. “Putting time in perspective: A valid, reliable individual differences metric” en Journal of Personality and Social Psychology, vol. 77, 1999, pp.1271-88.

 

APPENDIX

 

 


 

 

 

 

 

 

 

 

** Doctora en Economía por Boston University. Docente de la Facultad de Ciencias Empresariales y Economía de la Universidad de Montevideo (UM) e investigadora del Centro de Investigaciones en Economía Aplicada de la UM. Sus áreas de trabajo son la economía de la salud y la evaluación de impacto de programas sociales y políticas públicas, especialmente en los campos de la educación y de la salud.

 

**** Doctor en Economía por la Universidad de San Andrés. Docente de la Facultad de Ciencias Empresariales y Economía de la Universidad de Montevideo (UM) e investigador del Centro de Investigaciones en Economía Aplicada de la UM. Sus áreas de trabajo son la economía de la familia y la evaluación de impacto de programas sociales.

 

1 Programme for International Student Assessment.

2One of the 101 original observations refused to participate in all instances of the study.

3 For robustness, we repeat the analysis adjusting the regressions for the likelihood of having more than 10 books at home, the only variable that showed a slight significant difference by ITT in this sample.

4All baseline characteristics are balanced at the 5% significance level for this sub-group.

5Outcomes in columns (1), (2), and (4) remain significant after using the Holm-Bonferroni family-wise adjustment of p-values.

6These networks provide emotional support (e.g. love and empathy), instrumental support (e.g. money and time) and informal support (e.g. guidance and advice).

7Those applying to enter the treatment school could not exceed the grade appropiate age by more than one year.

 

 

 

Creative Commons License Todo el contenido de esta revista, excepto dónde está identificado, está bajo una Licencia Creative Commons