Citizen Schools Extended Learning Time Model
*Note: The National Mentoring Resource Center makes these “Insights for Mentoring Practitioners” available for each program or practice reviewed by the National Mentoring Resource Center Research Board. Their purpose is to give mentoring professionals additional information and understanding that can help them apply reviews to their own programs. You can read the full review on the Crime Solutions website.
In considering the key takeaways from the research on this program that other mentoring programs can apply to their work, it’s useful to reflect on the features and practices that might have influenced its rating as “No effects” (that is, a program that has strong evidence that it did not achieve justice-related goals).
1. A commitment to evaluation over time can have tremendous benefits.
The commitment of Citizen Schools to program evaluation is very impressive, with many studies over the last two decades being referenced for this review—far more than the NMRC typically sees when doing a review for Crime Solutions. It’s not an exaggeration to say that Citizen Schools has been continuously evaluated since 2001, with several cohorts of youth tracked not only through their time in the program, but also into high school through the completion of their senior years. These evaluations examine program implementation and strengths and weaknesses of the program model, as well as student short-term gains and longer-term trajectories.
This is a brave choice for a program, especially in terms of tracking the long-term outcomes of participants well after their time in the program. There is no guarantee that the work of mentors and other caring adults will resonate for youth over time, as the wisdom of a mentor and lessons imparted may fade over time and in critical moments. But Citizen Schools should be applauded for testing to see if the work they do in middle school has an impact on the high school experiences of the youth they serve. The program is designed to not only address immediate academic needs through an extended learning time approach, but also to provide enrichment, skill-building, and future planning that one would expect might change the long-term academic fortunes of the youth served. The only way to find out is to collect the data.
The drawback comes when the results are less than expected, when all the life that happens in the high school years washes away some of the growth that happened in the program. The two major evaluations here paint a pretty different picture, with the evaluation of the original Boston site showing evidence of some meaningful benefits for participants’ academic trajectories and the more recent evaluation of sites that had expanded out nationally showing little to no discernible long-term impact for the full sample.
But it is also clear from these studies that Citizen Schools is a data-driven organization that has made several changes to their practices and implementation over the years as the annual data from these evaluations came in. Readers are encouraged to read the evaluation reports that informed this review, as they offer a good example of a program telling its story and making changes over time through continual evaluation activities. And that early evaluation provided the evidence they needed to expand their model to other cities, which is but a dream for many providers who want to take their work to scale. So regardless of the findings of their impact evaluations, Citizen Schools deserves a lot of credit for doing this level of process and long-term outcome evaluation in the first place.
2. Bringing in a “second shift” isn’t easy.
Because Citizen Schools included process, or implementation, evaluation as part of that ongoing research work, they have been able to learn a great deal as to what works in their model and where the sticking points of implementation may lay. Practitioners who are developing similar programming can learn a lot about the challenges of engaging a “second shift” of adults to come in and pick up where the first shift of teachers and school personal leave off. Arguably the biggest hurdle in making that work is to coordinate the efforts of Citizen Schools staff and volunteers with the content of the school day and the culture of the school. At some of their sites, this coordination was noted as a major challenge, but the seemingly more successful sites found a way to share information with the Citizen Schools staff and those teaching the apprenticeship classes. This ensured that homework help was in accordance with how the subjects were being taught during the school day and that information about particular students’ struggles was shared so that they could receive some individualized support.
This “overlap” of shifts also allowed the program to mirror the behavior and discipline policies and procedures of the school. So, for example, if the school was implementing a behavioral curriculum that set up rewards for good behavior, the program could also offer that and give youth consistent messages about their behavior.
But it also sounds like managing student behavior was a frequent issue at many of the sites as noted in participant surveys. One can imagine that a staff comprised mostly of younger AmeriCorps volunteers and a group of volunteer mentors might struggle to manage large groups of youth in an afterschool setting where the normal disciplinarians of the school day are not around.
There were also issues related to staff turnover and the model actually requires Teaching Fellows to transition out of their role after two years. This means that the people responsible for running the program and who have built relationships with parents and figured out how everything works best in a particular setting simply move on after two years of building up that institutional knowledge. That’s a lot of experience and relationship walking out the door. One wonders if the model might have more stability if those Fellows were regular paid staff who could stay in place over time.
But, as one of the evaluation reports notes, Citizen Schools learned all these things as a result of their process evaluation work and, as the program expanded nationally, they used this information to codify the requirements and structures of the program in ways that appeared to improve implementation over time, if not the overall impact results. So once again, the lesson is that one can’t learn lessons if one isn’t looking for them in feedback and data.
3. If you build it, can it be maintained?
The most recent evaluation report noted that sustainability after the initial seed money implementation was challenging for many schools. This is not surprising as the Citizen Schools model prioritizes resource-deficient schools and communities, with the idea that if they can tap into the strengths in that community they can build a lasting programmatic infrastructure.
But that seed money notion is always harder to make work in real life and the evaluation makes it clear that coming up with the next round of funds to keep the program going was a common challenge. One of the main challenges was the turnover in school or district leadership, which often saw program champions move on and left the program without key advocates who could make sure it was included in future budgets or who could raise funds from different sources. Practitioners working on a start-up grant would be well advised to continue to find allies and advocates who can champion the program, as there is no guarantee that the original supporters will remain in the mix over time.
4. The mandatory vs opt-in debate is challenging.
The original iteration of Citizen Schools in Boston was offered as an opt-in option for youth who were interested in the offerings of the program. This approach often resulted in a strong critical mass of youth participants who really wanted to be there. It also likely kept the program at an appropriate size and scale for the local resources available.
As Citizen Schools expanded nationally, it seems that they decided to make their model mandatory for all youth in particular grades. This likely facilitated stronger participation and buy-in from the school staff and parents. Yet, it also meant that some youth were mandated to be there when they would have rather been doing something else with their time. Anyone who has worked with middle schoolers knows that they love their first tastes of autonomy and greatly value their independence and free time. From the feedback gathered in the evaluation, it sounds like this was an issue for some of the sites, as youth resented being held at school for several more hours, even if they were likely getting some needed help out of the experience.
There are examples of other programs offering “whole grade” mandatory models. For example, the Peer Group Connection program led by the Center for Supportive Schools employs a model where an entire freshman class is mentored for the duration of their first year of high school. But those activities happen during the school day and are integrated into the freshman experience. It seems like that dynamic changes once the clock hits 3:00 and students start thinking about life outside the school walls. So while there are good reasons to go with a mandatory “whole grade” participation approach, it can have downsides that practitioners need to be aware of, especially when working with older youth who have rich lives and a sense of autonomy to nurture.
5. High school “access” work can be good practice for college access work.
One of the striking things about the original Boston-based evaluations of Citizen Schools is just how much effort the program put into steering participants to what the program termed “high-performing” high schools. In fact, one of the major metrics they focused on was how many alumni went to and persisted in one of these schools. And if you look at how youth were supported in this transition, it looks a lot like the work of college access programs: providing youth and parents with information about various high schools they could apply to, touring schools to assess fit, helping with the application form and financial aid considerations, etc. While the evaluations of Citizen Schools did not track cohorts of students into college, one wonders if practicing that selection and application process for high school built skills that proved handy when applying to higher education.
Now, not every part of the country has a high school system that operates with the same type of choice that Boston’s odd mix of private schools, parochial schools, charter schools, and normal public schools does. Students there have a lot more choice and mobility that do students in many other cities. But teaching these skills about how to choose and apply to the right school may be tremendously valuable, and it’s a shame that the evaluation didn’t look at whether those skills were helpful years later when youth were out of the program but doing something similar in the collage application process.
6. It’s always good to read honest discussions of research limitations and biases.
One notable thing about the two main evaluation reports of Citizen Schools noted here is just how clear the authors were in calling attention to the potential for false positives, biased samples, and other limitations of the research that may have clouded the findings. Any good evaluation report will have a limitations section, but those can sometimes downplay the severity of the issues or gloss over flaws in data collection or analysis. But that was not the case here.
The 2010 study by Arcaira and colleagues notes that there is a strong possibility of hidden bias in the results, mostly caused by the fact that those youth who opt into Citizen Schools may be different than those who don’t in meaningful ways. This possibility threatens the validity of even their matched comparison sample. It’s also worth noting that when a program participant dropped out of the study, they were replaced by a student who did not, meaning that the program did not endeavor to offer support to program drop outs, focusing instead only on those students who stuck with it. Once again, this might mean that they have strengths or supports that other students do not, but that are unaccounted for in this design. Non-participants in the comparison group who dropped out were also replaced by others who persisted. The report authors ultimately conclude that these and other factors may have overestimated the impact of the program, or at least confused things to the point where the validity is in question. And for an evaluation where many of the positive results were of small statistical significance, that’s a real issue.
The 2016 evaluation by Abt Associates offers a similar honest assessment of validity threats. It notes that data around participant experiences was not collected in a fully structured way and that not all youth or staff answered the same questions or participated equally in the data collection. The evaluation also notes the general research concern that most of the participant reports were done via self-report surveys, which can be positively biased as respondents give answers that are socially acceptable or “nice” but not necessarily accurate (e.g., program participants may be prone to report overly positive outcomes to help the program look good or even to cognitively justify their own time investment in program participation).
The Abt evaluation notes another issue that impacted their ability to tell a full story: variation in implementation across sites. The authors stress that Citizen Schools sites varied considerably in their implementation. Some of that is due to local customization that is entirely appropriate to the model and the available local resources. But much of it also appeared to be the result of implementation challenges, staff turnover, dwindling funds, or a poor fit between the program and the school conceptually. This leads to an evaluation context where sites are doing potentially radically different things while the evaluation tried to say something global about the “model.” If that model varies too widely, you wind up comparing apples to oranges to the whole fruit basket. It’s hard to say whether the Citizen Schools model was effective when there was so much variation in what youth were offered and received.
But it’s refreshing to see evaluation reports that honestly describe these concerns. Practitioners should always insist that their evaluation reports provide similar context around the strengths and limitations of the evaluation design and available data.
For more information on research-informed program practices and tools for implementation, be sure to consult the Elements of Effective Practice for Mentoring™ and the "Resources" section of the National Mentoring Resource Center site.