One Summer Plus (both regular and SEL+ versions)
*Note: The National Mentoring Resource Center makes these “Insights for Mentoring Practitioners” available for each program or practice reviewed by the National Mentoring Resource Center Research Board. Their purpose is to give mentoring professionals additional information and understanding that can help them apply reviews to their own programs. You can read the full review on the Crime Solutions website.
In considering the key takeaways from the research on this program that other mentoring programs can apply to their work, it’s useful to reflect on the features and practices that might have influenced its rating as “No effects” (that is, a program that, based on review of available evaluation research, does not show evidence of effectiveness for influencing juvenile justice or related outcomes).
Note: While Crime Solutions does consider the two variations on this program model to be separate interventions that were reviewed independently, here we will discuss both variations of the One Summer Plus model: the one that provides youth with employment opportunities and mentorship and the other that reduces those work hours (by 40%) to allow for youth to participate in workshops and learning opportunities related to social-emotional development.
1. If you are going to do a large study, why not test multiple variations?
One of the more challenging aspects of mentoring programs for practitioners to articulate are the causal mechanisms that allow mentoring to have an impact on a young person. While it may seem intuitive that mentors positively influence young people by offering advice, helping the young expand their connections and future plans, and serving as a role model, many programs cannot fill in the blanks between those activities and the ultimate outcomes of the program—reducing crime rates among disadvantaged youth, as in the case of One Summer Plus. It can be challenging to explain the causal pathway that allows something like the dispensing of advice to change a youth’s attitudes and behaviors deeply enough that their behavior changes, over time in meaningful ways. It can be similarly difficult to describe how that pathway happens within the context of other factors happening in the youth’s home, educational, and social life. Mentoring takes place in such subtle ways and in such a complex stew of life that even seasoned practitioners and scholars can know a program “works” but not know exactly how or why.
One Summer Plus’s evaluation design offers a potential way to try and answer these questions. The authors in the main journal article cited in this review note that there are many potential factors that might lead to a program like this reducing crime (or at least arrest rates, which is the main outcome reported on in these studies): having a job simply takes youth off the streets for periods of time in which they might commit crime, having the income of a job may reduce financial incentives for certain crimes, and the positive experience of employment might lead to more positive outlooks and a recommitment to school and education. In the case of One Summer Plus, the program is premised on the idea that adding a formal mentor to that job might also lead to personal growth and a move away from criminal behaviors.
In this study, the program tested not only the version of the program where youth received a 25-hour a week job and a dedicated mentor, but also a version where a portion of those job hours were used on classes designed to improve social-emotional competencies through the use of cognitive-behavioral principles. The idea here was to test and see whether the jobs/mentors alone would lead to significant crime reduction, or whether those outcomes might be strengthened by even more emphasis on “soft skills” that might help youth not only at the jobsite, but also in other situations, including those where they might commit crime.
Unfortunately, while the evidence suggests that the program overall did slightly reduce arrest rates for participants compared to the control youth (around 4 fewer arrests per 100 youth), neither version of the program really stood out from the other in terms of effectiveness. In fact, it appears that the only type of arrests the program impacted at all were violent crime arrests, which were reduced slightly more for those in the SEL version of the program (compared to the control group), but at a level that was barely statistically significant. All other types of arrests remained essentially the same for both program types and the control group.
Other programs may benefit from trying this type of multiple model design when evaluating their services. For example, is a school-based mentoring program effective because of the mentor spending time teaching academic skills or is a mentor simply showing up and having some fun with the student something that breaks up a challenging school day and helps the youth feel more positively about being at school overall? Asking some mentors to focus more on academics while encouraging others to focus on fun and play might show a difference in impact. Comparing those groups of mentors to another variation that combines mentoring with dedicated tutors might also highlight different ways such a program could work to maximize both educational progress and other personal growth. In the case of One Summer Plus, this multi-model design helped them understand some basic things about how their services might facilitate change—they now know that the SEL component might not be the main driver of program outcomes, which now allows them to focus on other aspects of the program that might be most influential for those gains on violent arrests. The evaluators also collected other data that helped rule out other theorized pathways of change, such as noting that school records did not indicate that the program changed much around the youth’s academic engagement. Those violent arrests kept dropping well after the program ended, so it can’t simply be the direct influence of mentors or the jobs eating up time that might have been spent getting into trouble.
So, what was the aspect of the program that reduced violent arrests? The authors, conclude that the exact causal mechanisms at play here are still not fully explained, but they do wind up speculating about a few things, including a return to the topic of SEL.
2. There may be more than one way to get at SEL growth.
While it’s not surprising that an intervention designed to give youth from disadvantaged communities a job over the summer might have reduced youth violence during that time period, what is surprising, as noted above, is that the reduction in violence not only extended beyond the employment period but actually seemed to grow stronger over time. Initially, the similarity in outcomes between the “jobs” and “jobs+SEL” versions seemed to indicate that SEL growth was not a contributing factor. But what if that’s not the case? What if there are some sneaky ways of growing SEL competencies in youth that perhaps evened out these findings?
One of the big trends in recent years is to infuse the curricula and activities from other evidence-based educational and mental health interventions into mentoring programs. We have written about this practice here at the NMRC and viewed broadly, it’s a mixed bag in terms of success. In some cases, doing formal SEL, mindset, or other curriculum-based work in the context of a mentoring relationship seems to produce stronger results than just a “friendship” based approach might have. But the research also indicates that mentors and youth often struggle to fit in these “add-on” activities to the work they are doing together and that mentors can find it challenging to learn and implement a whole new set of skills in addition to just building a positive relationship. Sometimes that “add-on” programming can even feel like a burden or distract from the core work the mentor and mentee are doing together. In the case of One Summer Plus, the program decided to see if a staff-led SEL curriculum could “boost” the outcomes of the job/mentor combination that the program usually provided. So around half of the youth in the program took 10 hours a week out of their work time to focus on SEL skill development. Sounds good on paper, but as noted above, it produced essentially no differences in findings for crime reduction among the two groups of program participants.
The authors of the paper reach an interesting, although untested conclusion about why this is after eliminating several other plausible reasons: The work with job mentors in the non-SEL cohort built SEL skills and competencies effectively too, giving youth who remained at the job site a fairly equivalent boost in their SEL skills. The authors explain that having a job where youth spent 25 hours a week problem solving, managing conflicts, organizing their time, communicating effectively with others, and collaborating in a team environment, all with the help of a mentor, may have built SEL skills as much as the dedicated curriculum. Although this was untested in the evaluation, it seems more plausible than the explanation that the SEL curriculum and delivery was wholly ineffective, considering that the program was using a manualized set of materials and other mentoring efforts utilizing cognitive-behavioral principles have shown effectiveness.
This has implications for how mentoring programs approach things like SEL skill development. The natural inclination of practitioners is to run out and find a proven SEL curriculum and then ask mentors to implement that or ask youth to take time away from (or in addition to) their mentoring time to work on the SEL skills too. But the case of One Summer Plus hints that perhaps you can do more of that SEL development within the context of normal program activities than one might think. This may be especially true of programs that provide jobs or other similarly challenging contexts and activities to mentees. By giving the young person a job or other challenges that will test them in new and unexpected ways, mentoring programs may provide valuable on-the-fly learning environments that, with the help of mentors, will allow youth to build SEL skills and apply them in the real-world. Mentors can guide and step in if the youth gets in over their head with a challenge, but there is something to be said about developing these skills organically in the real-world settings where they will need to be applied, rather in doing it in a separate, formal learning environment where skills are taught using a manualized curriculum in a vacuum. That may be what happened with One Summer Plus. After all, SEL skills are the skills of human interaction and there is no better place to build and refine those than a fast-paced work environment—especially with a caring, supportive mentor there as a guide and a backstop if problems arise.
So, the next time you are thinking about how to do focused work on some topic, especially universal skills like those focused on in SEL, within your program, remember that the activities matches are already doing might already a space where that growth is happening or could happen with just a bit more intentionality. That may be easier, and perhaps as effective, as a grafting some other regimented intervention onto your program. But, as One Summer Plus did, the key is to test to see if the idea worked or not.
3. Money matters, especially to those who don’t have enough.
The other possible explanation speculated about by the evaluators of One Summer Plus as to how the model influenced arrest rates really sticks out in terms of its blunt practicality: These neighborhoods were so impoverished that even the income from this part-time, minimum wage job was plausibly such a shot in the financial arm that it relieved some of the desperation and stress that these youth and families were feeling. The authors speculate that:
“The program provides a relatively large income shock, averaging $1,400 in neighborhoods where one third of households are below the poverty line and median income is about $35,000. Additional income could change criminal behavior directly or increase parental supervision by reducing how much parents need to work away from home.
It’s troubling to think that an amount of money that small would have that profound an effect on something like criminal behavior and violence in a community. And it’s worth noting that while violent crime arrests, such as for assaults, appeared to be reduced, the rates of other types of crimes that might be financially motivated were potentially unchanged by the program. But it is not outside of the realm of possibility that a little extra money might relieve a lot of stress on a family, allow for some other enrichment opportunities for youth, and change things like hopefulness and goal setting that might have an effect on criminal behavior. There is a growing movement among economists for ideas like Universal Basic Income (UBI) that might provide people in a community with some relatively minor supplemental income as a way of meeting basic needs, reducing stress and anxiety, allowing for more prosocial engagement opportunities, and generally keeping the peace. The authors of the study do note that such infusions of cash can also potentially have negative consequences, such as increasing the ability to buy drugs or alcohol, which might increase criminality. But it does seem quite possible the money earned from these jobs helped these youth and their families not only over that summer, but, when combined with the work experienced gained, may have also led to future employment and earnings stability based on a resumé with some stronger work history and references on it.
Ultimately, the authors conclude that the growth in SEL skills, combined with the direct and future earnings of the youth produced the apparent impact on violent crime arrests. But it’s remarkable to think about just how much that small amount of money may be able to change the lives of those in poverty. The idea of providing income supports is intriguing enough to have been added to the platforms of many political candidates in recent years, including several in the 2020 Presidential Campaign. Only time, and additional research can answer questions about whether things like UBI are worthy tools to employ alongside or in combination with other interventions. But it’s also worth noting that the program spent more money administering the program per youth ($1,600) than it spent on the wages youth earned through their work. Given the discussion here, it would be nice to see even more of those funds make their way to the youth and families who so very clearly need it, rather than supporting administrative staff who most likely aren’t in those challenging circumstances.
Not every mentoring program is in a position to give youth a job that changes the financial fortunes for a family. But most are in the position to provide some help in supporting a family to meet basic needs, either by referrals to other organizations or through other opportunities. Remember, sometimes the best thing your mentoring program might do for someone to improve their circumstances isn’t the mentoring.
4. Think carefully about whether self-report outcomes or program records are the best way to express the benefits of your program.
As one last little side note about this study, the authors make several interesting points about the tension between tracking outcomes from self-reports from program participants vs examining official records—in this case, arrest records and school data. The NMRC Research Board recently added guidance around working with both of these types of records to its Measurement Guidance Toolkit because, as the authors of the One Summer Plus study note, records offer some real advantages. Most importantly, they avoid some of the biases that can come when asking youth to report about their own behaviors, especially negative or illegal behaviors. In the case of One Summer Plus, youth may be reluctant to accurately report their violent or criminal behavior because they don’t trust that the program will keep that information confidential. They may also be worried about losing out of their job and the potential future connects that provides if they admit to wrongdoing. They may also have really enjoyed their time in the program or want to see the program succeed, and may downplay their true behavior to avoid harming the program itself. Importantly, none of these dynamics may apply to those in the control group, thus potentially leading to a finding of more arrests than those in the intervention group that is due to reporting biases rather than participation in the intervention per se. For all these reasons, One Summer Plus decided to examine official records instead.
But the authors are quick to note that records have their flaws too. Records of all types can have inconsistencies, errors, and missing information, no matter how diligently they are collected and recorded. In the case of criminal records, the authors note that only about half of all violent crimes are even reported and, of those, only half result in an arrest. So, if arrest records are your source of outcome data, it might be the case that those records are capturing only a quarter of the actual violent criminal behavior that is happening in a community. The only saving grace in this case is that those records for both treatment and control groups would be similarly depressed, meaning a difference between the two groups as a result of the program would still be valid at some level. But neither option is ideal.
Programs should give real thought as to which types of measures—self-report or official records—do the best job of capturing the desired goals of the program. Ideally, an evaluation would look at both, but that can be time consuming and expensive, and contradictory results can leave stakeholders confused as to what was actually achieved. (The other thing to note here is that the evaluation also broke down arrests by type rather than aggregating all arrests, which allowed them to detect the larger impact on violent crime, specifically—without separating out those arrest types, it may have seemed like the program achieved almost nothing. Ideally, these types of breakdowns are specified at the start of the study so as to avoid “massaging” data, even if unintentionally, after the fact in ways that favor configurations indicating positive program effects.)
The NMRC encourages practitioners and program evaluators to look at the new guidance in the Measurement Kit on using records data, as these sources may offer more reliable and nuanced ways of thinking about outcomes. Information on truancy records, disciplinary referrals, and grades, as well as juvenile offending records, are covered in these new additions.
For more information on research-informed program practices and tools for implementation, be sure to consult the Elements of Effective Practice for Mentoring™ and the "Resources" section of the National Mentoring Resource Center site.