Project Arrive

In considering the key takeaways from the research on this program that other mentoring programs can apply to their work, it’s useful to reflect on the features and practices that might have influenced its rating as “No effects” (that is, a program that has strong evidence that it did not achieve justice-related goals).

1. Before one can evaluate, one has to clarify how to operate.

One of the interesting aspects of this evaluation is how much time and energy was spend developing an operations manual for the program that more tightly specified the intervention and programming. One doesn’t often think about manual development as a part of program evaluation, but the reality is that it can be impossible to measure the impact of something if that something is ill-defined and being done in a variety of ways. So it was imperative to the outcome evaluation that these mentoring groups be led and managed in similar ways across schools and even within schools. This reduced variability in how the program was being implemented and ensured that the evaluation was looking at one core intervention, albeit with some flexibility.

Taking the time to develop and share the manual and implementation materials also had the side benefit of giving the program some needed infrastructure that can carry the work forward even if various program leaders and key stakeholders leave. The development of a policy and procedure manual that governs how the program is implemented and clearly spells out policy choices is a critical aspect of program sustainability. It can also support program replication and dissemination. In fact, we applaud Project Arrive for making their manualized and standardized materials available to other practitioners on a publicly accessible website. As noted in the evaluation, this site is visited by as many as 1,300 visitors a month from all over the world—many of whom are using the materials to develop their own versions of the program or at least incorporating key ideas. That counts as a major resource for the group mentoring field. And none of it would have been possible if they hadn’t taken the time to put all those materials online.

Programs who haven’t taken the time to fully develop a policy and procedure manual can always do so using the template provided in the Resources section of the NMRC site. Between that template and the Elements of Effective Practice for Mentoring that Project Arrive used as a touchstone, practitioners have everything they need to codify and tighten the implementation of their program in a formal manual—regardless of whether an evaluation is imminent.

2. Group size and composition might rally matter in these programs.

There are a few hints in the evaluation that group size was particularly important in relation to results. Smaller groups tended to rate both their relationships with their mentors and their group dynamics more favorably than larger groups, and those two results were correlated with positive outcomes on academics and resiliency in the study. Larger groups might be harder for mentors to manage and might prevent all members from fully participating due to time constraints or feelings of shyness by mentees when faced with speaking to a larger group of peers.

The number of mentors also appears to have mattered here, with youth reporting many positive benefits of having co-mentors for each group. These co-mentors often brought different skills and temperaments to the role, meaning that youth had access to a greater range of knowledge and different personalities that might be a good fit for them. Having two mentors also meant that the group could still meet if one of the mentors was absent, something that contributed greatly to the low number of missed meetings in the program.

The mentoring field has long wrestled with the question of what the right mentor-mentee ratio is for group mentoring programs. In a recent podcast about Project Arrive, lead researcher Gabe Kuperminc indicated that the most cohesive groups in this study had about 8 youth matched with those 2 mentors—a 4:1 ratio. Although programs can certainly stretch that ratio in either direction and still find success, that 4:1 ratio may be a bit of a sweet spot, where the group is still manageable, and every youth has a chance to fully participate, while still bringing enough diversity of opinion and personality that the groups are not homogenous and boring. Practitioners should think carefully about finding a group size that works for their mentors and should strongly consider having co-mentors, or even 3 or more mentors for a group. Multiple mentors may mean less chaos, more options for a close mentoring relationship, and fewer missed sessions.

3. “Curriculum with creativity” may be essential to avoiding a cookie cutter mentoring experience.

One of the interesting side notes in the evaluation report is that groups seemed to find their groove when deviating, as needed, from the prescribed curriculum of the program. Now, the curriculum itself offered a lot of choice, where mentors and youth could choose their discussion topics. So that offered some flexibility and customization right there. But some mentors went further by introducing other activities or even setting the curriculum aside for more organic conversations when warranted. For example, some mentors noted that there were times when students would arrive at the meetings upset about something that had happened in school, or in the community or on the news. These mentors recognized that what their mentees needed was a chance to process, to vent, to share with each other and to hear each other’s voices. They assessed that sticking steadfastly to the set activity for the day would not meet the needs of students and made a decision to go in a different, but more meaningful direction for that day.

In the aforementioned podcast, Dr. Kuperminc describes this deviation as “curriculum with creativity” and argues that whereas all group mentoring programs need some kind of activity-driven curriculum to guide the groups, that there are plenty of times when deviating from that curriculum and meeting youth where they are at is the right thing to do. This has the potential to prevent youth from feeling as if they are being treated like widgets to be manipulated and engages them in meeting immediate needs and processing challenging topics in a supportive, peer-focused environment.

There were other things that groups did to customize the experience. For example, each group was free to develop little rituals and traditions that were done each time they met. These rituals contributed to the feelings of “family” that many participants spoke of. They offered not only a way of grounding the group in routine, but they also gave groups some control and autonomy over how they met and the way they ran their groups.

Unfortunately, not all mentors were adept at deviating from the curriculum in meaningful ways, and many struggled with group management and situations where going “off script” may have been warranted. In fact, the evaluation report posits that some outcomes could be strengthened with more training and supervision for mentors, which could not only strengthen the facilitation of the groups, but might also help mentors know when to deviate from the “script” and engage youth in more open conversation and sharing.

4. Improving resilience is an outcome with an ambiguous payoff.

Perhaps the most impressive findings in the report are those related to resilience outcomes. Project Arrive appears to have produced some really meaningful changes in several external areas that could boost youths’ resiliency: perceptions of school support and school belonging, school participation, and caring relationships with prosocial peers. The program was less effective with internal resiliency measures, with only problem-solving coming out with evidence of a strong positive benefit. There was even a hint that external assets were best explained through the relationship with the mentor and that internal assets were best explained by the relationship with peers in the group, suggesting that these programs are about more than just an adult-youth relationship. These groups appeared to get kids interacting with one another in ways that built resilience too.

But evidence of impact of the program on other outcomes was not so great. There were essentially null findings around juvenile justice involvement and youth grades. One might wonder why these things were not improved if the youth were making such great gains in these resiliency areas. The issue may be that resiliency is a process, not a stepping stone to other improvements. It’s a process that is activated, when something bad happens to a child and they either bounce back… or don’t. Ideally the protective factors (like the ones seemingly built up by Project Arrive) kick in in these moments and keep youth from sliding down the rabbit hole. But one can’t predict when and how those resilience factors will be called upon. The resilience assets built by the program may have helped some youth tackle immediate challenges at school. Yet, for most of the mentees, these built up factors mayl only become useful at some point in the future, when they have a negative experience where they need to draw on their assets. This highlights the strange contradictions that can happen in mentoring programs. Project Arrive may have not been doing the right things to immediately impact grades or delinquency, but they appear to have strengthened assets that can last these youth well into their future. We might never know the full impact of that boost in protective factors (although a longer-term follow-up of the program could certainly help in that regard). But it seems likely from the effect sizes reported here that they are quite meaningful.

For those who wish to learn more about the importance of viewing resilience as a process and not a set outcome, see this thoughtful blog post, also by Dr. Kuperminc.

*Note: The National Mentoring Resource Center makes these “Insights for Mentoring Practitioners” available for each program or practice reviewed by the National Mentoring Resource Center Research Board. Their purpose is to give mentoring professionals additional information and understanding that can help them apply reviews to their own programs. You can read the full review on the Crime Solutions website.