Eye to Eye
*Note: The National Mentoring Resource Center makes these “Insights for Mentoring Practitioners” available for each program or practice reviewed by the National Mentoring Resource Center Research Board. Their purpose is to give mentoring professionals additional information and understanding that can help them apply reviews to their own programs. You can read the full review on the Crime Solutions website.
In considering the key takeaways from the research on this program that other mentoring programs can apply to their work, it’s useful to reflect on the features and practices that might have influenced its rating as “Promising” (that is, a program that has strong evidence that it achieved several of its justice-related goals).
1. If you’re going to do an evaluation, make sure you describe the program well when writing up the results.
This may seem like a small thing to praise this program and its evaluators for, but it was refreshing to see a mentoring program described in real detail as part of an outcome evaluation. Most published studies like this will describe the program at some level, usually focusing on the demographics of the youth served and their mentors, as well as a general explanation of what the program is trying to accomplish. But this article really sets the standard in terms of the quality and depth of information provided about how Eye to Eye selects schools, selects mentee participants within those schools, the curriculum they use and the activities matches engage in, and even concrete details about how mentors are identified and screened. Far too often, these Crime Solutions reviews and subsequent profiles provide plenty of information about the evaluation, but practitioners are often left wondering how the program achieved their results or how they might proceed if they wanted to do something similar. By providing such rich descriptive information in the study write-up itself, this program has made it much easier for other practitioners to understand the practices that might best support youth with learning disabilities (LD) and attention-deficit hyperactivity disorder (ADHD).
2. Credible messengers once again prove to be valuable assets to others.
One of the factors that seems likely to be involved in the success of the Eye to Eye program is the use of older peers who themselves have LD/ADHD diagnoses as the mentors. Using these youth, specifically, in the mentor role might be crucial in helping mentees feel understood, listened to, and able to visualize themselves being a thriving adolescent or young adult. Mentors without those diagnoses may have also been effective in working with these youth—and the evaluation here did not test this or compare these diagnosed mentors to other types of individuals. But we have seen uses of credible messengers many times before in the mentoring field and there seems to be some real validity to using mentors who know exactly what it’s like to being in their mentee’s shoes. (See the Insights provided for the Arches and My Life programs for examples of other programs using mentors who had faced similar challenges to the youth they are serving.) It’s not surprising that measures of relationship quality for these matches were quite high and correlated with stronger outcomes inn the directions of lowered depressive symptoms and increased self-esteem and personal relationships. Sometimes we learn the most from those who have travelled the same path before us. What’s especially promising about the Eye to Eye approach is that the mentors also appear likely to have gotten a lot out of the experience, taking an ownership role in how the model was implemented in their school and building leadership and project management skills. Even though the evaluation didn’t explore this aspect in detail, one can assume that serving as leaders and mentors like this may have also helped the mentors feel more positive about their own LD/ADHD circumstances.
3. When designing a curriculum, get the best help possible from experts.
One of the great debates in mentoring is whether the impact of mentoring comes from the strong relationship (ideally) formed between mentor and mentee or whether those outcomes are the result of the activities they do together. And while the answer may well be that the best impacts often come from both of those things in tandem, the reality is that for a program like Eye to Eye that is working with youth with serious disabilities and conditions, having a good curriculum that’s designed to facilitate specific learning moments and interactions tailored to those youths’ needs may be critical.
But programs often wonder how to develop a curriculum that is the right fit for what they want to accomplish with youth. Eye to Eye offers an excellent example of how other programs might want to approach this. They started by identifying core socio-emotional objectives based on a longitudinal study of youth with LD/ADHD that had previously identified success attributes that helped those youth thrive. With those objectives in hand, the program engaged a number of groups in designing relevant activities that would speak to those objectives: a team of educators with LD/ADHD themselves, a focus group of young adults, and, perhaps most crucially, faculty and graduate/postdoctoral students at Brown, Harvard, and Columbia Universities. That’s a lot of expertise and different viewpoints all contributing to the formation of these activities. In the end, the program had developed an arts-based curriculum that uses fun and creative hands-on projects to get mentors and mentees talking about strengths and challenges related to their LD/ADHD. This is an excellent example of how researchers, subject matter experts, and client voice can all be harnessed to produce something that is custom tailored to the youth that are the focus of the program. And, because Eye to Eye is so transparent about their model (see point #1 above) they even make a national office email available in the study write-up for those who want to learn more about it or adapt it for their programs.
4. If you are evaluating your program and want a comparison group, make sure you know if those youth are being mentored somewhere else.
One of the nice things about this evaluation is that they compared the outcomes of mentored youth in the program with two different comparison groups: unmentored youth with LD/ADHD at similar schools and youth at similar schools who did not have LD/ADHD. A small but important detail in setting those groups up is that they excluded youth from both groups who indicated they were receiving mentoring through some other program. In this case, they didn’t want to compare Eye to Eye against other programs or service providers, but against a hypothetical situation that isolated the influence of Eye to Eye mentors compared to very similar youth who didn’t have the services of the program.
Now, this study was not a true random assignment design—they did do this kind of purposeful restricting of who got into those comparison groups. So while this may not have been a pure experiment, the program also seems to have avoided one of the big reasons that many mentoring program evaluations struggle to show results: the comparison kids going and finding mentoring somewhere else. While youth in the comparison schools were not restricted from seeking out and receiving other services, it’s likely that few did given the somewhat novel intervention offered here, which the authors note had few comparisons in the literature.
When significant numbers of the youth who are the counterfactual to the work of the program go and get similar services from somewhere else, it can wash away differences between the two groups at a meaningful level. You are no longer comparing mentored and unmentored youth, you are comparing mentored and differently mentored, and given the tight thresholds used in the statistical analyses of these types of evaluations, that can often make the difference between having several positive, statistically significant findings and having results that look like the program achieved nothing.
Another good example of this phenomena can be found in the Insights we wrote for the SOURCE program. That program emphasizes working with youth to apply to college the following year and does a lot of work to facilitate that process and get parents on board. The program did seem to do an effective job of getting youth to apply, which was good news. However, their evaluation also found that almost 94% of their comparison group youth applied to college as well, often with the help of other services and programmatic support, either through their school or from other similar nonprofits. The end result is that it looked like there program was no better than just “business as usual.”
Now, with a goal like college planning and application in mind, it may be a good thing that such a high percentage of the comparison group did apply—after all this is their time to do it and it would be impossible to tell a family to defer that decision for a year just for an evaluation trying to test the results of one program. But time and time again we see mentoring program evaluations that are undone by comparison groups of kids getting mentoring from other programmatic sources (sometimes in spite of promising not to). So this is a nice caution to mentoring programs and evaluators to set up your comparison groups carefully. In the case of Eye to Eye, they removed anyone already being mentored from those groups and seemed to avoid comparison youth getting similar support elsewhere, making it easier for them to show clear impacts that are more easily attributable to the work of the program. Of course, to make sure eventual comparisons are truly “apples to apples,” the programs involved need to be similarly restricted to those without pre-existing mentoring relationships, something which may be useful to also consider for other reasons such as prioritizing access to limited program slots or mentors.
For more information on research-informed program practices and tools for implementation, be sure to consult the Elements of Effective Practice for Mentoring™ and the "Resources" section of the National Mentoring Resource Center site.