Academic outcomes include performance in school, acquired skills in different subject areas (e.g., reading, math), longer-term levels of educational attainment, and attitudes and behaviors that can support school success. Many mentoring programs include academic outcomes in their logic model or theory of change as one of their most central goals for youth. In fact, a recent survey of mentoring programs in Illinois found that academic success was a top priority for over 80% of these programs.1 This is likely due to the strong policy relevance of academic outcomes—youth “success” is often measured, at least in part, through their ability to succeed in school and in related distal milestones, such as graduation from high school and acceptance into post-secondary education institutions. Mentoring also has a solid track record of benefiting youth in academic areas. DuBois et al.’s recent meta-analysis2, which synthesized the results of 73 evaluations of mentoring programs, found that, as a group, these studies showed evidence of positive effects on academic outcomes (e.g., grades, attendance, standardized test scores). Although modest in size (i.e., what would be statistically considered “small” effects), these benefits are nonetheless noteworthy and of a magnitude that would be widely considered to have policy relevance.
But deciding what to measure in this domain can be challenging. Affecting more “objective” measures of academic achievement or performance (such as official grades on report cards or standardized test scores) is often programs’ ultimate goal. Yet, these measures can be difficult and costly to gather and to ensure that they are measured in comparable ways across schools and districts. Many programs, thus, rely on youth reports of their academic performance as well as attitudinal or behavioral measures of factors that are associated with either performance or other longer-term goals like high school or college graduation.
One of the academic outcomes selected for inclusion in this Toolkit is academic performance, due to its direct policy relevance. The other four outcomes are attitudinal factors (growth mindset, academic self-efficacy, school engagement, and school connectedness) that show promising evidence of both influencing academic achievement and potentially being shaped by mentoring. As discussed, the recommended measures are not without limitations. Self-reported grades, for example, are clearly not a perfect proxy for achievement and other long-term outcomes of interest. School records can provide a more definitive measure of school success, but they can be costly and challenging to collect and analyze, particularly when assessing youth across schools and grade levels. Nevertheless, when utilized in a well-designed evaluation, the selected measures should give programs a good sense for whether the youth they are serving are benefiting academically.
Consideration was also given to reviewing measures for other important school-related factors, including educational aspirations, perceived value of school, and attendance. Some of these will likely be included in future updates. It is also worth noting that the recommended measure of school connectedness touches on bonding. School misbehavior and truancy, furthermore, are considered in the Problem Behavior section of the Toolkit.
When using any academic measure, it is important to understand that some academic outcomes change naturally over time, without intervention. This makes a comparison group of non-mentored, but otherwise similar youth essential when testing for program effects on these outcomes. For example, youth grades, on average, decrease during the transition from elementary to middle school.3 Thus, even if programs do not see improvement over time on this outcome, youth may actually be benefiting, relative to what would be expected during that developmental period. As a case in point, in the Public/Private Ventures evaluation of the BBBS Community-Based Mentoring program, mentored youth actually declined slightly in their self-reported grades over time.4 However, the control group declined more during the same period, yielding an overall positive impact of mentoring. The study was thus able to show that mentoring helped to prevent some of this steep decline. Without a comparison group, this benefit would have been missed.
Measures within this domain:
The recommended measure of academic performance focuses on grades. Grades can be collected through self-reports of performance or direct school records. School grades are of great interest to funders, policy makers, and potential program partners—especially for school-based programs whose partners’ main goal is often to foster youth’s academic skills. Even as early as elementary school, grades predict high school dropout and college attendance.5,6 Yet, asking youth to report on their own grades can be challenging. Timing issues (i.e., the last report card may have been issued several months before your follow-up assessment), framing differences (i.e., many schools have different grading systems and some do not use traditional grades at all), and potential challenges with memory (i.e., children may simply not remember their grades accurately) make it far from a perfectly “objective” measure. These factors also help explain why youth aren’t always completely accurate in their reporting of their own grades. One study7 for example, found a moderate correlation (about .5) between reported grades and actual grades in all but one subject area (Language Arts/Reading). However, similar to other studies,8 it also found that lower performing students (which many mentoring programs target) and younger children tend to be less accurate. Thus, self-reported measures of grades should be used with these cautions in mind and considered mainly when resources preclude gathering more objective measures. At the same time, collecting school records also comes with several steps and potential challenges including: developing a relationship with associated schools or districts; ensuring that you have signed permission for collecting these records; combining grades across multiple grading systems; and interpreting change over time or across different schools.
A “growth mindset” is the belief that intelligence can be improved with effort. This contrasts with a “fixed” mindset which holds that intelligence is a stable trait that can’t be changed. Research, most notably by Carol Dweck, suggests that a growth mindset can contribute to better academic achievement. One study,9 for example, measured the growth mindset of 373 7th graders and followed these youth for two years. They found that the students with a growth mindset were more likely to report focusing on effort and learning and not giving up in the face of challenges, whereas the students with a fixed mindset were more likely to report giving up easily and ignoring feedback. Over time, despite similar math skills at the start of the study, those youth with growth mindsets outperformed their peers. This same study tested an intervention which taught growth mindset to a group of students and compared their progress to a randomly assigned control group not receiving the intervention. Findings indicated that the intervention prevented the declining grades seen in the control group. Thus, growth mindset shows evidence of both promoting academic success and being a malleable trait that—with intervention—can be changed over time. Evidence of the ability of interventions to shape growth mindset thus far is limited to those that target this kind of cognitive process. Programs are thus advised to keep this caveat in mind when deciding whether to assess growth mindset as a potential outcome.
Academic self-efficacy refers to youths’ confidence in their performance capabilities related to schoolwork, including not only overall abilities but also their capacity to succeed at tasks (e.g., homework) with perseverance and effort. Numerous studies support links between academic self-efficacy and both academic performance and persistence.10,11,12 Efficacy beliefs about a task or activity are thought to influence how an individual will approach that task, including how much effort the individual will invest as well as how resilient he/she will be and how long he/she will persevere when confronting academic challenges.11 In line with these ideas, studies find that, when holding ability constant, students with high academic self-efficacy put more effort into their schoolwork, are more cognitively engaged in school, and use more effective self-regulatory strategies relative to students with lower academic self-efficacy.11,13 Research also points to academic self-efficacy as an important mediator of the effects of knowledge and skills on subsequent academic performance.11 Academic self-efficacy has received relatively little consideration in research on mentoring, although one study indicates that school-based mentoring has the potential to positively influence self-perceptions of academic ability (i.e., how capable one is relative to other students).14
Although definitions vary, for purposes of this toolkit school engagement refers to a young person’s level of behavioral engagement and active participation with learning activities in the classroom, including effort, attention, and contribution to classroom discussions.15,16 More active participation in classroom learning activities among children is associated positively with academic achievement and negatively with skipping school and dropping out.15,16 A number of studies suggest that mentoring and other supportive relationships with adults within the school setting (e.g., teachers) may promote academic engagement among youth.17,18,19 The potential for mentoring received outside of school to offer similar benefits is not clear.
School connectedness is the youth’s feelings of connection to the school environment. Researchers have defined it in many different ways—some include factors like feelings of safety and support from teachers and peers, whereas others include aspects of school bonding, school climate, engagement, and involvement.20 Studies support associations (when one item goes up or down, another follows the same trend) between school connectedness and a wide range of youth outcomes including school achievement and overall health status.21 It is also a potential protective factor, exhibiting negative associations with cigarette, alcohol21 and drug use,22 delinquency and gang membership,22 sexual activity,22 emotional distress,23 violence,22 and suicidality.23 Michael Karcher’s work24,25 further suggests that participation in school-based mentoring can improve school connectedness, hinting that it may be a particularly relevant outcome in school-based programs.
i This difference may have been caused, in part, by the fact that some youth may have had only one, both or neither of these subjects in school, which could affect how they rated their performance in this area and how this rating was associated with their actual performance.
1. DuBois, D. L., Felner, J., & O’Neal, B. (2014). State of mentoring in Illinois. Chicago, IL: Illinois Mentoring Partnership. Retrieved from http://ilmentoring.org/images/pdf/SoM-Full-Report.pdf
2. DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57–91. http://dx.doi.org/10.1177/1529100611414806
3. Gutman, L. M., & Midgley, C. (2000). The role of protective factors in supporting the academic achievement of poor African American students during the middle school transition. Journal of Youth and Adolescence, 29, 223–249. http://dx.doi.org/10.1023/A:1005108700243
4. Tierney, J. P., Grossman, J. B., & Resch, N. L. (1995). Making a difference: An impact study of Big Brothers/Big Sisters. Philadelphia, PA: Public/Private Ventures. Retrieved from http://bit.ly/2bUT3gf
5. Lloyd, D. N. (1978). Prediction of school failure from third-grade data. Educational and Psychological Measurement, 38, 1193–1200. http://dx.doi.org/10.1177/001316447803800442
6. Entwisle, D. R., Alexander, K. L., & Olson, L. S. (2005). First grade and educational attainment by age 22: A new story. American Journal of Sociology, 110, 1458–1502. http://dx.doi.org/10.1086/428444
7. Teye, A. C., & Peaslee, L. (2015). Measuring educational outcomes for at-risk children and youth: Issues with the validity of self-reported data. Child & Youth Care Forum, 44, 853–873. http://dx.doi.org/10.1007/s10566-015-9310-5
8. Kuncel, N. R., Credé, M., & Thomas, L. L. (2005). The validity of self-reported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of educational research, 75, 63–82. http://dx.doi.org/10.3102/00346543075001063
9. Blackwell, L. S., Trzesniewski, K. H., & Dweck, C. S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: A longitudinal study and an intervention. Child development, 78, 246–263. http://dx.doi.org/10.1111/j.1467-8624.2007.00995.x
10. Multon, K. D., Brown, S. D., & Lent, R. W. (1991). Relation of self-efficacy beliefs to academic outcomes: A meta-analytic investigation. Journal of Counseling Psychology, 38, 30-38. http://dx.doi.org/10.1037/0022-0188.8.131.52
11. Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of Educational Research, 66, 543-578
12. Schunk, D. H., & Pajares, F. (2001) The development of academic self-efficacy. In A. Wigfield & J. Eccles (eds., pp.15-31), Development of achievement motivation. San Diego, CA: Academic Press.
13. Linnenbrink, E. A., & Pintrich, P. R. (2002). Motivation as an enabler for academic success. School Psychology Review, 31, 313–327.
14. Herrera, C., Grossman, J. B., Kauh, T. J., Feldman, A. F., McMaken, J., & Jucovy, L. (2007). Making a difference in schools: The Big Brothers Big Sisters school-based mentoring impact study. Philadelphia, PA: Public/Private Ventures. Retrieved from http://issuelab.org/home
15. Finlay, K. A. (2006). Quantifying school engagement: A research report. Denver, CO: National Center for School Engagement. Retrieved from http://schoolengagement.org/wp-content/uploads/2013/12/QuantifyingSchoolEngagementResearchReport-2.pdf
16. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School Engagement: Potential of the Concept, State of the Evidence. Review of Educational Research, 74, 59–109
17. Anderson, A. R., Christenson, S. L., Sinclair, M. F., & Lehr, C. A. (2004). Check & Connect: The importance of relationships for promoting engagement with school. Journal of School Psychology, 42(2), 95-113.
18. Murray, C. (2009). Parent and teacher relationships as predictors of school engagement and functioning among low-income urban youth. The Journal of Early Adolescence, 29(3), 376-404.
19. Woolley, M. E., & Bowen, G. L. (2007). In the context of risk: Supportive adults and the school engagement of middle school students. Family Relations, 56(1), 92-104.
20. Libbey, H. P. (2004). Measuring student relationships to school: Attachment, bonding, connectedness, and engagement. Journal of School Health, 74, 274–283. https://eric.ed.gov/?id=EJ743600
21. Bonny, A. E., Britto, M. T., Klostermann, B. K., Hornung, R. W., & Slap, G. B. (2000). School disconnectedness: Identifying adolescents at risk. Pediatrics, 106, 1017–1021. http://dx.doi.org/10.1111/j.1746-1561.2009.00415.x
22. Catalano, R. F., Oesterle, S., Fleming, C. B., & Hawkins, J. D. (2004). The importance of bonding to school for healthy development: Findings from the Social Development Research Group. Journal of School Health, 74, 252–261. http://dx.doi.org/10.1111/j.1746-1561.2004.tb08281.x
23. Resnick, M. D., Bearman, P. S., Blum, R. W., Bauman, K. E., Harris, K. M., Jones, J., . . . Ireland, M. (1997). Protecting adolescents from harm: findings from the National Longitudinal Study on Adolescent Health. JAMA, 278, 823–832. http://dx.doi.org/10.1001/jama.1997.03550100049038
24. Karcher, M. J. (2005). The effects of developmental mentoring and high school mentors' attendance on their younger mentees' self‐esteem, social skills, and connectedness. Psychology in the Schools, 42, 65–77. http://dx.doi.org/10.1002/pits.20025
25. Karcher, M. J. (2008). The study of mentoring in the learning environment (SMILE): A randomized evaluation of the effectiveness of school-based mentoring. Prevention Science, 9, 99–113. http://dx.doi.org/10.1007/s11121-008-0083-z