2 Models for Measuring the Impact of Learning
Quick Backstory...
In 2012, Smeal College of Business adopted a new Sustainability Strategic Plan which presented an ambitious educational vision that every student would graduate “able to make the business case for sustainability.” Many curricular and co-curricular initiatives have been put into place (we have won national awards, made sustainability required for all business students...good stuff!) but the outcomes of these interventions has never been measured. To state the problem more broadly:
Thanks for being here. Click for more or to subscribe.
How do we know our education in sustainability is making a difference in the lives of the students and the organizations where they work?
Clearly if we are to improve our efforts and meet this ambitious goal, we need valid measures of educational impact. It turns out there are proven methodologies and models for this very task. One must look at the fields of training and professional development.
Approaches to Measuring Impact of Learning
This is not my field, but a colleague in the College of Education saw my ignorance and took pity on me. He helped me understand there are two prevailing approaches to measuring the outcomes and impact of training: the Kirkpatrick and Phillips models.
The Kirkpatrick Model was developed by Dr. Donald Kirkpatrick, now professor emeritus of the University of Wisconsin and past president of Association for Talent Development (ASTD). His book Evaluating Training Programs: The Four Levels is a seminal work in the area of evaluation. The model considers four levels at which outcomes can be measured (see below; image credit).
The Phillips Model (see below; image credit) essentially adds a “return on investment” component to the Kirkpatrick Model.
Dr. Jack Phillips is CEO of Jack Phillips Center for Research, a division of Franklin Covey. In his long career as a training and development consultant, he developed the Phillips Model to compensate for what he saw as a critical missing component in Kirkpatrick’s work: return on investment. Phillips book How to Measure Training Results has become, along with Kirkpatrick, a classic in the field.
Summarizing Kirkpatrick and Phillips: Measuring Business Education for Sustainability
These existing models from the world of training and professional development provide substantive direction for our task of measuring our progress in sustainability education.
My take-aways:
To evaluate our students’ ability to “make the business case for sustainability” requires us to define what we mean and define the knowledge and skills necessary to achieve this end.
The knowledge and skills are likely different for each major although overarching knowledge and skills will exist.
We need to determine the markers of learning that tell us the degree to which they have acquired and are applying the knowledge of skill.
I created the model below to demonstrate my thinking on how we might approach answering the question: “how do we know our education in sustainability is making a difference in the lives of the students and the organizations where they work?”
While at Smeal
We can measure students’ reaction to certain courses and experiences with sustainability content. We can also provide a pre and post-survey to assess their learning about sustainability while at Smeal and Penn State University.
After Smeal/Post-Graduation
We can measure the actual impact of this learning as our alumni enter the workforce. This could be done through an alumni survey, focus groups and other methods to understand actual changes in behavior and business results, the return on investment of those behavior and the “sustainability return on investment”.
Sustainability return on investment includes improved social and environmental performance of the business or organization.
In closing....we have a lot of work to do and it will take time to get validated lessons. We are committing to a multi-year effort to improve business education for sustainability. We believe it is time for it to enter a new wave of rigor, focus and data-driven accountability to results for business and society.
Imagine what we will learn as we follow our alumni in various job types in various industries. We will see how sustainability compliments their core skills in accounting or supply chain or finance. We will discover the actual scenarios in which their learning comes in contact with reality. And we can go back and re-engineer how we teach and how students learn.
Thanks for being here. Click for more or to subscribe.