Content originally presented by Gianina Baker, Ph.D., National Institute for Learning Outcomes Assessment (NILOA)
In January of 2018, The National Institute for Learning Outcomes Assessment (NILOA) shared a report analyzing a national study of institutional effectiveness communication called “Assessment that Matters.” One of the major findings and implications of this study is:
“Communicating effectively about student learning remains a challenge. Colleges and universities must more clearly and persuasively communicate relevant, timely, and contextualized information on their impact on students and value to society.”
Specific Challenges
When drilling down into this data to find what types of information and outcomes to publicly share and the challenges around that, a few themes emerged. Specifically, provosts said that sharing these outcomes are most important:
- Information on accreditation retention, persistence, graduation, & completion rates
- Licensure & certification exam pass rates
- Job placement & salaries
- Return on investment
- Costs
And, what was most important to provosts was not what to share, but how to share this information. Most institutions have tons of data, reports, studies, and opportunities to communicate all of this information publicly. So how have institutions traditionally approached presenting outcomes?
Current Institutional Communication Approaches
One of the ways institutions try to communicate is by flooding their audience with data. They present comprehensive findings by item, instrument or measure, where there is little connection between each instrument or measure. Their idea is to make all of their information available in the name of ”transparency,” with very little meaning or analysis of what the data represents in the overall narrative. Other examples where there are opportunities for better communication include:
- Overwhelming audiences with access to data without meaning-making
- Scattershot bullet lists of processes attempting to guess what people want
- Archives of reports that document the processes – but provide a confusing history of data collection or changes made
- Posting individual student stories of success (internships) without any context or groups of stories
- Providing limited publicly available information on their websites. Institutional websites are most often prime spaces to communicate effective student learning results, but are often forgotten about spaces
One of the biggest presentation and interpretation gaps between audience understanding and assessment evidence is best stated by one provost from the study:
“We can provide all the assessment results or data we like, but if others cannot interpret them accurately there is no benefit to transparency or accountability.”
Communicating outputs is not sufficient. Outcome results must be made more meaningful for target audiences.
“This is not easy to share in ‘sound bites,’ and communicating outputs such as employment rates and beginning salaries does not serve as a proxy for student learning and quality of programs…we need to find ways to help the general public make meaning of the results.”
We need to be much more explicit and meaningful about sharing the good work of assessment student learning on campuses. Specifically, we need to be intentional about what to share, how we share (mostly externally), and most importantly, how to communicate a nuanced, complex picture of student learning that couples evidence of learning outcomes with student success data.
A Step-by-step Visual
Here is a step-by-step continuum of how most institutions can be categorized as they go about communicating learning outcome assessment to the general public. As you think about how and what your institution communicates, where do you fall on this scale?
- Compliance posting of outcomes & select (favorable) results
- Oversharing of easy-to-measure outcomes & results, just to do something
- More tailored posting of outcomes, by program/major experience & results for different audiences
- More complete picture of all the places learning occurs, posting outcomes in student affairs/services & results for different audiences
- Nuanced, tailored framework for communicating outcomes and improvements in a comprehensive, student-focused and culturally responsive way
Effectively communicating information about student learning clearly remains a target of opportunity. So how do we effectively communicate?
Why Assessment Storytelling Matters – Evidence-Based Storytelling
A story in itself tells others a bit about us and of alternate theories of others, and seeks to make believers out readers. Stories allow us to go assumption hunting. Evidence-based storytelling is defined by the evidence of student learning used in support of claims or arguments about improvement and accountability, told through stories to persuade a specific audience.
When starting to construct an evidence-based story, here is a guide to be able to effectively tell our story and help students tell theirs.
Supportive claims or arguments. Here we are using Toulmin’s Argumentation Model to construct the beginnings of the story. This means we are selecting a variety of evidence of student learning, which may change depending on the audience, and providing the end relation to the claim that is made about the improvement of student learning. This justifies why this change will lead to the improvement we want to see in student learning.
Improvement or accountability. What argument do you want to make about your students’ learning? What type of evidence would be necessary to make the argument?
Told through stories: Elements for storytelling and persuading a specific audience. Here are the basic elements of storytelling that you can use to think through and craft an effective narrative to a specific audience.
- Audience: For whom is this narrative written? What counts as evidence for the different audiences of the report?
- What kind of story are you telling? (i.e., compliance, improvement, loss, struggle, quest, tragedy, fantasy, etc.) What context is needed for readers to understand the story? What is the setting?
- Who are the character(s) in your story? (Is there a protagonist in your story ─ someone who is driving the action and/or someone with whom your audience is likely to identify? What are the motivations of the characters?)
- What is the plot? (The plot is the causal sequence of events and includes setting and conflict.)
- What evidence do you have to assert your claims?
- Based on the story you crafted, what is the best medium through which to share it? Video, written narrative, shorter visual image pieces, a combination, others?
- If you are using visuals in your narrative – are they appropriate? Do they support the story you are trying to share or detract from them?
- How will you make your target audience(s) aware of the story?
In Conclusion
A good story is easy to read, introduces a problem, and shares how the problem was solved, highlighting the role of the institution in addressing the problem. We need context and a story because evidence gives stories substance, but stories give evidence meaning. Our stories can be our context, our histories, our missions, our organizational saga – it is how we see the world and why we do what we do.
An evidence-based story pushes back on assumptions, creates narratives that will resonate with internal and external stakeholders such as students, faculty, and employers, and most of all tells an improvement story. In fact, an evidence-based institutional assessment story is a way to bring together improvement and accountability in a meaningful mechanism.
***
For more detailed information on this topic, watch the Using Elements of Evidence-Based Storytelling to Tell Your Assessment Story webinar recording.
***
About Dr. Gianina Baker
Dr. Gianina Baker is currently the Assistant Director at the National Institute for Learning Outcomes Assessment, co-located at the University of Illinois Urbana-Champaign and Indiana University-Bloomington. As Assistant Director, Dr. Baker researches, writes, and presents on all topics related to the assessment of student learning outcomes on college and university campuses. Her main research interests include student learning outcomes assessment at Minority Serving Institutions and community colleges, access and equity issues for underrepresented administrators and students, higher education policy, and assessment of student learning in athletics. Prior to this position, she served in roles of Counselor, Director of Institutional Research, and Director of Institutional Effectiveness & Planning, all at Richland Community College in Illinois. She holds a Ph.D. in Educational Organization & Leadership with a Higher Education concentration from the University of Illinois, a M.A. in Human Development Counseling from Saint Louis University, and a B.A. in Psychology from Illinois Wesleyan University.
Want to see how Weave can help you simplify your assessment processes?
Fill out the form below for a personalized demo. You’ll receive advice on your current accreditation system, as well as an overview of how Weave can simplify your workflows, improve collaboration and make your assessment and accreditation process a success.
