Making Sense of Evaluation: What we’ve learned about data collection, meaning, art, and story.

3 hours ago

5 min read

It's been a big year for evaluation at AKA. If you read our work and know what we do, we are always pushing the boundaries of how to evaluate programs and people and what it all means. We've highlighted some of these efforts in previous blogs, publications, reports, and art. This is the first time we’ve attempted to make sense of evaluation. Like really figure out what it is we are doing, what clients are doing, and what matters.

Sensemaking evaluation is not new. In fact, it’s old, perhaps older than the field of evaluation itself. We looked to the American Evaluation Association to start making sense of sense-making evaluation. This led us to a great blog and some lessons learned about sense-making evaluation. Being new and not knowing much about sensemaking, we looked to lessons from Karl Weick, an organizational theorist and developer of the term “sensemaking.” He wrote a book in 1995 about sensemaking in organizations. His work reminds us that sensemaking is how we as human beings interpret what is happening and determine what it all means. Brilliant, right?

Fast forward 30 years. One of our clients asked us to develop a sensemaking workshop for their staff. We wholeheartedly agreed. We didn't know what to do or how to start. Over a series of calls, Zooms, and in-person meetings, we finally started to make sense of sensemaking evaluation. We reviewed evaluations our client conducted over the last year. We explored meaning, topics, place, and culture. We wanted to step back to see what kinds of information were being collected, what made sense, and what we needed to trash. For example, monthly progress reports and evaluations of staff outreach, training, and technical assistance made no sense. Staff did not always complete the survey. When they completed the survey, we were left with a number. A number to make sense of what had happened. Clearly, we needed to pause and make sure we were collecting the right data to capture our attention, funding agency needs, and community priorities. Here are some steps we followed to develop and host our sensemaking workshop.

Our Steps to Implement Sensemaking Evaluation Workshop

Collect info. We created a Qualtrics survey to collect additional data from our client and their staff to make sense of the sensemaking workshop we were asked to host. If you are considering a sensemaking workshop with evaluators and want to know the kinds of questions we asked, email us, and we will share the Qualtrics survey with you.
Email participants to help them prepare. Hi team! We’re looking forward to being together in person on Friday, September 4. The day will give us a chance to reflect on the past year, celebrate our accomplishments, learn from one another, and spend time together as a team. Before you come, think back over the past year: What conferences, training sessions, or other professional development activities did you attend or help facilitate? What did you share with Tribal communities, Tribal health programs, or partner organizations? Which partnerships or collaborations were especially important to your work? What accomplishment from this year are you most proud of? Think about your own experience: What stood out to you most this year? What surprised you or turned out differently than you expected? What did you learn from the work, the communities we serve, or one another? Was there a moment when you changed your approach or saw things differently? What is one lesson from this year we should carry forward? One Last Thing. Please submit any work photos from the past year to the Google Drive folder. Photos from events, training sessions, community activities, conferences, or other memorable moments are welcome. Thank you for taking a few minutes to reflect and share photos in advance. We’re looking forward to being together, celebrating the year’s work, and hearing everyone’s experiences.
Develop an agenda. Here is ours:

COLLECTIVE SENSEMAKING WORKSHOP AGENDA

Time	Facilitator	Activities
10:00 AM	Initials	Who We Are/Identity – Who we are shapes what we see. Welcome Icebreakers Prayer/ Acknowledgments
10:15–11:00 AM		Social – Meaning emerges collectively. Mapping Activity What stands out to you? Map the significant activities of the year on butcher paper, using both words and visuals. Include what was implemented, what was significant, and what stood out. Use photos shared by the team.
11:00–11:30 AM		Retrospective – We make sense by looking back. Year in Review What have we accomplished? Professional development (e.g., trainings, conferences) Community outreach (e.g., campaigns, distributions) Community products (e.g., reports, dashboards) Programming (e.g., subawards, internships) Personal achievements (e.g., what are you most proud of?)
11:30–11:45 AM		Team Building Activity – Indigenous rock, paper, scissors
11:45 AM–12:30 PM		Cues – Data points that capture attention. Enactment – We shape the story through our questions. Data Stories Discussion Review the Year in Review results. What is the value of our work? What has been the impact for you personally? How has our work affected Tribes, Tribal health programs, and Tribal-serving organizations? What stands out? How do we know?
12:00–1:00 PM		Lunch – Provided
1:00–1:15 PM		Team Building Activity – Yarn activity
1:15–2:00 PM		Peer Learning Circle What have we learned from our work—and from one another—this year?
2:00–3:00 PM		Ongoing – Understanding evolves over time. Looking Forward What is one lesson from this year that we should not lose? Looking at what we learned, what should we keep doing, strengthen, or do differently next year?
3:00 PM		Closing – Prayer and intention

Host the workshop. This is where things get really start to make sense. We drank too much coffee before the day started and were so excited about art and giveaways- it was a blur. Make sure you have food, drinks, and prizes- it creates a memorable experience and positive vibe for everyone in the room. We also spent a fair amount of time outside in the sun, doing team-building activities and creating stories around objects like old T-shirts, rocks, and earrings. Participants were asked to create stories from these found evaluation objects to make sense of them (decolonize data). The point of this activity was to show people the various ways that stories and meaning are created from experience, what people see, what they think they see, and what others tell them they are seeing. We spent time celebrating wins from the past year and asking participants about their greatest accomplishments. This was a time of celebration and amazement. Using a cafe-style format, groups traveled to each large Post-it note and responded by thinking of all of the ways their work created value and impact.
Evaluate. Of course you want to evaluate the sensemaking workshop. Use a mix of photos, quotes, collected sticky notes, data, food, and sentiments to make sense of the day. Make it better for the future. If you want a copy of our sensemaking evaluation, send us an email, we would be happy to share.

Now that we’ve made sense of our work and hosted a sensemaking workshop for others to consider, we have a few recommendations that might help you and your team as you envision, plan, and implement sensemaking.

Speak the language of your intended audience.
Know that the meaning we assign to data (and what is happening all around us) is based on our own lens and experience. What makes sense to me may not make sense to you. That is okay.
Share information when it is appropriate and consider what people need to know to make sense.
Stop collecting data and doing evaluation when you find a story, make sense of things, and have a collective meaning. In qualitative research, we call this saturation.

Everyone makes sense of data through evaluation. Language shapes how we deliver the meanings and messages we create from the data. We must consider what we are doing in this one wild life. We must consider what data is missing from the story. We must be intentional about finding these data in ways that honor all people, places, protocols, and prayers.