The Wartime Films Project: Setting Goals and Intended Outcomes for User-Centered Design Pilots

This post comes from the team working on the Wartime Films engagement project, and is part of a series outlining how NARA is using design thinking to reach new and existing audiences. This project was made possible in part by the National Archives Foundation and a generous gift from an anonymous donor. Read the whole series here.

Last month we looked at how research and analysis have helped us narrow our focus on particular audiences and a subset of relevant content. Today we’ll take a look at how we built our evaluation framework for the project.

Having narrowed our target audiences for the Wartime Films project and settled on WWI-focused content for our engagement efforts, we began to really concentrate on our goals for the pilot. Unlike a traditional publicity campaign seeking media responses, or a management document to commit to deliverables, we’re seeking particular outcomes for the target audiences, as well as ways to measure the impact of our engagement. Outcomes focus on social transformation, and are defined as changes in the behaviour, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly.

Working with Historypin’s research and evaluation team at Shift, we decided on the formula of actors + actions = intended outcome to shape our goals. Recognizing that we’re part of a much larger ecosystem of cultural heritage organizations encouraging discovery and reuse of our national treasures, we focused the formula on answering two key questions : what did we want to see happen in the project, and how could we measure our specific impact in the space through this particular pilot?

Using a widely-adapted, user-centered approach to planning called Outcome Mapping, we’re able to focus on social transformation, particularly how it pertains to the public discovery and creative reuse of primary source materials. For us, the desired “big picture” change is broken down into a series of multiple outcomes that multiple actors can work towards. To build our outcome mapping framework, we first pinpointed not only NARA’s wider goals for access and reuse in the project overall, but for each of our target audiences individually: teachers, local museums, and coders/digital humanists. The audience analysis we carried out in the beginning of the project was key in helping to define these goals, and placing them in the framework helped us organize the actions and results we were hoping to see.

We narrowed down the most important aims for each group and created a spreadsheet to organize our intended outcomes, the activities that can help us reach those outcomes, and methods to measure how effective the actions have been. The outcomes we settled upon for each group focused on issues of awareness, access, and community, each connected to larger organizational goals for NARA (see the National Archives 2014-2018 Strategic Plan).

The next steps of this process involved coming up with initial activities- such as teacher workshops and publishing raw metadata for coders- that can be logically linked to our outcomes. We then created measurements for these activities, which can be anything from surveys and interviews to observations like social media hits, teachers blogging, etc.

By approaching the Wartime Films evaluation from a social research perspective, the key outcomes and ways of measuring those outcomes are aimed at seeing an increase in social engagement. While the activities and measurements for those activities might change over the course of the project, having this framework in place allows us to ensure our overall goals stay consistent.