How to (measure) impact: the Gap Filler Way

Barbara Tańska

Looking for an internship a couple months into a pandemic already seems to be a crazy idea but starting one from Poland, working with a place-based organisation in New Zealand takes it to a whole new level. Fortunately, Gap Filler are into all sorts of crazy ideas, and so we started working together halfway through the memorable year that was 2020. With Gap Filler’s strategy review coming up later in the year, and some work on impact measurement they had been looking to pursue for a while already, there seemed to be just enough work that could be done remotely – and before I knew I was bending my brain around measuring the impact of an organisation that has been unquestionably influential, but often in quite intangible ways! 

In this piece I wanted to share a short summary of the twists and turns of my adventures with Gap Filler’s impact alongside some general lessons about placemaking’s impact measurement I learned along the way.

Why vs. how
Starting with a ‘Gap Filler’s Impact 101’ chat with co-founder and current Director, Ryan, and a broadly scoped ‘literature review’ on placemaking and impact, I quickly discovered that there wouldn’t be straightforward answers this time around. The field, it seems, hasn’t yet adopted the impact discourse so prevalent in community organisations and social enterprises in recent years. Placemakers still seem to have way more questions than answers around impact measurement; not to mention that the currently available answers certainly more often refer to the issues of ‘why’ than ‘how’. The majority of online materials are still focused around convincing placemakers to measure impact rather than providing them with guidance on the actual methods.

The takeaway point from the initial search was, hence, as follows: the popular ‘motivations’ for undertaking impact measurement listed online included ‘securing funding based on impact indicators’ and ‘more effective storytelling’. I was disappointed (and also surprised) to find that more nuanced aims such as ‘internal resource reallocation’ and simply ‘understanding the organisation’s role in the community’ – both at the heart of Gap Filler’s impact project – were rarely mentioned. As the methods and extent or depth of measuring impact usually closely follow the motivations for later usage of the impact ‘evidence’, this initial disappointing discovery hinted that this would be an underdeveloped area.

Theory of Change
Theory of change was the single most popular tool recommended in most of the impact measurement briefs I read. It is a method rooted in planning out a logic that links the on-the-ground work of the organisation (its activities and outputs), with short- and long-term outcomes, and finally with the overarching, often quite general, impact statements. Such trains of thought are often presented as linear ‘impact logics’ or more messy, multi-stranded  ‘theories of change’. Either way, they allow an organisation to identify the quantifiable outcomes of their activities and often act as efficient supportive material for telling the organisation’s story.

Indicator approach
I was definitely not the first person who questioned using impact ‘logic’ as evidence of one’s impact or even as part of an impact ‘plan’. I found another well established impact framework that directly addressed this issue. The indicator approach is based on analysing change in already recorded data, for example existing social research, census data or a local authority’s surveys, which fill the missing link between the tangible and measurable outcomes and the grand impact statements made in some theory of change models. Although I could find no such research about placemaking in Aotearoa (yet?), Prof. Joanna Woronkowicz (2016) conducted multiple studies on American creative placemaking projects, reaching a conclusion that the indicator approach does not effectively reflect placemaking’s social impact. The study questioned whether the ‘soft’ community impacts such as social cohesion, community activation or sense of surprise can be quantified at all, and recommended a more open-ended approach.

The grand experiment
Slightly deflated by the dead ends in my research ventures, I had another chat with Ryan that drew my attention to a couple more important things. What if Gap Filler do not really want to go the traditional way at all? What if a crucial aspect of their complex work is the creative process itself, and its open-endedness and openness to the unexpected? I saw Gap Filler as driven exactly by creativity and experimentation, which was also reflected in the feedback from their community partners I interviewed as a part of the preparations for the strategy review. People told the story of Gap Filler similarly to how Ryan narrated it to me – a story of bending and flexing behavioural norms, playing with form and function of spaces, surprising city dwellers with installations and events… Maybe those were the ways Gap Filler’s impact could be told – with quite unusual language and impact claims rather atypically made around experience and experimentation?

Delving into Gap Filler’s project archives also opened my eyes to how much work was actually happening before the ‘implementation’ stage. Much of their impact was already performed in the community engagement process – via site visits, discussions and workshops bringing different stakeholders together. They embrace creativity as an inherent part of their design processes but also way earlier, not only gathering feedback from but truly experimenting on the possible solutions together with community members.

What can we do then?
While I felt that coming up with an absolutely original super creative tool might be just beyond my capacity (I am infinitely jealous of Gap Filler’s creativity levels), I turned to placemaking’s ‘sister’ fields or what felt the closest to such, namely creative arts, community engagement and advocacy. Although this research was a rather hit-and-miss-intensive initiative, there were certainly a couple resources and voices I found extremely helpful and inspirational. Some of them are listed below this article. Crucially, I felt that people working in those fields understood the indeterminacy of their impacts, embracing creativity and uncertainty as a part of the impact measurement processes. Based on such research, I compiled a guide for Gap Filler’s impact endeavors, the highlights of which I summarised below. Even though they do not add up to a ready-to-be-applied framework, I do hope some of those are applicable within the wider placemaking field and I hope they do also shout: placemaking’s impact measurement is an experiment in itself!

As promised, here are my ‘lessons learned’:

1. The ‘process’ is sometimes at least as valuable as its outcomes.

Much of placemaking’s impact happens way before the ‘places’ are actually ‘made’. The stages of community engagement, consultation with the local authorities, collaborative design, creative ideas and concepts… they all matter and all create impact for the stakeholders involved, their organisations, and those working with them post-placemaking. The materials produced during such consultations (paper notes, mind maps, thought clouds) or even informal chats with people involved are already ways of recording organisational impact. Asking the stakeholders to reflect on their changing interactions with each other might also be a good idea! A good and very open-ended starting point to broaden the categories of what we think ‘counts’ as impact is BYP Group’s report on the basics of social impact evaluation (2018). One graphic I found especially useful was their Continuum of Impact.

The Anti-Form (Creative People and Places, 2016, p. 17)

 As the selection of those ‘sister’ fields and lessons was hugely subjective, this list is far from complete, with surely many more helpful tools or cues still floating in the digital space (do get in touch if you find any!). While the above points still do not add up to any ready-to-use framework, I’m afraid the final answers (if exist) can only be found through impact measurement… if you actually start the experiment!

This article is based on my experience (remotely) working with Gap Filler as an intern in the second half of the crazy 2020. I would like to very much thank Ryan Reynolds, Linda Glasgow, Coralie Winn, Rachel Montejo, Summer Hess, and everyone at Gap Filler, for the work we did together, your wholehearted support, and inspiration!

References

BYP Group. (2018). The Basic of Social Impact Evaluation. Accessed 10.11.2020 from https://www.bypgroup.com/s/RESOURCE-1-basics-of-evaluation-3pcn.pdf.

Creative People and Places. (2016). Evaluation in participatory arts programmes. Accessed 10.11.2020 from https://www.creativepeopleplaces.org.uk/our-learning/evaluation-participatory -arts-programmes.

DIY Toolkit. (2011). Theory of Change. Accessed 17.10.2020 from https://diytoolkit.org/ tools/theory-of-change/.

Woronkowicz, J. (2016). ‘Levitt Music Venues And Neighbourhood Change: Reflections On A Creative Placemaking.’ In Woronkowicz et al. (eds.) Setting The Stage For Community Change: Reflecting On Creative Placemaking Outcomes. Accessed 10.11.2020 from https://levitt.org/ckeditor/userfiles/ images/1478133733_Levitt_white-paper_setting-the-stage-for-community-change_creative-placemaking-outcomes-study_2016.pdf.

Other resources I found helpful:

https://www.fsg.org/publications/guide-evaluating-collective-impact

https://inspiringcommunities.org.nz/ic_resource/measuring-and-monitoring-cld/

https://www.jrf.org.uk/sites/default/files/jrf/migrated/files/1859354157.pdf

https://www.carenederland.org/wp-content/uploads/2018/06/MEL-for-Advocacy-Guidance-2018.pdf