Last roll of the social impact measurement dice

This week, the Centre for Social Impact (CSI) launched the first part of Amplify Online to the Australian not-for-profit sector, which is a quantitative social impact measurement tool. You can now register to use the tool using your organisation’s ABN number and Registered Organisation name. The benchmarking module of the tool will launch before the end of the year. 

Amplify Online is an innovative, self-service tool designed to make social impact measurement simple. Strictly evidence-based, it provides an accessible pathway for purpose-driven organisations to understand the impact they’re making.  

Australian charities and not-for-profits that take up Amplify Online can expect to save up to $90,000 per reporting project, with $300 million per annum conservatively expected to be saved across the sector. 

I attended the online launch webinar which had over a thousand people registered to attend and around 400 watching it live. After watching the presentation and demo of the new tool in action, here are my thoughts on its potential, some of the hurdles to overcome and a call for all in the sector to make it work.  

After working as a social impact measurement practitioner for nearly 15 years, I truly believe that this is the last roll of the dice to make an accessible and widely used common way of measuring social impact in Australia. 

Why it can work 

• The whole platform has been co-designed here in Australia with the not-for-profit sector with financial backing of the whole social sector. 

• Amplify has been developed by the most reputable academics in social impact measurement at CSI. They have been developing it since 2018 and it has involved large teams of academics, data analysts and software designers. 

• For organisations with a revenue of up to $5 million the tool will be free and pricing for other organisations will be tiered based on revenue. 

• It has in-built artificial intelligence which supports you through the process of building your customised indicators and setup of your Amplify account. Using language analysis on your projects objectives, it is able to suggest potential indicators that align and predicts social outcome you may want to measure.  

• Data collection is included within the system removing the need to use separate survey software. Importantly this includes email, kiosk and offline modes. 

• The indicators are validated measures with a unique scoring system developed by CSI which assesses the rigor behind the measure. 

• The tool automatically does the quantitative analysis of the data collected and provides data visualisation of the results including comparisons to national average and comparative scores where available. 

Hurdles to overcome for success 

• As mentioned, the tool will suggest measures which might align to your needs. However, you still need the skills to select the right measures for your program. 

• The tool has data collection built in but there is still a huge resource demand and skill set to manage that data collection including administration, data protection, consent and research ethics. 

• Benchmarking in general between social programs has many dangers and limitations. There so many variables in implementation, participants and external factors that influence the effectiveness of an intervention. 

• Comparisons based on poor data in an organisation’s system or that you are benchmarking against. 

• The changing and adaptation of measures and questions by users. Already during the webinar there were questions about adapting question text on indicators. To be clear you are then not using a validated indicator and no comparison or benchmarking is possible. 

• There is a risk for organisations in general measuring social impact – what if you don’t like the answers the data is telling you? Do you have the skills to investigate and have confidence in the results? 

• What if the data is wrong based on poor indicator choice, issues with data collection or having a too simplified understanding of why things have changed? 

• Related to the above point, how will most organisations without expert advice know if what they have achieved is causation vs correlation in creating social impact? 

• There will be varying levels of rigour applied in the design of an organisations approach and how the approach is implemented. There is a major of risk of over-claiming impact and miracle jumps from what the program achieves and what impacts it claims to be responsible for. 

My personal observations measuring impact 

Over the last 10 years, there have been some innovative new approaches and tools which has come out. There are many reasons why none of these have succeeded. 

Firstly, their usage has been fragmented across different sectors and there has been a lack of unified agreement to embed them as standard best practice. The honest truth is that none of them have been adaptable enough to align to the needs of such a diverse set of organisations both in relation to their size, resources and outcomes focus. 

Another issue has been the software itself that has been built to manage and report on outcomes measurement and impact measurement – it has always been clunky. No one has had the resources to create a platform that manages the whole process from measurement framework design to indicator selection, data collection, analysis of the data and presentation of results. You ended up buying into a solution that needed multiple other software tools to create the whole production line of a social impact report.  

In my own work, we used to have our survey tool, our quantitative analysis tool and our qualitative analysis tool. It’s a huge financial investment and even in some of my roles where we evaluated multiple programs, some of these tools sat idle for months of the year. 

The cost of these software solutions has also created barriers to utilisation and to be honest, this leads to it becoming about a competitive advantage in your sector rather than about ensuring that the social investment being made in programs is actually having an impact. 

Final thoughts 

Over the past 15 years, my own experiences in measuring social impact range from being the internal lone ranger trying, on the side of their main role, to measure impact, to managing the evaluation of multi-million-dollar behaviour change programs, advising not-for-profits on their social impact strategies and being a provider of one of these previous software solutions for social impact. 

My advice to organisations has generally been that if you are doing something around your social impact then you are ahead of most of your peers. The most important thing is that you know who your audience for measuring impact is and what level of rigour they expect you to use in evaluating your programs. 

My hope for the sector is that Amplify Online is embraced, embedded and established as good practice for all social sector organisations in Australia. 

Strategic Grants builds the capacity of for-purpose organisations to apply strategic thinking to their service delivery and fundraising programs. Find out more.