Supporting the performance monitoring framework

If you’re following Sustainability Victoria’s program evaluation journey you will have read in my last post about the environmental, social and economic impacts that the Performance Monitoring Framework will measure. 

We’re developing the framework such that it will be supported by

• the introduction of a common language of monitoring and evaluation across the organisation – essentially the use of common terms (such as ‘objective’ or ‘outcome’) and the clear definitions of those terms
• criteria to identify which projects will be fully monitored from activities to impact – an essential component of any monitoring framework given the cost of measuring impacts
• a costing model that will estimate the investment required to adequately measure the impacts of projects – to allow project managers to allocate a realistic proportion of their budget to monitoring and evaluation
• a guidelines document that steps staff through the framework and explains how to apply the framework to the projects they manage

The costing model is a particular challenge and I’m keen to see how other people have gone about estimating the investment required to implement an impact / outcome level monitoring framework.

A performance monitoring framework for Sustainability Victoria

In previous posts I’ve explored what evaluation is about in general and I’m keen to get to the nuts and bolts of what Sustainability Victoria is currently doing. We’re developing a Performance Monitoring Framework – essentially a set of indicators to capture the environmental, social and economic outcomes of our programs.

Our programs are diverse and include encouraging householders to use less energy, working with schools to promote recycling and funding the development of sustainable infrastructure.

The challenge is knowing that we do these things successfully and that we achieve the outcomes we want.

By outcomes I mean integrated triple bottom line benefits for Victorians.

Some of the environmental benefits that Sustainability Victoria knows it provides are

• a reduction in green house gas emissions (ride instead of drive)
• a reduction in energy use (switching off the lights)
• a reduction in solid waste that goes to landfill (using less of our limited resources)
• recovery of solid waste from landfill (recycling where-ever possible)
• a reduction in water usage (taking shorter showers)
• an increase in the use of renewable energy (solar, wind and wave)

We are working towards understanding the social benefits we contribute to, such as people

• experiencing greater comfort in their homes after implementing simple sustainable practices such as sealing out draughts
• having a greater sense of well-being by taking action and responding to climate change
• feeling more connected with their community having participated in events that encourage sustainable living through schools and sporting clubs

And, we’re investigating flow-on economic benefits such as

• a demand for professionals with green skills
• greater industry efficiencies after implementing sustainable practices
• expanded markets for sustainable products (such as compost from recycled garden waste)

The Performance Monitoring Framework project is about finding the right indicators and monitoring methods to capture these outcomes.

I’m keen to hear how other people have gone about understanding their outcomes and the triple bottom line impacts they have.

Show Me The Change National Conference

For those of you who haven’t come across it yet, there is an exciting international conference happening in Melbourne this coming May: Show Me The Change, complexity & the art of evaluation.

The conference is co-sponsored by Sustainability Victoria and will address an area of evaluation that is very complex and very relevant to the work that Sustainability Victoria does.

Sustainability Victoria is about encouraging people to live more sustainably everyday by using less energy, water and materials. To deliver this outcome, Sustainability Victoria delivers workshops, public forums, advertisements, training, accreditation, competitions, rebates, brochures, seminars and a lot more.

The challenge for managers is to know how effective these activities are at changing people’s behaviour. Do the families, businesses, teams, schools, communities, councils, health workers, researchers, government departments, clubs and individuals that Sustainability Victoria works with live more sustainably as a result of that work?

The Show Me The Change, complexity & the art of evaluation conference is a great opportunity to discuss that challenge and exchange ideas about ‘Evaluation of Behaviour Change’ for sustainability.

The conference website includes a blog with some interesting articles about the complexity of evaluation, the importance of people in the process and contains links to other useful evaluation sites.

Is anyone thinking of going to the conference?

Elements of an effective evaluation culture

From an organisational perspective I think there are some key elements to an effective evaluation culture.  

Firstly there needs to be an appreciation that evaluation is important. It sounds simple but frequently evaluation is viewed as a luxury and a nice to do if you have the time at the end of a project. Ideally, monitoring progress occurs throughout a project with results prompting project managers to question:  

• Was this where I expected to be?
• Is this a good result?
• Why is this not what I expected?
• What should I change to get back on track?
• Do I need to reconsider my objective?
• What have I learnt that can be used elsewhere?

Asking and answering these questions is critical to successful evaluation.  

Another aspect of a good evaluation culture is the availability of clearly defined objectives, tools and processes supported by good information management.  

At Sustainability Victoria, these key elements are summarised as:  

• clearly defined strategic goals and objectives that reflect legislative and policy obligations
• prioritisation tools that translate strategic goals and objectives to programs and projects via the business planning process
• tools that allocate resources to achieve the best possible outcomes
• a means of determining how efficiently and effectively programs and projects are delivered
• a Performance Monitoring Framework of targeted indicators that measure program and project outputs and outcomes against clearly defined objective
• a process of review that identifies why programs deliver or do not deliver desired outcomes and incorporates these findings into future decision making (i.e. the Strategic Plan and Business Plan).

The figure below shows these elements and how they link to one another.  

Some of these elements are well established while others are in their infancy. In particular the Performance Monitoring Framework is a key priority for Sustainability Victoria. It will consolidate a lot of good work already being done and address known gaps.  

Role of the performance monitoring framework in the evaluation culture of Sustainability Victoria

Why evaluate?

In my last post I described what evaluation is in the context of Sustainability Victoria’s work. You may ask why evaluation is important particularly as it requires an additional commitment of time and effort on top of doing the actual work.

There are many good reasons as to why organisations choose to evaluate:

• for transparency and accountability purposes, particularly for stakeholders or shareholders who want to know if and how investments are efficiently and effectively used and the outcomes achieved
• to raise awareness and support for a particular cause or to achieve a particular outcome
• to adapt management and strengthen decision making as lessons are learnt about current work practices
• to understand the impacts or outcomes of programs
• to improve the allocation of limited resources by prioritising work and applying efficiencies
• to support a case for increased funding by demonstrating achievement. 

For Sustainability Victoria the current focus is on understanding the outcomes or impacts of programs and subsequently adapting management and strengthening decision making.

Understanding the outcomes of sustainability programs is a complex task and a project has recently commenced to develop a performance monitoring framework. The framework will consist of indicators that capture the outcomes of programs – changes in people’s behaviour that reflect more sustainable living, subsequent reduction in environmental impacts and related social and economic benefits.

I’ll be sharing some of the challenges, learnings and questions that arise as I develop this framework. I invite comment and advice as the journey unfolds.

What is program evaluation?

In the About page of this blog I’ve described two different types of evaluation, effectiveness evaluation (knowing that you achieved what you set out to achieve with your program) and outcome evaluation (understanding the impacts of a program).

Successful effectiveness evaluation requires:

• knowing what your objective is
• measuring or monitoring your progress towards it
• getting to the end and knowing that you achieved your objective (or maybe not)
• being able to answer the question what worked (or what didn’t)
• applying that knowledge next time to do better.

Part of Sustainability Victoria’s journey is to work towards:

• clearly defining objectives through the business planning process
• establishing and using a performance monitoring framework
• reviewing whether project objectives are achieved
• adapting management according to findings.

The other component of the journey is identifying the impacts of programs. This is particularly challenging given the diversity of Sustainability Victoria’s activities and the organisations purpose – to show the way to using our resources more efficiently and reducing our everyday environmental impacts.

In your own searches you’ll discover many other definitions and approaches to evaluation.

It’s worth a quick scan of the following environmental, community, volunteer, health, education, aid and development sites to get a broader view of what evaluation is about.

• What is Program Evaluation and Performance Measurement? – US Environmental Protection Agency
• What is program evaluation? – US Department of Health and Human Services
• Evaluation: A Beginners Guide – Amnesty International
• Introduction to evaluation – Professor William M.K. Trochim, Cornell University
• Introduction to program evaluation – World Health Organisation Collaborating Center, University of Pittsburgh
• Monitoring & evaluation – Charities Evaluation Service
• WHAT is evaluation? – Evaluation Trust

I’m yet to find a site specifically focussed on the monitoring and evaluation of sustainability programs so if you happen to own, contribute to or know of one, please post a link.