Important notice – Victorian Government elections

As the Victorian State election will be held on 27 November 2010, the Victorian Government assumes a caretaker role from 2 November. During the caretaker period, content will only be added to this page in accordance with the caretaker conventions.

What is best practice monitoring and evaluation?

Some time ago I mentioned that we were researching what’s considered best practice for an organisational performance monitoring framework. We’ve spent some time talking to people about what’s worked in their organisations as well as doing desk top research.

An important discovery is that there is no single best practice, so for Sustainability Victoria, best practice monitoring and evaluation

• reflects the context of the organisation
• links an organisation’s vision and goals to program and project delivery
• is everyone’s business – a core component of everyone’s job
• is built on a culture of inquiry, learning through success and failure, and adapting that everyone subscribes to
• is evidence based and directly informs decision making at all levels

In practice, this means

• leadership and an on-going commitment to monitoring and evaluation
• building the capacity of all staff
• shared meaning including a common language understood by staff and stakeholders
• in-house expertise and support
• a standardised framework such as outcome or program logic
• an appropriate data management system

We’re very pleased to share our findings in the report below and welcome your comments!

What is Best Practice M&E – Research Findings

A draft framework

The project team developing the Performance Monitoring Framework have been busy on two fronts. Firstly, drafting the structure and content of the framework and secondly, developing monitoring and evaluation plans for 12 projects that will ultimately test the framework.

The structure of the framework (see below) reflects the links between the everyday activities that staff undertake to deliver projects and the ultimate outcomes that Sustainability Victoria is set up to achieve – environmental, social and economic benefits for Victorians.

By staff setting up meetings, responding to calls and managing projects on a day to day basis, services are provided to Sustainability Victoria’s stakeholders. These services include

• providing advice to a householder about improving the energy efficiency of their home
• providing tools to local Governments to help them track their energy and water use, and
• training plumbers to provide sustainable options to their customers.

These services are described broadly as outputs and lead to the people who receive these services having the capacity to live and work more sustainably by using fewer resources.

The people who receive these services have greater knowledge and capacity to make sustainable choices in their own lives, at work or at home. These changes are the outcomes of the projects that Sustainability Victoria delivers.

Those sustainable choices at work or at home lead to

• a reduction in energy and water use and less waste being produced (eg. more recycling)
• financial savings (eg. lower energy bills)
• personal benefits (eg. greater comfort at home)

And these are the environmental, social and economic outcomes that Sustainability Victoria strives for.

The content of the framework is indicators that reflect the results at each tier of the framework. For example, a project that assists householders to reduce their energy and water use is mapped out as

• Activities – Sustainability Victoria staff manage a service for householders that involves retrofitting homes and providing advice to reduce energy and water consumption
• Output – a service is provided that results in 2,000 households being retrofitted and 2,000 information packs of ‘sustainable tips’ being distributed to householders
• Outcome – homes are retrofitted to reduce drafts and improve thermal efficiency and householders implement some of the ‘sustainable tips’ (eg. shorter showers)
• Environmental Outcome – householders use less energy and water
• Economic Outcome – householders save on energy bills
• Social Outcome – householders live more comfortably in their homes

The framework is a means to understand and measure the outcomes of our work. We are keen to hear your thoughts about our approach.

Great resources

Our ongoing research of what’s best practice in monitoring and evaluation has revealed some truly fabulous on-line resources.

In the field of developing evaluation frameworks and associated tools, one of the most practical resources is outcomes central.org. It offers guidance on developing an outcomes framework, example outcome models to refer to, video demonstrations, downloadable tools and software to visualize outcome models. And all material is free!

Another site that offers some fascinating insights is Learning for Sustainability. It provides practical assistance to people who work with communities to encourage the adoption of sustainable practices. There is a wide range of topics covered including social learning, research methods, frameworks, guides and tools and a description of current approaches to the evaluation of behaviour change programs.

Both resources are very relevant to the work that Sustainability Victoria does and are informing our development of the Performance Monitoring Framework. I think there is much here for others working in sustainability, evaluation and many other fields too.

Testing the Framework

The implementation of an organisation wide framework to guide evaluation requires some practical testing to ensure that the broad approach is valid and contributes to decision making at the project level as well as strategically.

We will be testing Sustainability Victoria’s Performance Monitoring Framework in 2010-11 via the implementation of standardised Monitoring and Evaluation (M&E) plans for a suite of projects. These plans will follow the structure of the framework – they will outline how outputs, outcomes and impacts will be monitored for each project. The plans will also contain indicators for each project and the selection of these indicators will be guided by the framework.

The suite of projects for which M&E plans will be developed has been defined based on criteria developed by an internal Reference Group. The suite of projects represent:

• each of Sustainability Victoria’s Strategic Goals
• the types of projects that Sustainability Victoria delivers (policy influence, capacity building, providing incentives, providing information / advice / tools and demonstrating sustainable practices)
• the range of environmental, social and economic benefits that Sustainability Victoria provides
• different life stages of projects (i.e. pilot, commencement, mature, completion), and
• Sustainability Victoria’s broad stakeholder groups – Households, Community, Government and Business.

Evaluation is about learning

The Show Me The Change Conference was on in Melbourne this week. About 180 of us met like minded people, exchanged ideas and faced the challenge of evaluating behaviour change. There was much discussion and sharing of experiences and since the conference ended I’ve been thinking about the most valuable lesson that I took away. It was that evaluation is much more than reporting outcomes. Its real value is in inquiry, learning and adapting – a timely reminder in the program evaluation journey.

A framework for everyone

A key driver for the development of the Performance Monitoring Framework is to improve the organisations understanding of the outcomes of projects and programs and to use that understanding to adapt management and strengthen decision making.

The intent is that everyone at Sustainability Victoria will be a contributor and user of the framework and the lessons that come from its implementation. We want everyone to learn from their own and each others experiences and to be empowered to make better decisions.

So Board members, the Executive Team, Managers and Project Managers will all be involved in some way. We also intend to discuss the framework with some of our key stakeholders who may benefit from its implementation.

To ensure the framework is of strategic and practical use, two groups have been established to guide its development. A Board sub-committee is providing high level advice and guidance at key milestones throughout the project.

There’s also an internal reference group consisting of experienced staff from across the organisation with a real interest in evaluation and how it can be used to adapt management. The group is ensuring the framework is applicable across the broad range of sustainability projects that Sustainability Victoria delivers. Members are also promoting the benefits of evaluation and are contributing their own experiences from individual project evaluations. We don’t want to reinvent the wheel!

Researching best practice

We’ve been busy within Sustainability Victoria researching what’s considered best practice for an organisational performance monitoring framework. We’ve cast the net wide and are contacting organisations, agencies and academics in Europe and North America as well as speaking with other Government departments in Victoria.

Some particularly interesting reports include:

• A report by Cullbridge Marketing and Communications on behalf of Environment Canada - Best Practices in the Monitoring and Evaluation of Program Impacts (2002)
• The UK Department of Environment, Food and Rural Affairs – A Framework for pro-Environmental Behaviours (2008)
• Transport Canada – Results-Based Management and Accountability Frameworks

Initial findings suggest there is no one best practice example of an organisational-wide framework, although the Cullbridge Marketing and Communications report presents a great synopsis of elements of best practice from 15 case studies in the field of social marketing.

We’re finding instead that there are many guidelines, templates and recommendations on how to design an evaluation for a single program. Some key principles include:

• clearly define the purpose of the evaluation at the commencement of the evaluation (impacts, effectiveness or efficiency)
• identify who will use the results (project managers, senior management or stakeholders)
• identify who will be involved (staff, peers and / or stakeholders)
• define the performance indicators
• identify what information should you capture
• clarify how the results will be communicated
• ensure the results are communicated to all parties involved

Although these principles are broadly defined for designing the evaluation of a program they are equally as applicable to the design of an organisational framework.

We’re working on incorporating these findings into Sustainability Victoria’s framework as well as ensuring that the framework:

• is an integral part of a planning, monitoring and evaluation cycle
• has standard monitoring and evaluation language
• incorporates quantitative and qualitative data (the latter is crucial to understanding behaviour change!)
• considers a baseline understanding of the current situation in order to detect change
• incorporates a means to prioritise monitoring and evaluation or projects

Healthy Parks Healthy People Congress – next week

In case you haven’t heard, the inaugural Healthy Parks Healthy People Congress is on next week in Melbourne.

Having looked at the website it’s obvious that key themes of the Congress are relevant to Sustainability Victoria’s current challenge of understanding (and measuring) the social and health benefits that our sustainability programs deliver.

I particularly like Sir Gustav Nossal’s, the Congress Patron, summary of the challenge:

“The worlds of sustainability and health have developed almost independently of one another, despite the best intentions of each to address issues of common concern. It is time to reunite them in the interests of our quality of life”.

At Sustainability Victoria, we’re attempting to unite those worlds by asking if our programs result in participants

• having a greater sense of well-being and self-worth by taking action and responding to climate change
• feeling connected to their community by participating in sustainability programs where they meet and work with their neighbours
• having a greater sense of liveability resulting from implementing sustainable practices.

I’d like to hear from anyone attending the Congress about practical examples of how to unite the worlds, and measure the links between, sustainability and social / health benefits.

A common language

In my last post I mentioned the introduction of a common language for monitoring and evaluation across Sustainability Victoria. 

It may seem like a simple thing that everyone understands what a project is, what an objective is and what an outcome is, however that isn’t always the case. 

If one person focuses on giving a workshop as the objective of their project where-as another identifies their objective as a 10% reduction in energy use across Victorian homes than applying a consistent approach to evaluating those projects against their objectives is tricky. 

So, I’m in the process of defining some project management terms that are in use across Sustainability Victoria: program, project, objective, outcome, output, deliverable, activity and input.  

I have an opinion about each of these (I think an output and deliverable are the same thing) and have found some useful online references. 

Does Your Project Make a Difference?

One particularly useful document produced by the then NSW Department of Environment and Conservation in 2004 is Does Your Project Make a Difference? 

Has anyone used this document? 

I’m keen to hear others experience at developing and in particular embedding a common language across an organisation.