Strategic Evaluation
A paradigm shift in evaluation thinking
Duignan’s strategic evaluation (Duignan: 1997, 2004) approach represents a shift in evaluation thinking away from just focusing on evaluating individual programs to thinking in terms of meeting the strategic knowledge needs of an organisation, sector, region or country. Of course, evaluating individual programs is still part of the strategic evaluation approach, but only when the programs being evaluated have been identified as answering high-priority strategic questions for the for the organisation, sector, region or country.
The demands of making the conceptual shift to a strategic evaluation perspective should not be underestimated. It is best seen as a multi-year process involving changes in: stakeholders’ thinking; evaluation’s implicit theoretical underpinnings; organisational and institutional culture; organisational values; governance; strategy formulation; resource allocation; professional and sector capacity building and evaluators’ on-the-ground practice.
The strategic evaluation approach is particularly useful for anyone working at an organisational or sector level who is charged with evaluating organisational-wide, sector-wide, region-wide, country-wide or even multi-country-wide activity composed of a number of different individual programs.
'Evaluation should focus on meeting the strategic knowledge needs of an organization/sector not just evaluating a individual program'
Currently much evaluation planning starts from the point of view of working out how to ‘evaluate an individual program’. In fact, the evaluation discipline is often referred to as ‘program evaluation’. Strategic evaluation turns this orientation on its head and starts from the point of view of attempting to identify the strategic knowledge needs of the organisation, sector, region or country in which a particular program is embedded. Once these knowledge needs are identified, one can work back from them and determine what type of evaluation is appropriate for each individual program. Not all programs merit the same level of evaluation effort. Programs following well established intervention types which have been evaluated well in the past are usually a lower priority than more innovative and experimental programs that speak to current organisational or sector needs.
Basic toolkit needed to start doing strategic evaluation work
Before we turn to looking in detail at what we are trying to do in strategic evaluation, it is worthwhile itemising the basic toolkit which is needed for working successfully within a strategic evaluation approach. This involves a mix of conceptual and practical tools.
First, from a conceptual point of view, for strategic evaluation we need a theoretical approach which does not see evaluation as isolated from other organisational, sector, regional and country-level tasks. These tasks include: outcomes identification; strategic planning, prioritization, indicator monitoring, performance management, commissioning, delegation and contracting. This is why strategic evaluation is best viewed as a sub-set of outcomes theory - the theory that deals in an integrated and cross-disciplinary way with identifying and acting to achieve outcomes of any type in any sector (Duignan, 2009).
Second, for strategic evaluation we also need an approach to evaluation which does not just focus exclusively on any one single ‘type’ of evaluation.
We need to be open to: implementation (developmental and formative), process, impact and summative evaluation as, and when, appropriate. Evaluation needs to be conceptualised, and ideally evaluation funding provided, in a way that allows you to potentially allocate evaluation resources at the appropriate point in a program’s life-cycle. The type of evaluation to use will depend on the particular developmental stage of a program interacting with the priority sector knowledge needs of the sector in which the program is located. This is in contrast to evaluation funding just being ring-fenced, as it sometimes is, to a particular type of evaluation. For instance, in some instances evaluation funding is just available for impact evaluation but not for implementation evaluation (optimising a program’s implementation).
Third, for strategic evaluation we ideally need a standardised way in which evaluations of individual programs are undertaken (where such program evaluation has been prioritised within the strategic evaluation approach). If information is going to be fed up to an organisation or sector level, there should be some consistency in the way that such information is collected. This is in contrast to program-level evaluations being been done in a wide range of different, somewhat idiosyncratic ways.
Fourth, given that thinking about organisational or sector knowledge needs is central to the strategic evaluation approach, the approach requires a way of coordinating, integrating and representing organisational or sector strategy in an integrated, scalable and accessible fashion. This is necessary because thinking about possible strategic direction is a prerequisite for identifying strategic knowledge needs for an organisation, sector, region or country.
Fifth, using a strategic evaluation approach we need to be able to aggregate previous learnings relevant of what is being attempted in the organisational, sector or country-level strategy. This involves having tools for knowledge management around aggregating, summarising, storing and extracting previous findings about ‘what works’ from research and evaluation and current organisational, sector and regional intelligence. As part of this it is helpful to have a standardised way of specifying outcomes and being able to link evidence and previous learnings to this outcomes structure so that it can be directly used for evidence-based practice.
Sixth, there also needs to be evaluation capacity-building at all levels - from a high-level overview of evaluation provided for governance, management and stakeholders right down to enhancing specific technical skills for those doing evaluation work.
Seventy, strategic evaluation is not an evaluation approach that is in competition with other evaluation approaches. Its key feature is changing the viewpoint on how we think about evaluation. This is a move from seeing it as a program-centeric activity to making it an organisational, sector or even country-level activity. It draws on, and is consistent with aspects of a number of other evaluation approaches ranging from utilisation-focused evaluation to indigenous evaluation approaches.
Further elaboration of these points
Coordinating, integrating and representing strategy
As noted above, if we are going to use the strategic evaluation approach within an organisation, sector, region or country, we are going to need to integrate and collate the underlying strategy that is being pursued. Often we will be working in organisations, sectors, regions or countries where strategy has not yet been integrated or has only been integrated in a partial, or fragmented manner.
When deploying a strategic evaluation approach, one of the first steps is to bring together, as best we can, the underlying strategic underpinning of the organisation, sector, region or country that we are focusing on. Often ’strategy' will consist of different snippets of strategy in different formats in different places. This is in addition to that which is contained within any available formal strategic planning documentation.
One way that strategy can be collated and integrated is to use an approach that Duignan has specifically developed for this purpose. This is visual strategy modelling. This approach consists of bringing together a group of people knowledgeable about an organisation or sector strategy into a workshop. They bring to the workshop all of the strategic documentation they can locate regarding their organisation, sector etc. This may be in various formats, for example: one or more strategic plans; outcomes lists; targets; vision and mission statements; indicator lists; outputs lists; literature reviews etc.
A visual outcomes modelling process is then used to build a visual drill-down strategy diagram summarising the underpinning strategy that is being pursued by the organisation or sector. A visual strategy diagram shows all of the high-level outcomes being sought and the steps it is believed need to occur so as to achieve them. Using an approach such as this means that you can quickly develop an integrated strategy overview, plus detail, of the organisation or sector strategy. This then lets you have both a helicopter and drill-down view of your overall strategic direction that you can use to work out your key strategic information needs as the basis for deploying the strategic evaluation approach.
Aggregating, summarising, storing and accessing current organisational, sector, regional or country knowledge
Strategic evaluation requires effective knowledge management of research and evaluation findings plus organisation, sector or regional intelligence. This information needs to be aggregated, stored, summarised and kept up-to-date. It then needs to be provided in an accessible format for those who are involved in deliberating on what organisational, sector or country strategic knowledge needs are. This requires knowledge management databases and ways of interfacing them.
Evaluation capacity building at all levels
Evaluation capacity needs to be built at all levels of the organisation, sector or region in which the strategic evaluation approach is being applied. This will involve developing a common language for describing evaluation so that parties do not ‘talk past each other’ when discussing evaluation. This language for evaluation needs to be rich enough so that it allows for resources to be allocated to whatever type of evaluation (implementation (developmental and formative), process, impact and summative evaluation) needed in order to answer a particular strategic knowledge need.
An essential part of strategic evaluation is to create realistic expectations amongst stakeholders about what is, and what is not, appropriate, feasible, affordable and credible in terms of impact evaluation. If expectations are unrealistic the result can be that evaluation resources can be misallocated and wasted by just being focused on individual impact evaluations of individual programs without considering the priority of that particular evaluation spend. This is not to diminish the importance of individual program impact evaluations, just to say that they should always be undertaken with an awareness of the current strategic knowledge needs of the sector in which the program is being run. Duignan (2008) has developed a tool that allows you to assess the appropriateness, feasibility, affordability and credibility of any proposed impact evaluation designs.
Ongoing organizational, sector, regional or country-wide dialogue
Ideally a strategic evaluation approach will seek to have in place regular setting (workshop, meeting, consultation) where parties from organisations, sectors, regions or countries can have input into identifying what are current strategic knowledge needs and have information reported back on findings. Typically this consists of periodic meetings, workshops, two-way communication and similar consultation processes etc.
Having evaluation models which can deal efficiently with 'distributed interventions'
One consequence of strategic evaluation’s shift from ‘program-centric’ thinking to thinking from an organisation, sector, region or country-wide perspective, is that one starts thinking in terms of ‘distributed interventions’. This is in contrast to continuing to view interventions as just a series of separate programs.
It is often the case that a central agency of some sort is wanting to fund a large number of interventions which are focused on similar outcomes to be implemented in a range of different organisations, sites or localities. What is the most efficient approach to evaluating such muti-site initiatives? It is costly to, for instance, attempt impact evaluation at all of the sites. From a strategic evaluation perspective it is useful to have an approach for undertaking evaluation cost-effectively in such situations.
Duignan has developed an approach for undertaking implementation, process and impact evaluation of a set of similar initiatives being run at different sites.
Relationship between strategic evaluation and other evaluation approaches
There are a number of different evaluation approaches (Duignan, 2003), these include: systems thinking evaluation; utilisation-focused evaluation; empowerment evaluation; stakeholder-based evaluation; realist evaluation; goal-free evaluation; naturalistic or 4th generation evaluation; theory-based evaluation; and indigenous evaluation (e.g. Kaupapa Maori evaluation). Strategic evaluation has elements in common with a number of these approaches as is described below.
In terms of its relationship to systems thinking evaluation, strategic evaluation takes a broad systems approach in the sense that it focuses on the wider system in which a particular program is located rather than just limiting thinking to the evaluation of a specific program. However systems thinking is a discipline in its own right, with its own formal language and concepts. Any aspects of systems thinking in this formal sense can be used within a strategic evaluation approach. Strategic evaluation is not limited to, or tied to, just using systems thinking technical language and concepts.
Along with utilisation-focused evaluation, strategic evaluation is preoccupied with the users of evaluation. Strategic evaluation is very much in the spirit of utilisation-focused evaluation in that it is focused on evaluation being driven by how people will use evaluation results. In practice, the utilisation-focused evaluation approach is often deployed when focused on evaluating a specific program. It can also be used within the strategic evaluation approach once it has been determined that a particular program is a high-priority for evaluation.
In the same way, stakeholder-based evaluation focuses on stakeholder information needs and has much in common with strategic evaluation. It again is often focused on evaluating a specific program and can be used within the strategic evaluation approach once a particular program has been identified as high-priority for evaluation.
Realist evaluation is an approach to evaluation that puts an emphasis on finding out; ‘what works, to what extent, for who, in what contexts, and how?’ From the point of view of strategic evaluation, having this level of granularity in evaluation findings is a great ideal to strive for. Realist evaluation also moves the emphasis from evaluating ‘Program X’ to evaluating the components inside it. Strategic evaluation can be seen as always attempting to identify what are the priorities from the large ‘shopping list’ of what could potentially be found out if one had the resources to adopt a fully realist evaluation approach.
Goal-free evaluation is an approach which starts from the proposition that specific programs may not have captured the outcomes they should be ‘really’ seeking related to their set of stated goals or the domain in which they are working. It is then a matter of working out what the program’s ‘real’ goals should be. This approach has a lot in common with strategic evaluation in that in order to do this one needs to look at a sector-wide perspective on what are the high-level outcomes that are being sought by a program. You then work back from this wider perspective to define the ‘real’ goals against which a program should be being evaluated. Within strategic evaluation this is done by developing a comprehensive strategy diagram of what it is that an organisation or sector is seeking by its activity. From this overall organisational or sector model, one can work back to identify the role of a particular program in achieving outcomes and evaluate whether it is, or is not, doing so.
Naturalistic or 4th Generation evaluation is an evaluation approach with a particular philosophy of science about methodology (constructivist rather than positivist) and an emphasis on qualitative approaches. Strategic evaluation is open to the selection of methods to suit the evaluation task and methods from Naturalistic or 4th Generation evaluation that can be used with it.
Theory-based evaluation puts an emphasis on looking at the wider context in which a program is being implemented and emphasises the importance of surfacing program theory - the theory of change (intervention logic) that it is believed will lead to improvements in outcomes. Both of these perspectives are consistent with the way in which strategic evaluation is implemented.
Indigenous evaluation (e.g. Kaupapa Maori evaluation) takes a values, philosophy of science and cultural perspective based on indigenous views of the world. As a consequence, it looks at the context in which programs take place, for instance: power; resource allocation; institutional racism; and indigenous sovereignty. Taking into account these wider perspectives is the type of thinking that the strategic evaluation approach seeks to encourage. The key question indigenous evaluation raises when using the strategic evaluation approach is: ‘whose knowledge needs are priorities for being answered?’
Summary
The strategic evaluation approach has much to offer to those who are working from an organisation, sector, region or country-wide level. This is because its view-point starts from such a level rather than just focusing on the lower-level issue of evaluating individual programs one by one.
Dr Paul Duignan, mentors, runs workshop and consults on evaluation design for organisations, sectors, regional and government level initiatives informed by the strategic evaluation approach.
Duignan, P. (2009). Using Outcomes Theory to Solve Important Conceptual and Practical Problems in Evaluation, Monitoring and Performance Management Systems. American Evaluation Association Conference 2009, Orlando, Florida, 11-14 November 2009.
Duignan, P. (2008). Encouraging Better Evaluation Design and Use Through a Standardized Approach to Evaluation Planning and Implementation. 8th European Evaluation Society Conference , Lisbon, October 2008.
Duignan, P. (2004). Outline of the Strategic Evaluation Approach. Presentation to the Annual Conference of the American Evaluation Association, Atlanta, Georgia, USA. 3-6 November 2004.
Duignan, P. (2003). Approaches and terminology in programme and policy evaluation. In Evaluating Policy and Practice: A New Zealand Reader. N. Lunt, C. Davidson and K. McKegg. Auckland, Pearson Education.
Duignan, P. (1997). Evaluating health promotion: the Strategic Evaluation Framework, D.Phil., University of Waikato, Hamilton.