Evaluation 'the key' to traction on hard policy challenges

article-body

The Productivity Commission has taken a strong stand for policy evaluation to identify programs that work, and why, so governments can stop wasting time and money on those that fail.

In its recent report on combined efforts of Australian governments to reduce socio-economic disadvantage in the Aboriginal and Torres Strait Islander population, the commission points out that while targets are good, it helps to have an empirical understanding of the most effective and efficient ways to meet them.

“The critical role that robust policy evaluation could, and should, play in improving outcomes for Aboriginal and Torres Strait Islander Australians is widely acknowledged,” the report states as one of its key points.

Indigenous affairs is certainly a policy area where a large number of different programs have spent a lot of taxpayers’ money over decades attempting to achieve the same general outcome, now defined as Closing the Gap by the Council of Australian Governments. The commission argues that “setting targets and monitoring outcomes” should not be the main focus:

“Knowing more about what works and why is the key to designing policies that achieve positive outcomes for Aboriginal and Torres Strait Islander Australians.”

Leaving aside the specific terms of reference that gave rise to the report, the sentiment could apply to most long-standing challenges that successive governments all try to tackle year-in, year-out.

The PC acknowledges it can be challenging to evaluate the success of government programs but argues that “a much stronger evaluation culture in the Indigenous policy area should be promoted” and suggests:

“Options for invigorating evaluation include: an overarching review of policy evaluation in the Indigenous area; COAG committing to evaluating policy settings in a target area or a sub-set of policies in a particular area (say education); and adding a procedural, evaluation-focused target to the Closing the Gap initiative.”

The second option — starting with just one area of the national partnership like education or health — could provide an “exemplar of good evaluation practice” for other areas.

The third “might, amongst other things” commit governments to identify:

• policies that are directly relevant to each of the Closing the Gap outcomes targets
• those policies evaluated since the inception of the Closing the Gap initiative, or explicitly scheduled for review prior to targets expiring
• the proportion of past evaluations made publicly available
• the proportion of past evaluations showing the policies in question to have been worthwhile and cost-effective
• whether all new policy initiatives have a built-in evaluation mechanism

Knowledge imbalance

The gap that COAG has agreed to close is very well understood; much better than the effectiveness or otherwise of the specific programs that seek to accomplish the frustratingly elusive goal. The report notes that trying to tell and re-tell “the story of Indigenous disadvantage” is likely to result in diminishing returns. It adds:

“There is also a risk that efforts to better map the disadvantage landscape, and to report more extensively on changes in high level outcomes, will deflect attention from policy evaluation — especially as the evaluation of specific policies can be much more challenging than simply reporting on the collective impact of a gamut of policy and non-policy factors.”

The PC report says the biggest challenge facing evaluations is separating the effects of specific policies from other influences on the outcomes in question. About 80% of expenditure to improve the wellbeing of Indigenous Australians is delivered through programs that also support non-Indigenous Australians.

Any assessment of what works, and what does not, “must therefore encapsulate ‘mainstream’ as well Indigenous-specific policies and programs,” the report concludes, which sounds tricky, but is not impossible:

“At the very least, evaluation should help to identify approaches that have been largely ineffectual or very costly in terms of the improvements delivered. Robust evaluation can also help to guard against a ‘baby and the bathwater’ risk in the targeting approach.

“That is, where an outcomes target is not met, effective as well as ineffective approaches may be discarded as part of efforts to do better.”

Uncomfortable duty

Unfortunately the adversarial nature of democratic politics can discourage evaluation. An unflattering assessment of a program could be of great value to the nation, but also damaging to whichever current or former ministers are responsible if it ends up in the media.

Prime Minister Malcolm Turnbull acknowledged that risk on taking office, but says its something politicians just have to accept:

“There are many different routes you can take. You’ve just got to choose one, or a series of routes, and then recognise that you must constantly monitor them and adjust if they’re not performing as well as you think. This is a critically important point. When governments change policies, it’s often seen as a backflip or a backtrack, or an admission of error. That is rubbish. We’ve got to be agile, all the time.”

For public servants, diligently carrying out the elected government’s policies includes making an effort to keep information that could undermine its ability to govern — like that which demonstrates the failure of policies — out of the public eye. This, in combination with freedom of information laws and the potential for leaks, might lead to a reluctance to conduct objective evaluation in the first place.

Open data, for example, is considered a risk by some federal public servants because it “could lead to unfavourable findings about policy effectiveness”, according to the Department of the Prime Minister and Cabinet’s recent data management review. Just this week the department’s outgoing secretary, Michael Thawley, expressed the view that FOI laws are a “risk to good government”.

While an elected government deserves the full support of the public service in carrying out its mandate to govern, citizens also deserve empirical evidence of how well their money has been spent in the past, to inform their choice of who should receive that mandate in future.