Grant Hehir: striving for effect, not just efficiency

In a performance audit, the auditor’s objective is to express an opinion whether, in all material respects, the operations of government administration has been carried out economically, efficiently, effectively, in compliance with legislation and accompanying instruments and to recommend ways in which the operations may be improved.

• Economy means the acquisition of the appropriate quality and quantity of resources at the appropriate times and at the lowest cost.
• Efficiency means the use of resources such that output is optimised for any given set of resource inputs, or input is minimised for any given quantity and quality of output.
• Effectiveness means the achievement of the objectives or other intended effects of activities at a program or entity level.

In practice, effectiveness relates to whether:

• the best value for money strategy is chosen;
• that strategy is well implemented; and
• that it achieves the desired outcome.

The ANAO recently examined 277 of its performance audits tabled between 2010-11 and February 2016 for whether the objective or criteria allowed for an assessment of effectiveness, efficiency, and/or economy, and found the predominant focus of audits tabled during this period was effectiveness.  These effectiveness audits examined entities’ existing program strategies, and assessed the extent to which programs were implemented as intended. There was less focus on whether the strategies themselves had been effective in achieving the desired outcome.

The underlying reason for this is a combination of the interpretation of the requirements of the performance auditing standards and the quality of performance information. The performance audit standards require the use of identified criteria to assess the economy, efficiency or effectiveness of an activity. The criteria used by public sector performance auditors therefore generally reflect developments in the performance management frameworks as criteria in the context of a performance audit means reasonable and acceptable standards of performance against which the activity may be assessed. To assess performance against the criteria requires robust information and data about entity inputs, outputs, processes and results. In the absence of quality information and data the auditor is not able to establish sufficient and appropriate evidence (where sufficiency is the measure of the quantity of evidence and appropriateness is the measure of the quality of evidence; that is, its relevance and its reliability) to conclude on economy, efficiency and outcome effectiveness.

Practical, not purist

Despite the predominant focus on effectiveness in ANAO audits, there are examples of where the ANAO has applied practical methodologies to examine efficiency. In ANAO Report No. 7 2014-15 Administration of Contact Centres, the ANAO used Dimension Data’s 2013-14 Global Contact Centre Benchmarking Report, and spoke with representatives from centres operated by other government agencies and the private sector, to measure how the Australian Taxation Office’s (ATO) performance compared.   For example, to assess the appropriateness of the coverage of the ATO’s quality assurance (QA) of calls the ANAO compared the numbers of calls selected by the ATO for QA review for each assessor with two government and two large private sector contact centres, and with comparable information in the global benchmarking report.

Another example is ANAO Report No. 19 2013-14 Management of Complaints and Other Feedback. The audit objective was to assess the effectiveness of the ATO’s complaints and other feedback management systems in supporting service delivery. One audit criterion was whether the ATO effectively analysed complaints and other feedback to inform service delivery. As the ATO did not collect data on the cost of managing complaints, to assess the efficiency of complaints management the ANAO developed and applied its own cost assessment framework. Based on data provided by ATO business and service lines, the ANAO estimated that the average total unit staff cost per complaint was $611 in 2012–13, and cost $382 per complaint when staff support costs such as analysis, coordination, reporting and quality assurance were not included.

The Australian Public Service is currently in the process of implementing a new resource management framework. The Public Governance, Performance and Accountability Act 2013 (the PGPA Act) was introduced in 2013 with the aim of requiring Commonwealth entities to: meet high standards of governance, performance and public accountability; provide meaningful information to the Parliament and the public; use and manage public resources properly; and work cooperatively with others to achieve common objectives

The Department of Finance has emphasised that the reform program underpinned by the Act is based on the following principles:

• government should operate as a coherent whole
• a uniform set of duties should apply to all resources handled by Commonwealth entities
• performance of the public sector is more than financial
• engaging with risk is a necessary step in improving performance
• the resource management framework should support the legitimate requirements of the government and the parliament in discharging their respective responsibilities.

The PGPA Act established the basis for developing a resource and risk management framework to meet the needs of modern government, by improving the quality of planning, performance information, evaluation and reporting within government to enhance accountability.

The future of performance auditing

The ANAO will continue to adapt to changes in the public sector, including the evolving delivery of government services which is shaped by various drivers. One of these drivers is the performance management framework and how it will shape the focus of our business, including the development of the audit program. Success in delivering the ANAO mandate extends beyond evaluating implementation effectiveness in performance audits, to evaluating outcome effectiveness as well as strategy development, economy and efficiency. History tells us that the quality of performance information will be key to the ability to do this.

However, the ANAO should not be a passive recipient of what information is available. The ANAO is playing an important role in advising parliament, through the Joint Committee on Public Accounts and Audit, on the implementation of the more recent reforms to the performance management framework with the introduction of the PGPA Act. One aspect of this role includes undertaking performance audits on the implementation of the new framework. The focus here should not be critiquing the quality of central agency guidance, but rather assessing the commitment of entity leadership to implementing appropriate strategies to meet their obligations to the Parliament. These include transparency and accountability in entity operations. Another aspect is the use of the mandate to conduct audits of the appropriateness of the performance measures (however described). The ANAO should focus on performance information to encourage improvement in the quality of performance information under the new framework. Such audits should become a regular element of the ANAO work program.

The extension of the performance audit work program to more regularly examine the full spectrum of effectiveness, efficiency and economy, should not wait on the improvement of performance information.  The ANAO needs to be innovative in the application of its audit methodology and how it assesses entities’ performance, whilst still meeting the requirements of the performance audit standards.  For example, in the absence of quality published information and targets, entity performance can be assessed using other information by taking a practical, rather than a technical purist, approach. It is better to include some analysis about the costs and outputs of a program than to leave aside considerations of economy and efficiency altogether.

In the absence of good quality performance information and data an entity’s performance can be assessed using proxy indicators, trends, benchmarks and the existence or not of competitive procurement.  Questions that are reasonable for the performance auditor to ask include:

• Have inputs been procured through competitive processes and if not how do their costs compare with other entities (benchmarking) (economy);
• How does cost growth compare with relevant index benchmarks (economy);
• What has been the trend cost of output and does this provide any insights into efficiency and can changes in quality explain some of this (efficiency);
• During program design what delivery option analysis has been undertaken (or was the delivery framework decided at the same time the problem was identified) (cost effectiveness); and
• What indicators of outcome effectiveness do like entities use and can similar indicators be constructed (outcome effectiveness).

Reasonable assurance means a high, but not absolute, level of assurance. In forming an opinion, it is reasonable to point to whether the use of proxy data, trends and benchmarks raises questions as to whether a particular agency or activity is operating at an expected level of economy, efficiency or effectiveness.

The challenge will be — as it was when performance audit first commenced — how the audit office manages the reaction of agencies and the parliament to audit assertions and conclusions which have been formed on indirect measures. The key to success in this respect will be how well the audit scope and approach is explained, how entities are brought along the audit journey and, probably most importantly, how the capability and skills to undertake the work is developed.

This is an edited extract from a speech delivered by Grant Hehir in Brisbane on March 15, 2016.