Audits have been used for many years to confirm the integrity of financial statements and that proper accounting procedures have been used.
In recent decades auditing has also become table stakes not only for finance, but also quality, safety, environmental and regulatory management systems. As with accountancy, the auditing function does not evaluate the effectiveness of your financial system, nor does it do so for quality, safety, or environmental systems. Auditing only confirms that you are following acceptable practices (usually defined by a standard) and the outputs of the system have not been interfered or tampered with.
The evaluation and auditing of system effectiveness is not part of the auditing or the compliance function, so which function is it a part of and what should it be auditing?
Auditing as Quality Control / Assurance
Auditing has become the core function across almost all compliance domains similar to what quality control and assurance functions have done for quality management.
The American Society for Quality (ASQ) defines quality control as follows:
Quality Control – can be defined as "part of quality management focused on fulfilling quality requirements."
Quality Assurance – can be defined as "part of quality management focused on providing confidence that quality requirements will be fulfilled."
Quality control is more the inspection aspect of quality management. It answers the question of are we following the right steps the right way? Whereas, quality assurance on the other hand relates more to quality outcomes.
The confidence provided by quality assurance is twofold — internally to management and externally to: customers, government agencies, regulators, certifiers, and third parties.
Assurance is demonstrated when "all the planned and systematic activities implemented within the quality system can be demonstrated to provide confidence that a product or service will fulfill requirements for quality."
However, even assurance that the right steps have been done the right way is not enough to ensure compliance outcomes. Something else is needed.
A Shift that Shouldn't be Ignored
Before we look at the answer to these questions, we first need to recognize a shift that is happening with respect to regulatory designs.
Increasingly, regulatory and standards bodies are transforming their operations, taking on a more risk-based approach focused on outcomes and continuous improvement. This has resulted in the introduction of regulations and standards that are moving away from prescriptive to performance and risk-based requirements.
Organizations are expected to establish their own means (the how) by which they will achieve targeted goals and objectives. This affords greater latitude for organizations to better address complex and systemic problems. It also holds them accountable for the outcomes of their systems, where in the past they have only been responsible for the outputs of prescriptive requirements.
This shift has in many cases come with much confusion. It is not uncommon to find performance-based frameworks including prescriptive "shall" statements related to "how" it should be done. After years under the tutelage of prescriptive regulation the pull towards having something to audit is very strong which while understandable creates confusion for those adopting new and updated regulations and standards.
What Then Should Be Audited?
Increasingly, obligations that arise from regulation along with industry standards are requiring that organizations make progress towards what is often called, Vision Zero targets. These include zero harm, zero fatalities, zero incidents, zero emissions, zero violations and so on. Advancing these goals requires risk-based approaches and the continuous improvement of capabilities to generate appropriate levels of performance for progress to be made.
When we now think about compliance we should be considering the goals that are being targeted. An important distinction that can made is between "terminal" and "instrumental" goals.
Terminal goals are the highest level objective that we want to reach. They define the "ends" of our compliance programs, for example: zero defects, zero fatalities, zero violations, zero releases, zero fines, and others.
Instrumental goals are intermediate outcomes or results that are critical or that must occur in order to achieve the higher-level outcome. These are often used to define Measures of Effectiveness (MoE) for compliance programs as they provide clear indication of progress towards terminal goals. Measures of Effectiveness can be used to validate compliance programs to ensure that they are fit for the purpose of advancing outcomes.
The following are Measures of Success for compliance frameworks that support performance and outcome-based obligations (see previous figure):
Measures of Effectiveness (MoE) – critical to program success, independent of any technical implementation (i.e. the how). Focuses on the ends not the means.
Measures of Performance (MoP) – measures that relate to the operations of the compliance program, systems, and processes. These are the measures of capabilities needed to be effective.
Measures of Conformance (MoC) – critical to compliance, where failure maybe cause for reassessment of the program. These tend to be prescriptive legal requirements but may include voluntary practices.
Auditing has traditionally been helpful to verify Measures of Conformance but now needs to support Measures of Performance and Effectiveness. The latter is the task of governance and program management. Together they identify the destination and then steer the organization towards it. To be effective they need compelling answers to these questions:
Where should we be heading?
How will we get there? What is our strategy?
What capabilities and resources do we need to get there?
What obstacles are in the way?
How will we measure our progress?
The audit function now forms a validation function connected with progress towards targeted outcomes rather than only conformance to shall statements. Those in the pharma and medical device industry will recognize this distinction between verification and validation. You can build a pacemaker that meets all design specifications (which you can verify) and yet fails to keep your heart pumping. This is precisely the shift that is happening with safety, environmental and regulatory objectives. You can build a system that conforms to all the standards and yet fails to make any progress on outcomes. This is why compliance now should audit outcomes over outputs.
Additional Reading:
