Governance States Need Positive Priors
Much of the current discussion on AI governance focuses on evaluating decisions at the point of execution. An agent proposes an action, and the action is checked to see whether it is admissible. In earlier posts I raised two problems with this approach. It does not account for acts of omission, and actions that are acceptable on their own can still be unsafe in aggregate. There is a third problem, and it concerns what the decision is being evaluated against.
Positive Priors
To evaluate anything, we need something to evaluate it against. A prior is what must be known or established before an evaluation can take place. It is the reference point that gives the evaluation meaning. When we ask whether a decision is a good one, we are really asking whether it moves the system toward or away from something we have already defined.
A positive prior is a defined state that we intend a system to achieve and maintain. It describes what good looks like.
Safety is a good example. Safety is a positive state of a system. It describes the condition we intend the system to remain in, and any deviation or drift from that condition is what we call unsafe. This means safety must be defined before it can be evaluated. A decision cannot be judged as leading to unsafe actions unless we first know the state the system is meant to keep. That state is the positive prior.
The same is true of other outcomes we care about, such as quality, security, and integrity. Each is a positive state, and each needs to be defined before deviation from it can be recognized. Without these priors, evaluating a decision at the point of execution has nothing to measure against.
Two Causes of Risk
Risk theory adds a second principle. A risk requires two causes: an unsafe condition and an unsafe action. For an explosion to occur, an ignition source (the unsafe action) must be brought into the presence of a flammable gas (the unsafe condition). Either one alone does not produce the explosion.
This tells us that an action cannot be evaluated by examining the action alone. The same action can be harmless in one setting and catastrophic in another, depending on the conditions present. Knowing which hazards exist, that is, the sources of uncertainty an action will encounter, is therefore another essential prior. It provides the context against which actions must be evaluated.
Evaluating Against a Risk Profile
If this sounds like risk management practice, you are right.
In the domains I work in, such as oil and gas, pharmaceuticals, and chemical processing, consequential decisions are evaluated against a risk profile. The risk profile captures the hazards present and the positive states the operation must keep. A decision is approved only when the risks are reduced to what is tolerable, or when a sufficient margin is established to cushion the effects should something go wrong.
This work is done in advance. The priors are established before decisions are made, so that each decision can be evaluated against them when the time comes.
Commitments as the Backbone
For AI agents, Promise Theory offers a way to carry these priors forward. An agent can be designed to keep commitments made in advance, and those commitments form the basis for evaluating whether a decision aligns with what was promised. In this sense, commitments become the backbone of agent governance.
For that backbone to be sound, the commitments must express the priors. They need to include the positive states the agent is expected to keep and the hazards it will contend with. When they do, a decision can be evaluated against something meaningful: the promises the agent was designed to keep, in the conditions it is operating in.
Where Governance Begins
A governance state provides the basis on which decisions are evaluated before they proceed. It can only serve that purpose if it records what each decision must be evaluated against: the positive states the agent is expected to keep, and the hazards present in the conditions it operates in. Without these priors, a governance state describes what has been committed and what has happened, but it offers no basis for judging whether the actions will be safe.
This is why defining the priors comes first. Before an agent is deployed, we need to establish what safe operation looks like and what hazards it will encounter, and then express these as commitments the agent is designed to keep. Every decision the agent makes can then be evaluated against those commitments.
Raimund (Ray) Laqua, P.Eng., PMP, is the Founder and Principal Consultant of Lean Compliance Consulting, Inc., and the person who defined the practice of Lean Compliance. He advises organizations in highly regulated, high-risk industries on how to meet their obligations and contend with risk. Ray is one of the first licensed software and computer engineers in Canada and has practised digital engineering for more than 40 years. He chairs the Digital Engineering Committee for Engineers for the Profession (E4P), serves on OSPE’s AI in Engineering Task Force, and advocates nationally for Professional Digital Engineering (PD.Eng.) licensure.
