Is AI Adoption Normalizing Risk?

Some things are best described as the absence, or privation, of something good. Normalization of deviance, drift to failure, unsafe conditions, and unsafe acts all take their meaning from a positive prior. We first define what it means to be safe. Integrity is a measure of how well that state is maintained, and any loss of integrity is what we call “unsafe.”

The Challenger accident shows how this works. The solid rocket booster joints were designed so that the O-rings would never erode. On earlier flights, engineers found that they had. Because those flights returned safely, the erosion was reviewed and formally accepted as a tolerable risk, and the standard shifted from “no erosion” to “some erosion is acceptable.” Each successful launch reinforced the new standard. By the time of the cold-weather launch, the original definition of safe had been replaced, one reasonable decision at a time, through the organization’s own review process.

A system risk approach such as STAMP helps us understand how hazards behave and how risk propagates through a system. It treats safety as a control problem, and every control system needs a reference to steer toward. For safety, that reference is a definition of what it means to be safe. In engineered systems, safety is the intended state we define and work to maintain. It is an outcome delivered by the parts of the system working together as designed. What emerges, often from interactions we did not foresee, is the departure from that state.

The same premise applies to AI systems. Whether or not they are built with general-purpose AI, they are engineered systems, and much of the current conversation about AI safety skips the first step. We talk about guardrails, safeguards, and red-teaming. Each of these assumes we know what failure looks like. Failure can only be defined against an intended state, and few organizations have defined that state for their AI systems: what each system promises to do, for whom, within what limits, and under what conditions. Without that definition there is no reference against which to measure drift, no basis on which to design controls, and no basis to identify failure.

AI systems also produce many unforeseen interactions. Models are probabilistic. They are composed with other models and tools, and they are deployed into environments that keep changing. Unsafe outcomes will emerge from these interactions. We can recognize them as unsafe only if we have first defined what it means to be safe.

Normalization of deviance is already visible. Hallucinations are accepted as a known limitation of the technology, and that acceptance is itself normalization of deviance. Outputs that miss the mark are corrected downstream by people who absorb the cost. Each tolerated departure becomes the new baseline. This is the Challenger pattern, applied to intelligence systems.

AI safety begins with defining what it means to be safe for each system. Loss of integrity is a measure of safety: how far we have drifted from that definition.

The lesson of Challenger is that each reasonable decision can increase the drift from what it means to be safe.

As AI adoption accelerates, pressure will build to accept whatever these systems do as the cost of progress. Without defining what it means to be safe, we will not see the departures we are accepting, and the loss of integrity will grow unnoticed.

We do not have to let that happen. Define what it means to be safe for your AI systems, measure the loss of integrity against it, and treat every departure as a warning to investigate.