There Is No Such Thing as a Safe AI Model
Compliance Outcomes Emerge from the System Not from the Parts Alone
Regulations such as the EU AI Act and management frameworks such as ISO/IEC 42001 and the NIST AI RMF have settled on the AI system as the thing to govern. Yet most of the conversation is about parts: models, platforms, agents, and the vendors who supply them.
This matters because safety, security, quality, performance, and reliability are compliance outcomes, and they are properties of a system. Compliance here means meeting all of an organization's obligations: regulatory requirements, duty of care, and the voluntary commitments it makes to the public, its customers, and other stakeholders.
These outcomes emerge by design and intention when the parts work together as engineered, in a specific context, for a specific purpose, under specific controls. Each component brings characteristics to that design. The system is where those characteristics become outcomes.
A system will never deliver these outcomes on its own or by chance. Unless they are designed in as goals the system is regulated to achieve, the laws of cybernetics tell us it will optimize away from them, toward speed, cost, and whatever else it is pressured to deliver.
The EU AI Act makes this explicit. It requires high-risk AI systems to be designed and developed to achieve an appropriate level of accuracy, robustness, and cybersecurity, and to perform consistently in those respects throughout their lifecycle. These are requirements of the system, and they have to be designed in and sustained.
So a certified model or a secure platform tells us very little about whether an intelligence system will be safe in use. Those claims describe characteristics of the part on its own. Compliance outcomes depend on how the parts are integrated, operated, and kept under control over time.
This is why compliance has to be practised at the system level. Each component can be assessed against a standard, and that evidence has value. Yet compliance outcomes come about only when the whole system is designed, integrated, and operated to deliver them. Only the organization that puts the system into service can see the whole, and only that organization can make the case that those outcomes are being achieved.
Organizations must take responsibility for their intelligence systems. Vendors cannot do this for them, and neither can the technology. That responsibility calls for compliance engineering: the practice of designing compliance outcomes into the system, regulating the system to achieve them, and sustaining them for as long as the system is in service.
Before you put your next AI system into service, ask one question: who is engineering its compliance outcomes? If the answer points to a vendor, you have work to do.
Let's talk about how to build compliance engineering capability in your organization: https://calendly.com/lean-compliance/30min
