Safety Design Principles for AI Adoption in Organizations

How do we deliver safe AI?

This is the question every organization grappling with AI adoption must answer. Yet too often, discussions about AI safety focus narrowly specific aspects of the technology (such as LLMs), as if AI exists in a vacuum.

The reality is more nuanced. AI is a technology that operates within existing systems—systems that in highly-regulated, high risk industries already have well-established definitions of safety, existing regulations, industry standards, and proven best practices. Understanding this context is essential to approaching AI safety effectively.

From an engineering perspective, I propose three principles that organizations should consider when designing or adopting AI solutions for their business.

Principle 1: Protect Existing Safety Systems

Do no harm to what already works.

The first principle is to ensure that AI technology does not diminish the effectiveness of existing safety measures and controls. Just as we strive not to harm the environment, we must strive to not compromise our ability to deliver existing levels of safety.

This requires understanding the impact that AI technology has on established safety systems. When you introduce AI into a manufacturing process, a healthcare workflow, or a financial control system, you must ask: Does this maintain or enhance the safety controls we already have in place? Does it create new failure modes in existing safeguards?

Consider two examples from a process safety perspective where Management of Change (MOC) applies to AI deployment:

Example 1: AI Technology Evaluated by MOC Process safety regulations require an MOC to be conducted before modifying any safety-critical operation. When an organization introduces AI to monitor equipment conditions or predict failures in a chemical facility, this constitutes a design change that triggers an MOC. The MOC risk assessment must evaluate how the AI system affects existing safety controls. For instance, if operators begin relying on AI alerts instead of conducting scheduled inspections, the organization has effectively changed its detection and mitigation strategy. The MOC process should identify this shift and determine whether compensating controls are needed to maintain safety integrity.

Example 2: AI Used Within the MOC Process Itself More subtly, when an organization uses AI to automate parts of the MOC workflow—such as reviewing and approving change requests—this change to the MOC process itself requires an MOC. The original MOC system had segregation of duties: one person requests the change, another reviews technical details, a supervisor approves it. This prevented single points of failure in safety-critical decision-making. Replacing human reviewers with an AI system is simultaneously a design change (new technology), a procedural change (new workflow), and an organizational change (altered roles and responsibilities). Without conducting an MOC on this change to the MOC process, the organization unknowingly degrades a fundamental safety control while believing they've simply improved efficiency.

The engineering discipline here is straightforward but often overlooked:

• Map your existing safety controls.

• Understand how AI integration affects each one.

• Verify that safety is maintained or improved, not degraded.

Principle 2: Protect Operational Systems

Isolate the hazard, then control the risks.

AI technology has inherent uncertainties that exceed those of traditional technologies. These uncertainties create opportunities for emerging and novel risks. In the language of systems safety expert Nancy Leveson, AI creates the potential for "hazardous processes" within organizations.

The response to any hazard follows a consistent pattern: first isolate the hazard, then handle the risks it introduces. This means establishing guardrails—safeguards that protect the organization, its workers, customers, and stakeholders from the consequences of using hazardous technology which in this case is AI.

What makes AI a potential hazard is the nature of its uncertainties:

• Probabilistic outputs rather than deterministic ones

• Opaque decision-making in complex models

• Emergent behaviours that weren't explicitly programmed

• Dataset dependencies that may not generalize

• Adversarial vulnerabilities unique to machine learning systems

The STAMP (Systems-Theoretic Accident Model and Processes) and STPA (System-Theoretic Process Analysis) methodologies provide systematic approaches to dealing with these kinds of uncertainties. Originally developed for aerospace applications, these frameworks have proven valuable in cybersecurity and are now being applied to AI systems.

Using STPA, organizations can identify unsafe control actions, understand loss scenarios, and design constraints that prevent hazardous system states. This is not about preventing all possible failures—that's impossible with any complex system—but about understanding failure modes and designing appropriate controls.

Principle 3: Protect Organizational Systems

Make AI technology itself less risky.

The third principle focuses on reducing risk at the source: designing AI technology that is safer regardless of how it's deployed or used.

The 2016 paper "Concrete Problems in AI Safety" by Amodei et al. helped crystallize many of these challenges. It identified specific technical problems such as avoiding negative side effects, reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Since its publication, we've seen the creation of dedicated AI Safety institutions and the emergence of AI Safety Engineering as a discipline.

We now understand that different categories of AI systems present different risk profiles:

• Narrow AI systems trained for specific tasks have bounded but still significant risks

• Agentic systems that take actions autonomously introduce new categories of risk

• Advanced AI systems (AGI or ASI) would present risks of an entirely different magnitude

For many organizations, the responsibility for AI safety is a shared risk with companies creating large language models and foundation models. However, organizations are developing their own specialized models for specific use cases. These custom models require their own risk assessment and safety measures tailored to their particular context and application.

This means organizations cannot simply rely on foundation model providers to solve all safety problems. If you're fine-tuning models, creating retrieval-augmented generation systems, or deploying AI agents, you have safety engineering work to do.

An Integrated Approach

From an engineering perspective, all three principles must be considered together, not just the last one.

An AI model might be state-of-the-art in terms of its training methodology and robustness testing, but if it's deployed in a way that undermines existing safety controls or without adequate guardrails for its uncertainties, the overall system becomes less safe, not more.

This integrated view reflects how safety is actually achieved in mature engineering disciplines.

Aircraft aren't safe just because engines are reliable. They're safe because of redundant systems, rigorous maintenance protocols, pilot training, air traffic control, and countless other layers of protection. Similarly, safe AI systems require attention to technology, process, and context.

Practical Next Steps

For organizations looking to implement these principles:

For Principle 1 - Protect Safety Systems

• Document existing safety controls before AI deployment

• Conduct impact assessments on how AI affects these controls

• Establish verification processes to ensure safety is maintained

For Principle 2 - Protect Operational Systems

• Adopt systematic hazard analysis methodologies like STPA

• Create clear governance structures for AI risk management

• Implement monitoring systems to detect emerging risks

• Design guardrails appropriate to the level of uncertainty and consequence

For Principle 3 - Protect Organizational Systems

• Engage with AI safety research and best practices relevant to your sector

• Conduct thorough testing including adversarial and edge case scenarios

• Participate in industry collaborations on safety standards

• Budget for ongoing safety evaluation as systems and understanding evolve

Conclusion

The question of how to deliver safe AI is indeed the question of the day.

But it's not a question that can be answered by focusing on AI technology alone. Safety emerges from the interaction between technology, systems, processes, and people.

By preserving existing safety systems, managing AI-specific uncertainties, and designing inherently safer AI, organizations can move beyond the hype and fear that too often characterize AI discussions. They can adopt AI in ways that are both innovative and responsible.

This is the work of engineering: not making perfect systems, but making systems that fail safely, that operate within understood bounds, and that deliver value without unacceptable risk. It's work worth doing, and doing well.

---

Raimund Laqua, P.Eng is Founder of Lean Compliance, and Co-Founder of Professional Engineers.AI.