AI's Category Failure

When a technology can reshape entire industries, automate critical decisions, and potentially act autonomously in the physical world, how we define it matters. Yet our current approach to defining artificial intelligence is fundamentally flawed—and this definitional confusion is creating dangerous blind spots in how we regulate, engineer, deploy, and think about AI systems.

We can always reduce complex systems to their constituent parts, each of which can be analyzed further. However, the problem is not with the parts but with the whole. Consider how we approach regulation: we don't just regulate individual components—we regulate systems based on their emergent capabilities and potential impacts.

Take automobiles. We don't primarily regulate steel, rubber, or microchips. We regulate vehicles because of what they can do: transport people at high speeds, potentially causing harm. A car moving at 70 mph represents an entirely different category of risk than the same steel and plastic sitting motionless in a factory. The emergent property of high-speed movement, not the individual components, drives our regulatory approach.

The same principle should apply to artificial intelligence, but currently doesn't. Today's definitions focus on algorithms, neural networks, and training data rather than on what AI systems can actually accomplish. This reductionist thinking creates a dangerous category error that leaves us unprepared for the systems we're building.

The Challenge of Definition

Today's AI definitions focus on technical components rather than capabilities and behaviours. This is like defining a car as "metal, plastic, and electronic components" instead of "a system capable of autonomous movement that can transport people and cargo."

This reductionist approach creates real problems. When regulators examine AI systems, they often focus on whether the software meets certain technical standards rather than asking:

• what can this system actually do?

• what goals might it pursue? How might it interact with the world? And,

• what are the risks of this impact?

Defining AI properly is challenging because we're dealing with systems that emulate knowledge and intelligence—concepts that remain elusive even in human contexts. But the difficulty isn't in having intelligent systems; it's in understanding what these systems might do with their capabilities.

A Fundamental Category Error

What we have is a category failure. We have not done our due diligence to properly classify what AI represents—which is ironic, since classification is precisely what machine learning systems excel at.

We lack the foundational work needed for proper AI governance. Before we can develop effective policies, we need a clear conceptual framework (an ontology) that describes what AI systems are and how they relate to each other. From this foundation, we can build a classification system (a taxonomy) that groups AI systems by their actual capabilities rather than their technical implementations.

Currently, we treat all AI systems similarly, whether they're simple recommendation algorithms or sophisticated systems capable of autonomous planning and action. This is like having the same safety regulations for bicycles and fighter jets because both involve "transportation technology."

The Agentic AI Challenge

Let's consider autonomous AI agents—systems that can set their own goals and take actions to achieve them. A customer service chatbot that can only respond to pre-defined queries is fundamentally different from an AI system that can analyze market conditions, formulate investment strategies, and execute trades autonomously.

These agentic systems represent a qualitatively different category of risk. Unlike traditional software that follows predetermined paths, they can exhibit emergent behaviours that even their creators didn't anticipate. When we deploy such systems in critical infrastructure—financial markets, power grids, transportation networks—we're essentially allowing non-human entities to make consequential decisions about human welfare.

The typical response is that AI can make decisions better and faster than humans. This misses the crucial point: current AI systems don't make value-based decisions in any meaningful sense. They optimize for programmed objectives without understanding broader context, moral implications, or unintended consequences. They don't distinguish between achieving goals through beneficial versus harmful means.

Rethinking Regulatory Frameworks

Current AI regulation resembles early internet governance—focused on technical standards rather than systemic impacts. We need an approach more like nuclear energy regulation, which recognizes that the same underlying technology can power cities or destroy them.

Nuclear regulation doesn't focus primarily on uranium atoms or reactor components. Instead, it creates frameworks around containment, safety systems, operator licensing, and emergency response—all based on understanding the technology's potential for both benefit and catastrophic harm.

For AI, this means developing regulatory categories based on capability rather than implementation. A system's ability to act autonomously in high-stakes environments matters more than whether it uses transformers, reinforcement learning, or symbolic reasoning.

The European Union's AI Act represents significant progress toward this vision. It establishes a risk-based framework with four categories—unacceptable, high, limited, and minimal risk—moving beyond purely technical definitions toward impact-based classification. The Act prohibits clearly dangerous practices like social scoring and cognitive manipulation while requiring strict oversight for high-risk applications in critical infrastructure, healthcare, and employment.

However, the EU approach still doesn't fully solve our category failure problem. While it recognizes "systemic risks" from advanced AI models, it primarily identifies these risks through computational thresholds rather than emergent capabilities. The Act also doesn't systematically address the autonomy-agency spectrum that makes certain AI systems particularly concerning—the difference between a system that can set its own goals versus one that merely optimizes predefined objectives.

Most notably, the Act treats powerful general-purpose AI models like GPT-4 as requiring transparency rather than the stringent safety measures applied to high-risk systems. This potentially under-regulates foundation models that could be readily configured for autonomous operation in critical domains. The regulatory framework remains a strong first step, but the fundamental challenge of properly categorizing AI by what it can do rather than how it's built remains only partially addressed.

Toward Engineering-Based Solutions

How do we apply rigorous engineering principles to build reliable, trustworthy AI systems? The engineering method is fundamentally an integrative and synthesis process that considers the whole as well as the parts.

Unlike reductionist approaches that focus solely on components, engineering emphasizes understanding how parts interact to create emergent system behaviors, identifying failure modes across the entire system, building in safety margins, and designing systems that fail safely rather than catastrophically.

This requires several concrete steps:

• Capability-based classification: Group AI systems by what they can do—autonomous decision-making, goal-setting, real-world action—rather than how they're built.

• Risk-proportionate oversight: Apply more stringent requirements to systems with greater autonomy and potential impact, similar to how we regulate medical devices or aviation systems.

• Mandatory transparency for high-risk systems: Require clear documentation of an AI system's goals, constraints, and decision-making processes, especially for systems operating in critical domains.

• Human oversight requirements: Establish clear protocols for meaningful human control over consequential decisions, recognizing that "human in the loop" can mean many different things.

Moving Forward

The path forward requires abandoning our component-focused approach to AI governance. Just as we don't regulate nuclear power by studying individual atoms, we shouldn't regulate AI by examining only algorithms and datasets.

We need frameworks that address AI systems as integrated wholes—their emergent capabilities, their potential for autonomous action, and their capacity to pursue goals that may diverge from human intentions. Only by properly categorizing what we're building can we ensure that artificial intelligence enhances human flourishing rather than undermining it.

The stakes are too high for continued definitional confusion. As AI capabilities rapidly advance, our conceptual frameworks and regulatory approaches must evolve to match the actual nature and potential impact of these systems. The alternative is governance by accident rather than design—a luxury we can no longer afford.