The AI Prediction Trap
AI agents are being built to act on the predictions of AI models. That should give us pause.
The belief behind this is simple. The model predicts the future, and the agent acts on that prediction as if it were reality.
When an agent acts on a prediction, it helps make that prediction real. Its actions shape what happens next. The future arrives looking the way the model said it would. The model appears to have been right, and we trust it a little more the next time. Over time, the prediction and the plan become the same thing.
Predictive policing shows how this works. A model predicts crime in a neighbourhood. More patrols are sent there. More incidents are recorded. The model looks accurate, and the next prediction points to the same place.
This is the trap of the self-fulfilling prophecy.
Models are useful. They may see patterns we miss. Yet every model is wrong in some way, because a model is a simplification of reality. Even when it reads the trend correctly, the future is still open. It is shaped by people, through human agency, through will, and through our capacity to govern ourselves.
This matters most when a model predicts an outcome we would not or should not choose. A forecast of harm is a warning. If we let an agent act on it, the agent may help bring that harm about. When a prediction points toward a bad outcome, the better response is to change the prediction.
The same trap applies to us when it comes to AI itself. Some say AI may end the world. I understand the concern. Yet if we accept that prediction as destiny, we stop working to prevent it, and it becomes more likely. A prediction like this is a call to act. We change it through how we govern AI.
So when I look at AI agents that act on predictions, I find myself asking three questions.
Is the future this agent predicts one we want?
Are its actions helping to make that prediction come true?
If the prediction is unfavourable, what are we doing to change it?
This is where compliance and assurance come in. Some suggest the key question for AI governance is whether an agent's decisions are admissible, meaning within acceptable limits. That is conformance. It matters, but it is not enough. An agent can make only admissible decisions and still carry us toward a future we never intended. Assurance asks a different question: are we on track to keep our promises and achieve the outcomes we committed to? An agent acting on a prediction pursues the model's future. Governance steers toward ours.
Decision-making is not deterministic, and for good reason. It is where human agency and judgment are exercised. It is where a future is created, not just reacted to. An agent that acts on a prediction as if it were reality makes the decision deterministic. We should keep that decision with people, including the decision about how the AI story ends.
