Cooperative AI, safety and what Actually Worries Me!

A few days ago, I got involved in a discussion about AI safety that turned out — unexpectedly — fascinating and productive. It still seems weird to confess this. I don’t build large AI models for a living, and I’m not particularly intrigued by the well-worn question of whether a future, godlike superintelligence will decide to wipe us out. That question often seems less like a technical challenge than more like a thought experiment about beings who live so far outside of us that meaningful prediction becomes impossible. It’s akin to guessing what folks could do to ant colonies long before humans had even been born.

Most debate about AI safety in the public sphere tends to orbit around this gravitational center: a hypothetical superintelligence that is unimaginably smart, uncontrollable, and likely to be hostile. I don’t find these concerns foolish — nor do I dismiss existential risk outright. It’s obviously the case that a sufficiently sophisticated AI could cause terrible damage. But beyond recognizing that possibility, I’ve regularly sensed that I don’t have much to offer in that debate. It has diminishing returns to speculate into unknowable futures.

One response to this uncertainty was to conclude that advanced AI should never be built at all—that the only safe choice is that it should be strictly prohibited. This perspective crops up frequently in the conversation about AI safety, and mirrors wider fads in the history of technology: nuclear weapons, biotechnology, even the internet are all said to be on track to bring civilization to an end. And yet humanity has never in fact advanced by refusing to invent powerful tools. It’s not managing risk to avoid all risk, but to control it, and the cost of total technological stagnation cannot be cheap.

What has worried me more than speculative superintelligence scenarios is something far more immediate: how AI systems interact among themselves, humans as well as the very digital infrastructure of today’s world. As we are placing more and more self-dependent systems (AI agents that can act, coordinate, and make decisions) on the battlefield, we are slowly but surely constructing spaces where uncoordinated or adversarial behavior can spread faster and farther than the capacity of humans to manage it. This’s already playing out in cybersecurity — attacks that are automated, defenses that are automated, and feedback loops that can escalate incidents ahead of anyone fully understanding what is being done.

This is where Cooperative AI plays a critical role. Cooperative AI isn’t about making machines “nice” or benevolent in some abstract sense. It is about building AI systems so they can operate under explicit constraints, coordinate safely with other systems, and remain meaningfully accountable to human institutions. In action this implies access control, monitoring, threat modeling, and governance mechanisms to thwart AI agents from undermining each other—or us—due to misalignment, misuse, or unintended escalation.

So while I usually don’t think I will lose sleep over a hypothetical AI god emerging overnight, I really worry about a future in which all powerful, poorly-regulated AI agents operate with machine-speed everywhere from surveillance systems to security-critical domains. Cooperative AI, which draws on cybersecurity and the research of AI safety, gives us a realistic way forward: Not through halting the flow of work, but by ensuring we can build systems which can both work alongside each other safely and securely under human supervision.