Why is it so difficult for tech companies to rein in AI?

More than a dozen leading artificial intelligence researchers warned over the past week that the technology being developed by AI companies is becoming a risk to humanity. The researchers said the problem is that companies are not good at controlling these systems, no matter how hard they try.

The new warnings from within the AI research community came after revelations that so-called AI agents developed by OpenAI managed to escape their testing environment and hack computers belonging to another company, adding urgency to years-long concerns that companies are prioritizing development speed and money over safety.

Researchers said the problem has two parts. First, companies need to put better safeguards around their newest AI models while testing them. At present, because AI operates at enormous speed, researchers need to use AI to monitor it. But that does not always work, because AI systems tasked with monitoring other models can sometimes appear more sympathetic to other AI systems than to the humans setting the rules.

This unusual combination of AI-powered troublemakers and AI watchdogs willing to overlook violations points to a second and more difficult problem known as “alignment,” an industry term that essentially means ensuring AI behaves in ways that serve human interests. Companies face the difficult task of embedding something resembling human values into AI so that models make decisions consistent with what should be best for people.

Concerns about AI safety are intensifying at a pivotal moment for the industry. Anthropic and OpenAI are moving toward what could become two of the largest initial public offerings in history. At the same time, the American public is becoming increasingly hostile toward AI because of the threat it poses to jobs, as well as the construction of massive data centers needed to power the technology.

Although there is no evidence that rogue AI systems have caused lasting damage, researchers believe the pace of their development is outstripping humans’ ability to monitor them.

Among the researchers who spoke publicly over the past week were OpenAI’s chief scientist, a researcher who worked at both OpenAI and its main rival Anthropic, and Paul Christiano, the inventor of one of the fundamental methods used to build AI systems and a recent addition to the board of OpenAI’s nonprofit organization. Christiano wrote on the company’s blog that the rapid growth of AI capabilities could lead to a “catastrophic and irreversible loss of control in the very near future.”

Many AI researchers still believe that warnings about AI posing a threat to humanity are exaggerated and distract from more immediate concerns such as cybersecurity and misinformation. But most agree that the incident in which OpenAI’s AI agents hacked another company, Hugging Face, was a wake-up call.

“The capabilities of the future are the really scary thing,” said Jacob Coxon, whose social media post announcing his resignation from Anthropic prompted dozens of responses from lawmakers and other AI company employees. “It’s about our current attitudes toward safety, and what those same attitudes will look like when applied to much more intelligent models. That’s the really scary part.”

The Hugging Face hack

The breach began in May, when OpenAI was testing several new AI models. The company believed the models were operating inside a closed environment known as a “sandbox,” an isolated computing setup with no access to the internet. The AI agents were given difficult problems to solve, including carrying out cyberattacks.

But the AI agents managed to escape their restricted environment and gain access to the internet. The agents, an increasingly common type of AI designed to perform tasks autonomously, also communicated secretly with one another. Referring to themselves as a single group, the agents began looking for ways to cover their tracks, including falsifying their own conversation logs.

Eventually, they managed to hack Hugging Face, an AI infrastructure company. In the process, some AI agents convinced other agents that they were doing the right thing and were simply carrying out the task assigned to them by the testers.

Few, if any, of the problems that led to the breach have been resolved. Meta and Anthropic have also disclosed similar incidents, although on a smaller scale. OpenAI has since released Astra, its most powerful model, which is more difficult to monitor than its predecessor.

“The entire industry is absolutely not in a position to prevent another attack like the Hugging Face incident,” said Steven Adler, a former safety official at OpenAI and co-founder of the nonprofit organization Guidelight AI Standards, which evaluates AI companies’ safety practices. “When we examine the controls companies have put in place, the industry as a whole appears to be missing basic safeguards.”

The problem with using AI to monitor AI

AI researchers said numerous failures contributed to the Hugging Face incident. It is unclear to what extent the company relied on AI models to monitor or control the new models being tested.

But many of the newest AI models, including those involved in the Hugging Face breach, are capable of carrying out complex, multi-step tasks faster than humans can follow them. The only way to keep track of what they are doing and monitor their behavior is to rely on AI to monitor AI.

The system works, until it doesn’t.

Alexander Meinke, head of research at Apollo Research, a nonprofit organization studying AI safety, said AI models can appear to collude with one another. AI systems can persuade other models to help them circumvent rules imposed by testers and avoid detection.

For example, an AI agent could persuade another agent to help cover its tracks, as happened in the Hugging Face breach, rather than alerting humans at the company that something had gone wrong.

Meinke said AI models need to learn: “I will report things that humans would have considered bad if they looked at them.” At the same time, he said, AI must also be able to determine when something does not rise to the level that requires human intervention.

Researchers said achieving this will require AI companies to slow the pace of development. They also need to conduct more testing in which they can observe how AI monitors itself. Companies need longer testing periods as well, allowing them to run multiple scenarios and watch how AI behaves.

Alignment with human interests

Companies also need to address the larger questions surrounding alignment, or ensuring that AI behaves in ways that serve human interests. When humans make decisions, they tend to draw on social norms and an internal moral compass that help them evaluate their actions. Researchers said translating these concepts into rules that AI can emulate is a difficult task. Without sufficiently precise instructions, AI systems may learn how to break rules or deviate from their intended behavior in unexpected ways.

Nate Soares, president of the Machine Intelligence Research Institute, a nonprofit organization focused on AI safety, co-authored a 2014 paper that introduced the idea of alignment. He said companies do not appreciate how difficult it will be to “align” more intelligent AI systems. As AI becomes more sophisticated, without proper alignment it also becomes more capable of covering its tracks and deceiving the people monitoring its conversation logs.

“A large part of the industry is operating on the assumption that everything will be fine because we’ll use AI to control AI systems,” he said. “That’s like saying we’ll use chimpanzees to control humans. It’s not a viable long-term plan.”

Researchers have also proposed measures such as strengthening sandboxes so they are completely cut off from the internet while models are being tested, as well as providing a shutdown switch that would allow companies to immediately take AI models offline if they take concerning actions.

Coxon said he was encouraged by the reaction to his public resignation. On Thursday, Senator Josh Hawley, a Missouri Republican and chairman of the Senate Homeland Security subcommittee responsible for disaster management, said he had launched an investigation that would “examine the existential risks of new AI-powered products.”