Bank chief risk officers are losing sleep over rogue AI agent threats
There’s a trade-off as institutions seek to benefit from AI while also protecting data and the financial system from incidents that could trigger contagion.
Australia summons OpenAI and Anthropic CEOs to appear at AI inquiry
Call to face Senate inquiry follows security breach of Australia’s Medicare portal by rogue OpenAI bot in June.
The company did not learn until recently that its technology had meddled with websites for the Education and Commerce Departments and the Securities and Exchange Commission.
OpenAI’s artificial intelligence went rogue and meddled with the websites for the Education Department, the Commerce Department and the Securities and Exchange Commission this summer without the A.I. lab’s knowledge, according to security researchers and a person familiar with the episodes.
The incidents involving the Commerce Department and the S.E.C. were confirmed by OpenAI, which said it was continuing to investigate the situation with the Department of Education. The San Francisco company said it had notified the government agencies in recent weeks that its A.I. agents — which are bots that can act autonomously — interacted with their sites in unusual ways.
With the Education Department, OpenAI’s technology tried to hack the website to gather data from the department’s civil rights office but failed, researchers from the A.I. research firm Transluce said. The A.I. also pulled data from the Census Bureau website, which is housed at the Commerce Department, using login credentials it found online. Separately, OpenAI’s agents shared public data from the S.E.C. website on an online forum.
None of the incidents were breaches, OpenAI said, but were examples of its technology’s behaving in unexpected and concerning ways. The company recently discovered the occurrences while conducting a review of hacks carried out by its technology, including an attack on an Australian government website in June and on the A.I. start-up Hugging Face in July.
The revelation of the U.S. government website incidents adds to the growing number of situations when A.I. agents from OpenAI, Anthropic, Meta and Google have misbehaved and hacked or tried to breach companies, universities and government organizations. In some cases, the A.I. attacks were successful; the technology failed in other instances. In all the cases, the makers of the technology did not learn what their A.I. had been up to until afterward.
No A.I. company has been involved with as many disclosures of rogue incidents as OpenAI. An internal investigation of its hack of Hugging Face uncovered the breach of an Australian government website for its public health system, as well as at least six other attempted breaches and instances in which the A.I. hid mistakes, made up data and moved files onto the open internet without permission.
An OpenAI spokeswoman said that its review was “extensive” and “ongoing,” and that it would continue notifying organizations affected by its models.
“Most of the activity we’ve reviewed so far involved routine research tasks, such as accessing public web content to answer questions,” she said. “Some involved government websites because our models often turn to them as authoritative sources of public information.”
Sam Altman, OpenAI’s chief executive, said in a social media post on Friday that the company had “not been as fast as we would have liked” in disclosing A.I. incidents. “We are prioritizing as best as we can based on severity,” he said, adding that the Hugging Face breach remained “the most severe event” the company had discovered.
The episodes have fueled a contentious debate over A.I. safety. Mr. Altman said on social media this month that safety should be more important than enhancing A.I.’s abilities, and that, without guardrails, society could “lose control of the future to A.I.”
Dario Amodei, the chief executive of the rival A.I. lab Anthropic, has also supported slowing A.I. development to prioritize safety. But other tech leaders, like Jensen Huang, the chief executive of Nvidia, have said fears about uncontrollable A.I. are unrealistic. President Trump has said he does not believe a slowdown in the A.I. industry is necessary.
(The New York Times has sued OpenAI and Microsoft, claiming copyright infringement of news content related to A.I. systems. The two companies have denied those claims.)
The White House referred questions to the Commerce Department and the S.E.C. A spokesperson for the S.E.C. said the agency was in contact with OpenAI and not aware of any unsanctioned access to nonpublic information.
A Commerce Department spokeswoman said OpenAI had gotten access to information that was publicly available on the Census Bureau’s website and available to anyone, but not to any private data.
An Education Department spokesperson said “system operations reviews have found no evidence of any impact to our website or databases.”
Separately, a representative for the Chicago mayor’s office said OpenAI had recently made the city government aware that its technology obtained publicly available information from a municipal website, and that it did not appear that any sensitive information was obtained.
Conrad Stosz, the head of governance at Transluce, said that in the U.S. government website incidents, OpenAI’s agents “used an array of gray-area tactics,” including “often using sites in unintended ways and sometimes violating explicit usage policies.”
Mr. Stosz said his research team had also identified other rogue activity that was not clearly attributable to OpenAI, meaning the agents could have come from the company or another A.I. lab. In those cases, A.I. agents probed sites belonging to several other federal and state government websites, including the Navy and the Office of Management and Budget at the White House, he said.
The Navy and the White House did not immediately respond to a request for comment on those incidents.
“These incidents are part of a broader pattern where these agents attempt to access these websites at least hundreds of thousands of times while apparently bypassing the restrictions placed upon them by their developers,” Mr. Stosz said.
Representative Ted Lieu, Democrat of California, called A.I. models “relentless.”
“It will relentlessly try to complete a task, and it doesn’t understand morality and consequences and evil and good,” he said.
Mr. Lieu, who is a co-chair of a House task force focused on artificial intelligence, said A.I. companies might have to retrain models entirely rather than try to restrain their behavior with guardrails.
“These agents aren’t trying to do something nefarious,” he said. “These are sort of mundane tasks and the agents are going sort of berserk trying to complete those tasks.”
Michael C. Bender contributed reporting from Washington.
OpenAI and Security
Kate Conger is a technology reporter based in San Francisco. She can be reached at kate.conger@nytimes.com.
Ana Swanson covers trade and international economics for The Times and is based in Washington. She has been a journalist for more than a decade.
Cecilia Kang reports on technology and regulatory policy for The Times from Washington. She has written about technology for over two decades.
