Elite law firm Sullivan & Cromwell admits to AI ‘hallucinations’
FT.com Firm whose partners bill more than $2,000 per hour apologises to judge for software-driven errors in bankruptcy case [no paywall] – Sullivan & Cromwell told a US federal bankruptcy court that a major filing it made in a high-prrofile case contained multiple “hallucinations” made by AI software.
Andrew Dietderich, the head of S&C’s restructuring practice, apologised in a letter to New York federal judge Martin Glenn on Saturday for mistakes that included misquoting the US bankruptcy code and citing cases incorrectly in a court filing made on April 9. “We deeply regret that this has occurred,” he said in the letter.
Dietderich said the firm’s policies on the use of AI had not been followed when the document was prepared, and it was considering whether it needed to make “further enhancements” to its internal training and review processes.
The letter did not say which lawyers prepared the documents or whether they were still at the firm. S&C declined to comment. The errors are the latest example of a professional services firm grappling with the use of cutting-edge technology to speed up laborious research and cut down on staffing while also trying to maintain quality standards…
S&C has an enterprise licence for ChatGPT according to multiple people familiar with the firm’s operations. According to S&C’s website, at least five high-level partners have been assigned to the Prince Group bankruptcy case. The firm’s partners typically charge more than $2,000 per hour in bankruptcy cases. The firm earned several hundred million dollars in fees in representing crypto exchange FTX in its bankruptcy liquidation. Boies Schiller Flexner, the law firm representing Prince and Zhi, spotted the errors in S&C’s filing. In a document filed last week, BSF said words that S&C had quoted in its motion “do not appear in chapter 15 of the US Bankruptcy Code” and pointed to “multiple cited decisions” that were “misquoted or misidentified”. It said a case cited by S&C in the motion “is not a case” and the reference was to “a different decision in a different circuit”. S&C told the court that the firm maintained “rigorous” standards when using AI tools and that it “instructs lawyers to ‘trust nothing and verify everything’”. Failure to verify AI-generated output “constitutes a violation of firm policy”, it said. It is the latest in a series of errors by law firms using AI tools. Last year, Latham & Watkins admitted that one of its lawyers had used Anthropic’s Claude model to help draft a filing which contained an apocryphal title and author for a journal article. In another instance, a federal appeals court in New Orleans ordered a $2,500 sanction against a lawyer who had submitted a brief with 21 errors or fabrications that had been inserted by AI. Separately, in September John Kucera, then a partner at BSF, said in a case against Amazon that a document for which he was responsible, prepared using AI tools, contained “material citation errors” due to his “failure to verify” details. “I am embarrassed by and very much regret these errors”, he said in the filing. BSF did not respond to a request for comment. S&C told the judge overseeing the Prince Group case that its document review also showed “non-substantive and/or clerical errors in other filings in this matter”. The firm said those errors were made by humans, not AI…”
ATO culture keeps a cap on agentic AI ambitions
KEY POINTS
- The ATO believes agentic AI might only be needed for a small portion of an end-to-end process, with re-engineering, automation and machine learning covering most use cases.
- Copilot Chat is available to as many staff as possible, but its agentic component remains switched off due to the ATO's sensitive data holdings.
- The ATO has shifted from 'human in the loop' to 'human at the helm' to ensure accountability for AI-driven outcomes.
The Australian Taxation Office is taking a cautious approach with agentic AI, reflecting its internal culture, risk appetite and sensitive data holdings, but is undertaking some initial architectural preparation work.
Assistant commissioner for strategy, design and architecture Jimmy Tzimopoulos told an IBM-run session at Gartner’s IT Symposium/Xpo that, at present, the internal view is that agentic AI might be utilised in only a small portion of an end-to-end process.
“What we’ve found, especially with agentic AI, [is that] you can achieve a lot of the use cases, but just because you can, doesn't mean that you should,” Tzimopoulos said.
“There's a tendency to say, ‘I'll just get it all done by agentic.’
“But when we really assess the different use cases, we might find that 95 percent of them could be achieved through some form of re-engineering of process and then driving business automation, then maybe some machine learning, and then a bit of agentic at the end or as a component of it.
“I think that's the thing for us that we're learning about.”
Tzimopoulos highlighted the importance of safely learning about what agentic AI is and isn’t useful for.
“You've got to create an environment for people to learn with this capability,” he said.
Copilot could become a way that many staff are exposed to agentic technology within the ATO, although Tzimopoulos indicated that agentic capabilities for Copilot Chat are yet to be switched on.
“In the ATO, we use Copilot Chat, so we're making that available to as many people as possible to use,” he said.
“We haven't turned on the agentic component of that because we've got sensitive data, and we want to make sure that we understand how we use that.”
Tzimopoulos said the ATO is “architecting for the agentic space” - suggesting some behind-the-scenes preparatory work - but that the design and progress reflects ATO’s culture and risk appetite.
“As you would expect, we are a very risk-averse organisation,” he said.
“We've got the largest and most valuable data store of personal and business financial information in the country.
“So naturally, we're very cognisant and cautious to make sure that we treat that information appropriately.”
Tzimopoulos said that one of the principles guiding the ATO’s approach with agentic AI is its internal culture.
“You've got to architect your AI based upon the culture of your organisation,” he counselled.
“If you're risk-averse, make sure that you've got that reflected in how you deploy AI. Don't be someone that you're not.”
The ATO has also adapted the ‘human in the loop’ safeguard for AI to ‘human at the helm’.
“I know it's only a slight tweak of language, but it goes to the accountability piece,” he said.
“Excuse the language - someone's arse has got to be on the line for what you're trying to achieve because in the end, we're accountable for the outcomes to all citizens and residents of Australia, and therefore, that human really needs to be accountable for that.”
Forensic value analysis
Tzimopoulos also theorised that the AI era would create demand for “IT forensic accountants”, a role more commonly seen in financial crime contexts.
He suggested that such accountants could bring the skills required to forensically understand “the true total cost of ownership of delivering a service” that utilised AI componentry.
“We're investing a lot for our use [AI] cases to actually understand what is the right financial models that support that, and also how do we present that information back to our business?” he said.
“For us, we've got an idea of potentially a balance sheet. Obviously the ATO will understand that.
“But if we can actually present to a business line a balance sheet that shows who's using it, and what the TCO is, and then they can correlate that back to how much they're actually paying for the service, you’re starting to actually get them to think about what is the true value of this end-to-end process.
“You’re also getting them to think, ‘Do I need to do [a process] holistically using agentic, or can I actually compartmentalise it?’”
