Less than 24 hours after OpenAI published more than 700 preprints describing key mathematical results by an in-house AI model, the company withdrew three of the manuscripts for errors.
On October 6, OpenAI released 722 preprints to a GitHub repository on purported progress in solving 372 math problems in geometry, computer science, algebra and other subjects. The society called the publication an effort “to push the boundaries of human knowledge and enable new advances in mathematics.”
But on October 7, OpenAI announced the withdrawal of three manuscripts due to a signature error. The error invalidated an argument in one manuscript and the construction used by two dependent articles, according to the announcement. Each of the retracted articles has notices explaining the “discrepancy” and that OpenAI has “revised 14 additional manuscripts with proof repairs, corrected statements, clearer assumptions and dependencies, and a correction of an outdated citation.”
That an error spread across three of OpenAI’s manuscripts is not surprising, said Alex Townsend, an associate professor of mathematics at Cornell University in Ithaca. He suspects more errors will be found, he said.
“Given the skepticism of AI in the math community, I think OpenAI should have announced manuscripts that have been Lean verified first,” Townsend told us, referring to Lean, the open source programming language that verifies mathematical proofs. “They could have made a separate announcement for the others and asked for help to verify them. That would have been clearer to the community and the general public.”
An OpenAI spokesperson told us the company identified the errors during an audit. As with “other research manuscripts,” the process is iterative, the spokesperson said.
“We appreciate the careful review and feedback from the mathematics community,” the spokesperson told us. “When errors are identified, we will work to correct them quickly and remove articles if a solution cannot be found. We want to give the community the time and space to evaluate the work, and we continue to improve the clarity and presentation of future articles.”
In a post on
The OpenAI spokesperson told us that the company’s collaborators on the Mathematics and Artificial Intelligence Advisory Group recommended publishing the results without waiting for “full formalization” and that around 50% of the results were published without confirmation.
“We hope to learn from the feedback on this release and work with AGMAI to understand whether they have updated recommendations regarding release protocol in the future,” the spokesperson said in an email. “Our intention is to follow their protocol as best we can.”
Andrew Sutherland, a senior research scientist in the mathematics department at the Massachusetts Institute of Technology, said he appreciated that OpenAI acted quickly after discovering the errors, calling it “the responsible thing to do.”
However, Sutherland pointed out that many mathematicians are “unhappy with the way OpenAI has behaved so far,” particularly in its handling of the Sept. 8 announcement that an AI system had solved the Navier-Stokes problem, which asks whether a series of equations describing how gases and liquids move become unreliable at some point.
After the publication, a group of prominent mathematics researchers expressed concern that OpenAI rushed Navier-Stokes’ announcement “leaving no time to write a proper paper, isolate new methods and ideas, and cite relevant previous work by others.” More than 8,000 researchers agreed with these sentiments.
“I think the removals and corrections will be viewed positively by the mathematics community, but it will take much more than that to regain the trust they have lost,” Sutherland told us.
The Association for Human Mathematics also criticized the most recent version. “Mathematicians did not request that this work be done,” said the group’s statement, published October 7. “Publishing more than 700 files at once is not a display of scholarship, but a display of power. We urge mathematicians to discontinue their work with OpenAI and return to a vision of science centered on human understanding.”
Like the retractable watch? You can make a tax-deductible contribution to support our workFollow us on or Bluesky, like us on Facebookfollow us on LinkedIn, add us to your RSS readeror subscribe to our daily summary. If you find a retraction, it is not in our databaseyou can let us know here. For comments or feedback, email us at (email protected).
Related
Gn bussni
