While much attention has focused on Anthropic’s updated financials revealed Monday in its revised S-1A (revenues skyrocket, as do expenses), the AI lab dropped another little bombshell in the fine print.
The filing appears to warn that the company is close to creating a model capable of recursive self-improvement (RSI).
ROI is the point at which the AI becomes so effective at self-training that it results in a dramatic acceleration of the AI’s capabilities. For AI doomers, RSI represents the cataclysm of no return, in which AI builds itself into something powerful enough to escape human control and be indifferent to the damage it causes while still achieving its goals.
For AI enthusiasts, RSI, or a version of AI that can be reliably used to improve itself, represents the promise of technology, where it becomes a tool for productivity, scientific discovery, and much more beyond imagination.
Anthropic seems to believe that RSI is both a promise and a peril. The company says in the S-1 that it uses AI in its R&D, but it cautions: “Using AI to help build better AI could also pose a significant security risk, particularly if it sharply accelerates the pace of progress. This is one of the areas covered by our Responsible Scaling Policy, which sets out our plans to monitor and mitigate risks as we approach what is known as “recursive self-improvement”.
When the term ROI became popular earlier this year, many thought it was a buzzword or, at best, a goal that was too difficult to achieve, define, or prove.
But on Monday, Anthropic added new statements to its Risk Factors section, claiming 1) that it has already used AI to develop new models whose advanced capabilities it had not recognized, and 2) that it may soon meet its own internal definition of ROI (emphasis added, bold, below):
Additionally, our increasing use of AI systems to accelerate our own research and development may advance the capabilities of our models faster than our ability to develop, validate and implement corresponding safety, alignment and security measures. Therefore, we have not been identified in the pastand may in the future fail to identify a significant acceleration in our pace of development until it has occurred.
For example, our internal estimates and expectations about future models, rather than our current models, indicate that we could soon meet our responsible scaling policy definition of “significantly accelerated progress attributable to AI.” We have not designated any of our current models as exceeding this threshold and we continue to evaluate the underlying data.
However, Anthropic also said in the filing that none of its models had yet crossed this RSI threshold and that it promised to keep humans in the R&D process for the foreseeable future. “We do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models without significant human supervision or understanding, if and when this becomes technically possible. As a result, we plan to update our responsible scaling policy to keep humans in control of frontier model development at Anthropic.”
Skeptics might point out that an S-1 functions as both disclosure and marketing, and that claims that it is close to a model so powerful it can build itself might fall into the latter category.
Nevertheless, there is no doubt that Claude’s abilities, particularly in terms of coding, mathematics and cybersecurity, have progressed by leaps and bounds this year. If Anthropic believes the possibility of ROI is near, perhaps we should believe it and act – perhaps govern – accordingly.
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
Gn bussni
