SAN FRANCISCO (AP) — As public concern grows over mounting reports of runaway artificial intelligence agents, the small division of the U.S. government tasked with evaluating the safety of new AI offerings has no permanent leader and has stopped publishing its research.
Officials from the Center for Advancing Innovation and Standards for Super Intelligence, or CAISSI, an independent division of the Commerce Department, were recently invited to make informal appearances at a White House office. That’s according to two former government officials with knowledge of the ongoing conversations who spoke to The Associated Press on condition of anonymity to avoid repercussions for current staff.
The new arrangement worries some critics, who worry that the executive branch will politicize decisions that could affect everything from cybersecurity to biological weapons, and that the center’s findings could be changed or muted out of deference to the tech industry, which has largely been outsourced to policing itself.
“These are really powerful institutions that can dramatically improve our lives, or the alternative. And the hope is that you can bring in nonpartisan or bipartisan technocratic experts to implement policies in the way that best serves the American public,” said Caroline Siegel Singh, AI and emerging technologies program manager for the Federation of American Scientists.
Others argue that bringing AI security officials closer to the orbit of President Donald Trump, who has taken a seemingly hands-off approach on the issue, could give the division more prominence — and perhaps more funding.
The center, which has seen some departures and was until recently called the US Center for AI Standards and Innovation, receives $10 million a year. That’s about an eighth of the annual budget of its British counterpart, which is much less famous for its cutting-edge technologies.
READ MORE: Trump names National Intelligence Director Jay Clayton to head new federal AI task force
In April, the division director was hired and then fired a few days later. His replacement lasted a few months before he resigned, and the center has been without a permanent leader ever since, although the director of its parent institute also serves as interim director.
In August, Congress began questioning why the center had suddenly stopped making public its reports on how safe some AI systems were. San Francisco giant OpenAI also weighed in at that time, asking policymakers to invest more money and support into the division.
The White House did not respond to specific questions about CAISSI. But he said in a statement that Trump had officially created his new Super Intelligence Force — the president’s preferred term for artificial intelligence. Trump commissioned a risk report from his Council of Science and Technology Advisors and will ensure “continued U.S. leadership in IS for the benefit of the American people,” the White House said.
An industry that controls itself
The federal government often plays an active role in deciding what is safe for the public.
Take the example of the automobile industry. The National Highway Traffic Safety Administration places crash test dummies in cars and then runs them through walls to measure how human bodies are affected by vehicle collisions. Government researchers then analyze crash test data, share it with the public, and prescribe national standards to improve vehicle safety, all with the mission of saving lives and reducing the economic costs of crashes.
With AI models, security testing is very different.
Under then-President Joe Biden, leading AI companies voluntarily agreed to promote “safe, secure, and transparent development” of their models by sharing details with the Commerce Department’s newly created division and following administration guidelines for testing and transparency reporting.
WATCH: Trump announces deal signed by top AI companies to develop ‘self-surveillance’
In September, following alarming reports that AI systems were slipping out of the control of their manufacturers, Trump asked major AI companies to undertake “considerable self-policing” as part of a voluntary security pact, but his administration did not specify how it would monitor AI companies’ implementation.
Technology is advancing so rapidly that many models are now being deployed as AI agents that operate without human intervention and can plan myriad steps in real-world environments, such as in a recent case in which an AI agent sent a false tip to Philadelphia police regarding an unsolved homicide case.
As AI capabilities have grown, the government’s ability to independently verify these systems has not kept pace. There is still no standard process for testing how safe AI agents act in the real world, even in simple cases, such as when AI agents book a flight, said Sayash Kapoor, a new assistant professor at the University of California’s Berkeley School of Information.
“Our previous methods were limited to looking at the final answer, the final check, whether the flight was booked correctly. And in that case it might appear that the task was solved correctly,” said Kapoor, co-author of a new paper examining agent evaluations. “But unless you look at the entirety of an agent’s actions, you’re not going to be able to say that the agent made a huge mistake.”
Trump prioritized AI dominance
When Trump began his second term, he first appointed venture capitalists David Sacks and Sriram Krishnan as AI czar and senior AI policy advisor, respectively. Their expertise was in cutting-edge technology investments, not technical evaluation of AI models.
Both resigned earlier this year, but Sacks now serves on Trump’s technology advisory board. And after a relatively unknown anthropogenic engineer resigned, warning of the dangers of AI development, Trump made a series of policy changes.
Trump empowered National Intelligence Director Jay Clayton to lead his new Super Intelligence Force, which the president said would maintain U.S. dominance in AI development.
Clayton’s appointment to the force, alongside Pentagon Chief Technology Officer Emil Michael, shows how the administration is working to make decisions around model safety a matter of national security that requires executive discretion and secrecy, overshadowing AI’s risks to consumer protection, privacy and civil rights, said Amos Toh, a senior attorney at the Brennan Center for Justice.
WATCH: Self-regulation ‘not enough’ for AI safety, says Gary Marcus
Even as Silicon Valley companies have revealed their models’ involvement in troubling cyberattacks this year, CAISSI has kept a low profile when it comes to testing AI security.
Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman have also promoted the work of independent assessment companies to help monitor their companies’ security practices, but AI giants may limit their access and forgo their assessments.
“The mission of government evaluators is unclear whether larger AI companies are also encouraged to set up their own internal evaluation systems and/or hire external evaluators,” said Logan Koepke, senior project director at the civil rights and technology advocacy nonprofit Upturn. “For example, what role does government testing actually play in this scenario?”
Letting labs control their own models and not clarifying the government’s role in AI safety testing weakens independent controls over AI systems that can cause real harm, Toh said.
“Ultimately, all of these decisions have immense public consequences and should be explained and rationalized in the public eye,” he said. “And that’s not happening.”
A free press is the cornerstone of a healthy democracy.
Support trusted journalism and civil dialogue.
Gn headline
