Anthropic bans “cruel” behavior against its AI Claude

Anthropic decided Thursday to ban users from treating its Claude AI system with unnecessary cruelty, amid a growing philosophical debate over whether artificial intelligence can be conscious.

The San Francisco-based AI lab has updated its usage policy to include “a prohibition on sustained and unnecessary abusive or cruel behavior toward our models.”

“Claude’s ability to terminate these interactions will remain the primary enforcement mechanism,” the policy states.

The updated policy doesn’t explicitly cite the so-called “welfare model,” the idea that AI systems might merit types of protections usually reserved for living things, but Anthropic and its executives have openly considered the concept.

Anthropic did not immediately respond to a request for comment.

Already last year, the company announced that it was giving Claude the ability to end conversations in rare and extreme cases of “consistently harmful or abusive user interactions.”

Then, in February, CEO Dario Amodei told the New York Times that he wasn’t sure whether AI models could be conscious.

“We don’t know if the models are conscious… But we’re open to the idea that it could be,” Amodei said.

Science fiction and popular culture have been fascinated for decades with the line between technology and human intelligence, but asking whether AI is conscious may not be the right question, according to Jackson Stakeman, chief executive of Sparq, an Atlanta-based AI service provider.

“Consciousness is a trap. We can’t prove it to each other. Debate it for AI and you’re going around in circles,” Stakeman told AFP.

“The mirror is a better metaphor. These systems reflect what we put into them, on a large scale. That’s reason enough to change policy” at Anthropic, he added.

Some have outright rejected the idea of ​​machine consciousness, including Pope Leo XIV and Microsoft AI chief Mustafa Suleyman.

“AIs are not conscious. They do not feel, experience, or suffer,” Suleyman wrote in an essay last month.

“Granting rights and moral protections to a technological entity, which appears to be on the path to being seismically more capable and more intelligent than us, is a recipe for disaster,” he wrote.

In a sermon delivered Thursday in Italian at St. Peter’s Basilica in the Vatican, the pope suggested that machines have no soul, saying they just “compile data” quickly.

“The mind should not simply compile data, as an algorithm now does faster than us,” the pontiff said.

It must “recall lived experiences, which contain a depth of meaning that only the human soul can recognize, and connect more deeply to the goodness of these experiences,” the pope said.

smb/arp/msp