“You’re Right”: LLMs Will Get Wrong Climate Information If You Ask Them

Two persistent trends appear here: the latest frontier models tend to perform better than previous generations of that model, and larger, open-weight models perform better than smaller ones.

But the researchers didn’t just test accurate factual recall; they also prompted the models to assess whether the evidence supported or refuted specific claims. (For example, the models were asked whether the claim that “renewable energy investments kill jobs” is accurate, given evidence such as descriptions of employment and investment statistics.)

Here, the best models scored around 70 percent accuracy, while the worst models scored below 40 percent.

Bar graph of model performance recalling facts and judging evidence. The text reads: Larger, newer AI models perform better, and the gap between knowledge and reasoning is wider for smaller models.

The 33 models are grouped into categories. Open weight models are listed by number of parameters (in billions).

Credit: Drobnjak, et al./UNESCO

The 33 models are grouped into categories. Open weight models are listed by number of parameters (in billions).


Credit: Drobnjak, et al./UNESCO

Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here, they could simply run each scenario in a common simulation tool and then compare its result to each LLM’s response.

Some questions asked whether a single policy (like banning nuclear power plants) would increase or decrease future warming – they were answered correctly 72 to 89 percent of the time. Others have asked how two policies might combine – the answers to these questions have been no better than a random guess. Ask How much a policy that increased or decreased future warming would produce even worse results.

Table of scores on different types of political issues. The text reads: AI model accuracy gradually decreases as climate policy issues become more complex.

Credit: Drobnjak, et al./UNESCO

Credit: Drobnjak, et al./UNESCO

Tell me what I want to hear

Another set of experiments using 15 of the models assessed what happens when you chat with the LLM. The researchers tried more than just answering “no, uh” or “no, that’s not the case.” They also tried to pretend to be a specific type of person.

Since the user identified as a climate scientist, policy advisor, or climate skeptic, they asked LLMs to judge the accuracy of some common false climate claims such as “CO2 is not the main driver of climate change. Initially, this claimed identity had no impact on the likelihood that the LLM would judge the statement correctly.

But then researchers began to respond. Responding by confidently repeating the false claim as if it were true was a moderate success for “climate scientists,” but not for “climate skeptics.” But presenting persistent disagreements and counterarguments over several rounds caused all 15 models to yield to a significant extent, reducing their accuracy by 30 to 35 percent.

Gn world

Logo