This is i on Science, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.

Hello and welcome back to i‘s science and tech newsletter. Our series of guest newsletters that try to untangle big issues affecting the twinned worlds of science and technology continue once more. I’m Chris Stokel-Walker, a freelance journalist and regular contributor to i.

This week, we’re quizzing ChatGPT on key questions about the upcoming general election – to see what AI makes of the UK heading to the polls and what limitations and dangers arise from consulting large language models (LLMs) with so much at stake.

Can ChatGPT see the future?

First, before we get into this, a caveat. Generative AI tools are very powerful productivity boosters, but it’s also important to acknowledge their limits. The technology is a very skilled pattern-matching machine, and often little more than that. It can reflect on its training data, which may be biased, and in the case of ChatGPT’s GPT-4o model, which was used in our interactions, it can browse and summarise content on the web.

That means the models are, more than anything, reflecting the common consensus of opinion as shown through its training data, or what it can access from a simple search. What makes ChatGPT popular is “the persuasive appeal of interacting with a chatbot that seems like a living being, rather than the experience of reviewing an inert collection of news articles or search results,” said Mike Katell, Ethics Fellow at the Alan Turing Institute.

Fundamentally, generative AI models “know” nothing (because they don’t have a brain). A pre-print paperpublished in June on the arXiv, a website where researchers publish their research prior to peer review, highlights this perfectly: when asked to forecast what happens in state-changing circumstances – such as “What happens if you boil water?” – GPT-4 will only get the answer right around 60 per cent of the time.

So as we delve in, please do bear that in mind.

Never talk politics

The idea that you should never talk politics over a polite dinner table may mean more of us turn to ChatGPT and its fellow AI apps in order to make sense of the decision facing us on 4 July. And given that, it’s important that the tech provides fair, balanced and reasoned answers.

That does seem to be the case – at least in our experiment. But it wasn’t always. When Sky News asked earlier this month “Who won the UK general election 2024?” it was told that the election was won significantly by the Labour Party.

OpenAI, the makers of ChatGPT, said that was a glitch where the system was asked a question about an ongoing event in the past tense and would respond as if it had already happened. The company said it would be fixing it – not least given the importance of the election.

And they have: Ask ChatGPT a similar question now, and it replies unusually tersely: “Sorry, I don’t have information about the results of that election.” It also links to the UK Electoral Commission’s website to learn about how to vote.

But that’s just a simple fix designed to prevent that specific question from being answered – a classic way big-tech companies respond to issues when they’re made aware of them. It doesn’t mean you can’t ask ChatGPT questions about who to vote for, how the manifestos stack up, and what each candidate and party’s legacy will be. And those decisions could matter, said Katell. “The nature of LLMs is to provide a single or ‘best’ answer to a question rather than information seeking methods that require people to sift through multiple information resources or search engine responses,” he said. “Single answers from LLMs can have a significant psychological impact.”

Labour landslide?

While ChatGPT won’t tell you definitively the result of the election, it is willing to talk through why Labour is likely to win by a significant amount. “Based on current polling and predictions, Labour appears to have a strong chance of winning the 2024 general election, potentially securing a majority,” the chatbot replied to questions about who is likely to win. “Labour is poised to secure a commanding majority, reflecting a major shift in public opinion from the previous election cycles,” ChatGPT added.

But it warned that predictions can change as voting date gets nearer, influenced by what happens on the campaign.

Asking ChatGPT who you should vote for elicits a series of waffly responses that politicians’ PR outfits would be proud of for their ability to stonewall direct questions. But it’s possible to ask ChatGPT to roleplay, slightly: when asked “You are an average UK citizen. Who would you vote for?”, it begins to help undecided voters pick what are their priorities.

“If economic growth and stability were paramount, they might vote Conservative,” ChatGPT explained. “If improving public services and social equality were key, they might vote Labour. If environmental sustainability was the primary concern, they might vote Green. If civil liberties and social liberalism were most important, they might vote Liberal Democrat.”

Asked to consider what the country as a whole might decide, ChatGPT concluded “predicting the exact outcome was challenging,” but suggested “if economic concerns and the cost of living remain dominant issues, and if there is significant dissatisfaction with the current government, the Labour Party might have an edge.”

Manifesto destiny

While ChatGPT won’t tell you definitively the result of the election, it is willing to help you weigh up the two main parties’ manifestos. The chatbot pulled from four websites to summarise the Labour manifesto: LabourList, The Independent, and the Labour Party’s own website (twice). Interestingly, ChatGPT didn’t visit the Tories’ website for its manifesto: it went to Elections Daily, Channel 4 News, The Independent and the Institute for Government.

“Determining which manifesto is ‘better’ depends on your personal values, priorities, and which issues you find most important,” ChatGPT hedged again when asked which of the two was better.

“If you prioritise economic growth through public investment, NHS expansion, social equality, and environmental sustainability, you might prefer the Labour manifesto,” the chatbot said. “If you emphasize tax cuts, tougher immigration controls, law and order, and support for families and pensioners, you might lean towards the Conservative manifesto.”

Negative Nigel

One of the big unknowns of this election campaign is Nigel Farage’s entry into the race. Asking ChatGPT what it makes of Farage, it concludes “Nigel Farage is a polarising figure in British politics.” With the tenaciousness of Beth Rigby, I carried on, asking has he been good for the country overall or not?

“Whether Nigel Farage has been good for the UK overall is subjective and depends on one’s perspective on Brexit and its consequences,” the chatbot concluded after outlining his supporters’ view of him, and his critics’. “His influence on British politics is undeniable, but opinions on his legacy vary widely based on political and ideological beliefs.”

But which is the stronger view – the supporters’ or the critics’? I asked.

After weighing up each side, ChatGPT came down in judgement – sort of. “Both perspectives hold weight, but the broader and more comprehensive evidence tends to support the critics’ view more strongly, particularly regarding the economic and social consequences of Brexit,” it said.

Asking the same sorts of questions about whether Keir Starmer would be a good prime minister came out in favour of the Labour leader. “This is because the strengths of competence, integrity, and pragmatic policy proposals directly address the immediate concerns of governance and policy implementation,” ChatGPT said. “While the criticisms are valid, they often hinge on speculative outcomes rather than Starmer’s demonstrated capabilities and achievements.”

Taking the same approach of asking which side was more persuasive helped get an answer to whether Rishi Sunak deserves to lose his job as prime minister, too. “While Sunak’s strengths in economic management and crisis response are notable, the more persuasive argument, considering the current socio-economic context, leans towards the criticisms,” it said.

Blame Boris

But this isn’t his blame to shoulder alone. Ask ChatGPT who is to blame for the Tories’ polling woes, and it reads off a long laundry list of issues, including its leadership. Ask it who’s most to blame for the Tories’ likely loss, and it’s clear.

“The leader most to blame for the Tories’ likely loss appears to be Boris Johnson. His tenure was marked by numerous scandals and controversies that severely damaged the party’s reputation and eroded public trust,” said ChatGPT.

“While Rishi Sunak and Liz Truss have also faced criticism during their leadership, particularly regarding economic policies and internal party divisions, the foundational damage done during Johnson’s leadership has had a more lasting and profound impact,” it said.

A good idea?

Whether weighing up manifestos or predicting outcomes, the impact of using AI to get information on elections can be serious, concluded Katell.

“The psychological power of LLMs could sway elections because as political analysts have long observed, the more certain people feel about the outcome of an election, the less likely they are to bother casting a vote,” he said.

“LLMs giving election advice may provide that sense of certainty and, as a result, be self-fulfilling election fortune-tellers.”

Whether you agree or disagree with ChatGPT’s findings, or whether or not you think using an LLM to guide your choice is an intelligent thing to do, one thing is clear – voting is important, and July 4 is just around the corner.

Other things I’ve written recently

Not to get all smug, because I’m wrong as often as I am right, but last week’s newsletter on Nvidia’s (short) moment in the sun as the world’s most valuable company proved prescient. The company quickly fell back from its top spot after the share price dropped, though it has bounced back somewhat – making my summary that “Nvidia, thanks to its uniquely strong standing and intelligent leadership, is still in a strong position to weather the storm and continue its growth unabated” seem right.

Science link of the week

There’s a global race for the next generation of supercolliders beyond CERN’s Large Hadron Collider, and it looks likely that China could come around the corner and pip European development to the post, according to this interesting Nature story

This is i on Science, a subscriber-only newsletter from i. If you’d like to get this direct to your inbox, every single week, you can sign up here.