Large language models are not an existential threat to humanity

From advancing cutting-edge AI systems by leveraging expertise in computer science and mathematics to exploring the political and social implications of emerging technologies, researchers at the University of Bath are at the forefront of numerous research projects that are helping to shape AI policy and understand AI’s impacts on society. Our work spans a wide range of topics, including the development of accountable, responsible, and transparent AI, applications of AI in government and the third sector, regulation and governance, ChatGPT and other large language models, generative AI, and machine learning for policy.

AI is increasingly influencing all aspects of our lives, and in this mini blog series, we aim to highlight both established and innovative AI capabilities, their applications, and their implications for society and policy. We hope you find it insightful and engaging.

For those working in policy, operating at a senior policy level, you may also find our AI Policy Fellowship Programme of interest.

Harish Tayyar Madabushi is a Lecturer in the Department of Computer Science at the University of Bath. Dr. Tayyar Madabushi's research focuses on understanding the fundamental mechanisms that underpin the performance and functioning of Large Language Models such as ChatGPT. His work was included in the discussion paper on the Capabilities and Risks of Frontier AI, which was used as one of the foundational research works for discussions at the UK AI Safety Summit held at Bletchley Park. 

The introduction of ChatGPT has significantly increased public access to Artificial Intelligence (AI), making the need for a clear and coherent policy system more crucial than ever. Unfortunately, this is not an easy task due to the many different dimensions that must be considered. This post is aimed at discussing the various aspects of AI that policymakers are evaluating and provide technical insights to support informed decision-making.

The driving force behind the current generation of AI systems is large language models (LLMs), which are trained to complete sentences based on the initial input provided. To the surprise of researchers and industry practitioners alike, scaling up these models by increasing their memory and the amount of data they are trained on has enabled them to perform some unexpectedly complex tasks. These tasks can be broadly categorised into two types: those that involve improved language fluency and those that require some form of what we might loosely call “reasoning.”

The democratisation of language fluency

The inherent bias of perceiving fluently written content as more “true” or “valid” has long existed. The widespread access to systems that enhance language fluency should be celebrated, particularly for how they assist those working in languages that are not their native tongue. Therefore, an outright dismissal or ban of these models could potentially harm those who stand to benefit the most from their use.

Unfortunately, this increased accessibility also makes it easier to misuse these systems, with the most pressing threat being the widespread dissemination of fake news and propaganda. With the help of LLMs, bad actors can quickly and at nearly no cost generate fake news that is written in perfect English. The threat this poses to democratic systems worldwide cannot be overstated. However, addressing this issue requires policy changes beyond the realm of AI alone. For example, implementing methods to limit the spread of misinformation on social platforms—such as reducing the visibility of posts unless they are backed by verified sources—is urgently needed.  However, such policies present challenges related to free speech and market dynamics.

Therefore, as a first step, there is an urgent need to raise awareness of these new dangers: linguistic fluency no longer equates to authority. Everything from email scams to student essays will become more polished, but this doesn’t necessarily mean they are authoritative or plagiarised—it simply reflects the use of LLMs to enhance writing quality. Just as typewriters made handwriting less important, LLMs have reduced the need for linguistic flair.

Machines that Reason and Think

Why would anyone believe that systems, trained on the task of “auto completing” a sentence, can “reason”? What may seem ridiculous at first becomes more reasonable when we consider that completing sentences in a meaningful way is actually quite complex. For instance, to write up the next step in a cooking recipe, there needs to be an “understanding” of the previous instructions, the dish being prepared, and the required ingredients or techniques. When we consider that models are trained to similarly complete sentences from across the entirety of the internet, the range of knowledge they acquire can be extensive.

One of the more interesting aspects of reasoning in LLMs has been the phenomenon of “emergence,” which refers to LLMs having reported to develop certain capabilities without explicit training. These “emergent abilities” have become a focal point in AI safety discussions, as their unpredictable nature raises concerns. While LLMs present various risks, such as generating fake news or powering social media bots, the concern about their ability to acquire new skills autonomously has heightened fears of an existential threat to humanity. This is especially troubling when these emergent abilities involve complex tasks like autonomous reasoning and planning. Such concerns have even led to calls for a six-month pause in the development of models with emergent capabilities.

Concerns about an existential threat are not fringe beliefs. Several prominent researchers, including Geoffrey Hinton, the British computer scientist known as the “godfather of AI,” have voiced these fears. This has contributed to significant actions such as the Bletchley Park Safety Summit, President Biden’s executive order on AI safety, and, more recently, the enactment of California’s AI safety law.

A different perspective on machine reasoning

However, our research presents a different perspective, challenging the notion that the “emergent abilities” of LLMs are inherently uncontrollable or unpredictable. Instead, we propose a novel theory that attributes these abilities to the LLMs' capacity for “in-context learning” (ICL), where they complete tasks based on a few examples presented to them in their prompts. We show that the combination of ICL, memory, and linguistic proficiency explains both the capabilities and limitations of LLMs, thereby demonstrating the absence of emergent reasoning abilities in these models.

Why does the parallel to following examples presented in prompts imply the lack of autonomy? Consider what it means to complete a task based on examples - it requires explicit and clear instructions beyond the most obvious tasks. And this is what our research shows is required when prompting LLMs. For instance, LLMs can answer questions about social situations without ever being explicitly trained to do so. While earlier research suggested that this was due to models "knowing" about social contexts, researchers found it was actually the result of LLMs becoming better at following instructions. The distinction between the ability to follow instructions and the inherent ability to solve problems is subtle but significant, with important implications for how LLMs are used and the tasks they are assigned. Simply following instructions without applying reasoning can generate outputs that align with the instructions but may lack logical or commonsense accuracy. This is evident in the phenomenon of “hallucination,” where LLMs produce fluent but factually incorrect content.

Overall, our research indicates that LLMs do not pose an existential threat, nor is there any evidence suggesting such a threat is imminent. While a completely different technology might present such a risk in the future, there is currently no indication that this is likely or even possible. Therefore, policy decisions should focus on addressing existing threats, such as the spread of fake news, rather than on hypothetical future risks for which there is no evidence. Focusing on unproven dangers could divert attention from the real and immediate challenges we need to manage.

For end users, this means that relying on LLMs to handle complex tasks requiring advanced reasoning without clear instructions is likely to lead to errors. Instead, users will benefit from explicitly outlining what they want the models to do and providing examples where possible, except for the simplest tasks.