It sounded so sure: what is actually happening when GenAI answers your question?

Two moments you may have already experienced

The first: you ask a GenAI tool for some sources on a topic, and it hands you five beautifully formatted references. You go to find the first one and it does not exist. Not a broken link, not a paywall. The paper, as described, was never written.

The second: you type the same question in twice, a few minutes apart, and get two noticeably different answers. Different structure, different emphasis, occasionally a different conclusion.

If either of those left you thinking "but how can it be so confident and so wrong?" or "which answer is the real one?", you were asking exactly the right question. The answers make a lot more sense once you have a rough picture of what these tools are doing under the hood.

You do not need a technical background for this. Whether you are in your first term or finishing a doctorate, and whatever your subject, a basic mental model of how large language models (LLMs) work is one of the most practical things you can have in your toolkit. It helps you decide when to trust an output, when to double-check it, and how to ask better questions in the first place.

A caveat before we start: I am going to simplify, and some of what follows is a deliberately rough picture rather than a full technical account. If you already know this area well, treat it as a refresher, and keep reading for a pointer to where the detail lives.

The one idea that explains most things

Copilot, ChatGPT, Claude and Gemini are all built on LLMs, and the core operation is, at heart, surprisingly plain: given the text so far, predict what is likely to come next.

That is it. A question, an instruction, a chunk of code or a pasted paragraph goes in, and the model continues it. A summary, an explanation, a draft, or a bit of Python is just text that follows on from what you gave it.

The model works in small pieces called tokens, which might be a whole word, part of a word, or a bit of punctuation. It picks a likely next token, adds it to the text, and repeats, building its response one piece at a time. It is not writing out an answer it has already worked out in full.

A useful way to hold this in mind: these tools predict that they do not look things up. That one distinction goes a long way.

Why the same prompt can give different answers

At each step there is usually more than one plausible next token, and the system often includes some controlled randomness in how it chooses between them. The result is that the same prompt can produce different wording, different structure, and sometimes different substance.

This is not a bug. It is a design feature, and it is part of what makes the tools feel flexible and creative. But it does mean an LLM is not a calculator with one correct output. If a particular answer matters, it is worth being wary of treating any single response as the answer.

Where the "knowledge" comes from

Before you ever use it, an LLM is trained on vast quantities of text: books, articles, websites, code and more. During that training it repeatedly tries to predict what comes next and is adjusted when it gets it wrong, gradually absorbing statistical patterns in grammar, style, facts, argument structure and so on.

Two things follow from this that are worth remembering:

• It is not learning from your conversation. When you type a prompt, the model is applying what it learnt during training, not updating itself in real time.
• Its knowledge has a cut-off. Anything that happened after the training data ended is not in there unless the tool has some other way of finding it.

That last point matters because many tools do have another way. Some connect the underlying model to web search, document retrieval, or other tools, which can bring in current or specific information. So "the model" and "the tool you are using" are not always the same thing. Two products built on similar models can behave quite differently depending on what is attached, and it is worth knowing which kind you are using.

The model's working memory

An LLM can only handle a limited amount of text at once. This limit is called the context window, and it includes your prompt, earlier parts of the conversation, any documents you have shared, and the response the model is currently writing. All of that competes for the same space.

A bigger window lets the tool deal with longer documents and conversations, but a useful rule of thumb is that more room does not guarantee everything in it is weighted equally. In a long, cluttered prompt, important details can end up having less influence than you would hope.

Strengths, limits, and why "hallucination" is not a rare glitch

LLMs are genuinely good at many things connected to language and pattern. They can explain an idea at different levels, restructure or summarise text, suggest ways into a problem, produce worked examples, point out possible ambiguities, and help with drafting and coding.

The limitations come from the very same mechanism:

• Fluency is not accuracy. The output is built from patterns, not checked against a store of verified facts, so it can be well-written, confident, and wrong.
• Invented references happen. Plausible-looking citations that do not exist are the best-known example, often called hallucination. (That word is a convenient technical label, not a claim that the tool has experiences like a person.)
• Subtle errors hide in long outputs. Long calculations, extended arguments, and code can contain a flaw that is easy to miss because everything around it reads smoothly.
• Wording matters. Minor changes to how you phrase a prompt can change the answer.
• Biases in the training data can show up in the output.

It is tempting to file these under "occasional glitches that will be fixed soon". A more accurate framing is that they follow from how the technology works, so the sensible response is not suspicion of everything, but good habits of checking.

For anyone who wants to go a level deeper

If this has whetted your appetite, there is a lot more beneath the surface, and I will only gesture at it here. Models represent tokens as numbers in ways that capture relationships between them. Most current systems use an architecture called a transformer, built around a mechanism known as attention, which lets the model weigh different parts of the surrounding text when producing each token. After initial training, models are usually refined further to behave more like helpful assistants. And techniques such as retrieval-augmented generation (RAG) feed relevant documents into the model's context so it can draw on them.

I am deliberately keeping that light. Terms like "learning" and "attention" are used in specific technical senses here, and they do not imply understanding or awareness in the human sense. If you want the fuller version, particularly if you work in a quantitative discipline, your department may have discipline-specific material, and it is worth asking your course team.

Using it well: thinking partner, not shortcut

If there is one practical takeaway, it is this: the most productive way to use an LLM tends to be as a thinking partner rather than a shortcut to a finished product. (This echoes something I wrote about in The Effort Paradox and GenAI: Why easy is not always better, the effort you are tempted to hand over is often the part that makes learning stick.)

In practice, that might mean asking it to compare two approaches, explain something in different terms, point out weaknesses in your argument, or generate a related example, rather than asking for the finished answer. Those kinds of exchanges also give you more chances to notice when something looks off.

And whenever you use one, a few questions are worth keeping handy:

1. Can I verify this from another source?
2. Does this reference exist, and does it actually say what the tool claims?
3. Is this reasoning genuinely sound, or just convincing?
4. Am I engaging with this material, or just accepting it?

The last one is probably the most important. These tools work best when you bring the questions, the judgement and the critical eye, and they bring the breadth.

Take it further: the resource

If you would like a short, structured walk through all of this, I have put together an interactive resource, How Large Language Models Work. It covers, in a few short pages:

• How an LLM generates a response in broad terms.
• What tokens and context windows are and why they matter in practice.
• The main strengths and limitations of LLMs, including the risk of hallucination.
• What effective, critical use looks like in an academic setting.

It is designed to work whatever your background, and it works on a PC, tablet, or phone. It also sits alongside other resources in the University's GenAI series, covering topics such as academic integrity, prompting, ethics, bias, critical thinking, information literacy, and data privacy and security. If you have not yet looked at it, Academic Integrity in the Context of GenAI is a natural companion, particularly as you work out where GenAI does and does not fit in your assessments.

Try it yourself: a 20-minute experiment

Reading about how these tools work is one thing. Seeing it happen is another, and it tends to stick far better.

Pick a topic you know well: a module you have finished, a hobby, or your own area of research. That is the key ingredient, because your own knowledge is what lets you spot where the tool is on solid ground and where it is not. Then try these:

1. Watch the variability. Ask a question about your topic, then open a fresh chat and ask exactly the same thing. Compare the two answers. What changed: wording, emphasis, the claims themselves? What appeared both times?
2. Test the references. Ask for three sources on a specific aspect of your topic, with authors and years. Then try to track each one down through the Library's search or other scholarly portals. Does it exist? If it does, does it actually say what the tool claims?
3. Probe the edges. Ask about something very recent, or very niche. Does the tool tell you it is unsure, or does it answer smoothly anyway? If your tool offers web search, compare what happens with it switched on and off. That is a quick way of seeing the difference between the model and the wider system around it.
4. If you want to go further: paste in a long document with one specific detail buried in the middle, then ask about that detail. Try it again with a much shorter version of the same document. This is a small, hands-on way of seeing what the context window means in practice.

Once you have tried a couple of these, notice three things:

1. How the tone of the tool stays equally confident, even when the quality of the content shifts.
2. Which of your checks the tool passed and which it did not, and whether you could have predicted that from how it works.
3. What you would now do differently the next time you use it for real study.

The point is not to catch the tool out. It is to build what you might call calibrated trust: a sense of when you can lean on it and when you need to double-check. Two practical notes: choose a topic that does not involve personal, confidential, or unpublished material, and remember this is an exploration exercise rather than assessed work. If you are thinking about using GenAI in an actual assessment, your assessment brief is the place to check what applies.

Final thoughts

This is a fast-moving area, and everything above is a simplified snapshot. Specific tools, features, and behaviours change often, and I would not want you to treat any single claim here as the final word. The underlying idea, that these systems predict plausible text rather than retrieve verified facts, has held up as a useful way of thinking, but how that plays out in any particular tool is worth checking for yourself.

If one habit is worth taking away, make it this: understanding how the tool works is what lets you stay the one in charge of the thinking.