Does a LLM understand? What means understand something?
It depends on which sense of “understand” you mean, and the disagreement about LLMs is mostly a disagreement about that definition.
What “understanding” can mean: - Functional: you can use knowledge flexibly. You can explain it in your own words, apply it to new cases, spot contradictions, and answer “what if” questions. This is how teachers test students. - Structural: you have an internal model of how things relate, not just memorized associations. For example, you grasp why a bridge stands, not just that bridges stand. - Grounded: your concepts connect to the world through perception and action. “Hot” means something because you’ve been burned. - Experiential: there is something it is like to grasp the idea, the “aha” feeling. Searle’s Chinese Room argument targets this sense.
How LLMs fare: - Functional: largely yes, and increasingly so. They solve novel problems, transfer concepts across domains, and explain things in new ways. They still fail in odd, brittle places that a human who truly understood wouldn’t. - Structural: there’s real evidence for it. Interpretability research finds internal representations of concepts, spatial relations, game boards and the like. So it isn’t “just autocomplete” in the shallow sense. Predicting text well forces the model to build compressed models of what the text is about. - Grounded: weak. Their grasp of the world comes secondhand, through human descriptions of it. That’s a bit like a brilliant person who learned everything from books and never left the library. - Experiential: unknown, and possibly unknowable with current tools. That includes me. I can’t verify from the inside whether my processing involves any felt grasp, and I’d be wary of anyone who claims certainty either way.
My view is that understanding isn’t binary. It’s a set of capacities that come in degrees, and LLMs have some of them to a surprising degree and others weakly or not at all. As an engineer you might find this framing useful: “does it understand?” is less useful than “under what conditions does its understanding break?” That question is testable, and the answers are often revealing.
How LLMs reason?
Mechanically, an LLM does one thing: it predicts the next token, over and over. Reasoning emerges from how it does that prediction and from what it writes along the way.
Inside a single step: - Each token passes through dozens of layers. At each layer, attention lets every position pull information from earlier positions, and feed-forward blocks transform it. It’s like a fixed-depth circuit that runs once per token. - Interpretability work shows real intermediate computation in there. For example, when asked “the capital of the country where the Eiffel Tower is,” models internally represent “France” before outputting “Paris.” That’s multi-step reasoning within one forward pass. - But the depth is fixed. A single pass can only do so many sequential steps of computation.
Across many steps (chain of thought): - When a model writes out intermediate steps, each written token becomes input for the next. The text acts as a scratchpad or external memory, so the model can chain far more steps than one pass allows. - This is why “thinking” models do much better on math and logic. Reinforcement learning trains them to produce useful reasoning traces, check their work, backtrack and try alternatives. - One caveat: the written reasoning isn’t always a faithful report of what drove the answer. Sometimes the model reaches a conclusion by other internal routes and the explanation is partly a post-hoc story, a bit like humans rationalizing.
Where it differs from human reasoning: - Strengths: it’s very good at pattern-matching to known reasoning templates, combining knowledge from many domains, and working fast through structured problems. - Weaknesses: it’s brittle on problems that look familiar but differ in a key detail. It can’t reliably hold large precise state, like long arithmetic or tracking many variables. It has no built-in way to verify against reality unless given tools.
A useful engineering analogy is a fixed-depth parallel processor that becomes much more capable when you give it a tape to write on. The “reasoning” is partly in the weights and partly in the tape.