The question behind the fluent answer
Large language models can answer questions, generate code, write poetry and hold seemingly intelligent conversations. The question that matters to me is what we infer from that performance: do they understand what they are saying?
My short answer in the original article was no. The longer answer was also no, but with an important observation: they are remarkably good at producing the appearance of understanding.
This is an argument about that distinction, using plain language and analogies. “Thinking,” “understanding” and “consciousness” are not interchangeable terms. The technical notes below explain why the mechanics of token prediction alone do not settle all three questions.
What is an LLM actually doing?
At the core of an autoregressive language model is a prediction task: given a sequence of tokens, estimate a probability distribution for the next token. A token can be a word, part of a word or another text fragment, depending on the tokenizer.
The model applies a neural network to the context. Attention layers, feedforward blocks and other operations transform numerical representations into scores, which become probabilities. A decoding procedure selects a token, adds it to the context and repeats.
Calling it autocomplete captures that interface. It does not describe the full complexity of the computation required to make those predictions useful across so many different tasks.
The training objective is not human understanding
A human might write to communicate an experience, pursue a goal or explain an idea. During next-token pretraining, the numerical objective is usually to reduce prediction loss.
A compact way to write that objective is:
L(θ) = −(1/T) Σₜ log pθ(xₜ | x₁, …, xₜ₋₁)Here, θ is the set of model parameters, xₜ is the observed token at position t, and T is the number of predicted tokens. A higher probability assigned to the observed token means a lower loss. This is a mathematical criterion, not a direct measurement of truth, experience or subjective understanding.
There is also an important distinction between training and use. Training changes parameters to improve an objective. In ordinary inference, a fixed model generates a response; it is not taking a gradient step on a new training loss for every answer.
Instruction tuning and preference-based training add further objectives. In RLHF, for example, human comparisons help train a reward model, and optimization encourages outputs that receive higher reward. That can improve useful behaviour, but receiving approval is not itself evidence of an inner experience. Ouyang et al.’s Training language models to follow instructions with human feedback describes one such procedure.
Why does it feel so intelligent?
Human language contains patterns of explanation, argument, association and problem solving. A model trained on a large amount of it can learn to produce those patterns with striking fluency.
Ask about classical and quantum mechanics and the response may look like a teacher explaining a familiar subject. We naturally associate coherent explanations with an intentional speaker who knows what they mean. My concern is the jump from observing that output to assuming that the system has a human-like mental life behind it.
The capability is real. The interpretation is a separate question.
A dog in human experience and in a model
A person’s concept of a dog can connect sight, sound, touch, memory, emotion and action. We remember particular dogs, predict behaviour and know what it feels like to approach one cautiously or affectionately.
A text-only model encounters representations derived from training data. “Dog” appears near descriptions of barking, tails, walks and pets. Those relationships influence the model’s representations and predictions.
That is different from a human history of meeting dogs. But a learned representation is also more than a literal bag of character fragments. Tokens have numerical identifiers and learned embeddings; the network transforms them into context-dependent representations. Multimodal models may additionally process images or other inputs. The analogy here concerns the difference from human experience, not a claim that every model is text-only or incapable of learning structure.
Similar output, different route
Imagine two systems producing the same warning: “That’s dangerous — don’t do it.” A human may connect the situation with pain, memory, social responsibility and a goal. A language model produces the warning through its learned computation and the current context.
The similarity of the sentence does not establish that the two systems arrived at it through the same kind of experience.
High-dimensional vector spaces help describe the model’s computation. Their coordinates are not generally a neat catalogue of concepts such as “woman,” “lighthouse” or “danger.” Features can be distributed across directions and layers. Human brains are not neatly labelled coordinate systems either; the comparison is an analogy, not a literal map of either architecture.
For an accessible visual introduction, see 3Blue1Brown’s explanation of transformers, starting at the discussion of embeddings.
What impressive results do and do not establish
Models can write substantial essays, solve programming problems, translate text and perform mathematical tasks. Those are useful abilities. They need to be evaluated on accuracy, robustness and the conditions in which they fail.
My point is that success on a task should not automatically be interpreted as conceptual awareness or consciousness. Equally, the fact that a system was trained by prediction does not prove that it has learned only superficial associations.
The practical questions are concrete: does the answer hold up, does the method generalize, and can we check the result independently?
What might be missing from human-like thinking?
Biological intelligence develops in a body. Electrical activity, chemical signalling, metabolism, memory and interaction with an environment all contribute to the system we are comparing with a language model.
I find the relationship between physical processes and subjective experience particularly interesting. Could embodiment be necessary for some forms of understanding? Does the physical substrate matter beyond the computation it supports?
These are questions and philosophical hypotheses. The suggestion that consciousness might require particular biological processes or quantum phenomena is speculative; this article does not establish that requirement. Nor does it show that a digital system could never be conscious.
The distinction I want to keep
LLMs are sophisticated and useful tools. Coherent conversation can still tempt us to attribute more to a system than the evidence supports.
I want to appreciate their capabilities while being careful about what I infer from them. A fluent answer is an observable result. A claim about subjective experience needs a different kind of argument and evidence.
Technical context and research
The original essay took a deliberately strong position. Several qualifications matter when using it as an explanation:
- Prediction does not rule out internal models. Li et al.’s Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task found an internal representation of Othello board state in a model trained on move sequences. This is evidence from a specific synthetic task, not a demonstration of consciousness or of a complete human-like world model.
- LLMs need not be text-only. The GPT-4 Technical Report describes a model accepting text and images. “No visual input” cannot be a general definition of an LLM-based system.
- Preference optimization is an additional training stage. Ouyang et al. (2022) distinguish supervised demonstrations, preference comparisons and reinforcement learning. These objectives should not be reduced to next-token pretraining alone.
- Consciousness is a separate research question. Butlin et al.’s Consciousness in Artificial Intelligence: Insights from the Science of Consciousness proposes indicators grounded in several theories. It does not provide a simple proof that fluent language implies consciousness, or that computation excludes it.
For the architecture and scaling background, read Vaswani et al., Attention Is All You Need (2017) and Brown et al., Language Models are Few-Shot Learners (2020).
A second visual explanation
My original article also recommended New Research Reveals How AI “Thinks” (It Doesn’t). Treat videos as explanations to examine alongside the research, rather than as proof of a philosophical conclusion.
And yes: the emoji section headings in the LinkedIn version were suggested by GPT. Even statistical machines can format nicely. :)