On October 8, 2026, Yale University highlighted research showing that large language models (LLMs) contain internal structures that link words and numbers to the "roles" they play. The team, led by R. Thomas McCoy and colleagues, approximated the numerical representations inside LLMs using these structures. They then confirmed that rewriting some of those representations could push a model's output in an intended direction. The findings are a preprint posted to arXiv on August 30, and the study does not directly compare the human brain with AI. Even so, it offers an important clue to what information AI holds internally when it understands text or performs calculations, and how it uses that information.
How Do LLMs Tell "Subject" From "Object" Internally?
"The cat chases the dog" and "The dog chases the cat" use the same words, but their meanings are opposite. If the numerical representations for "cat," "dog," and "chase" were simply added together, the two sentences could not be told apart.
To capture meaning correctly, a model must distinguish which of the two animals is the subject and which is the object. The same is true in arithmetic: whether 3 goes in the numerator or the denominator changes the result, even with the same combination of numbers.
The team used Tensor Product Representation (TPR), a method for expressing this relationship between "fillers" and "roles" numerically.
In this method, elements such as words and numbers are each represented as vectors (ordered lists of numbers), and roles such as subject and object are assigned separate vectors. The element and role vectors are bound together through a tensor product, and the resulting combinations are summed. In this study, the researchers also added a coordinate transformation so that the result could be compared with an LLM's internal representations.
With this mechanism, even the same word "cat" can be treated as a different numerical representation depending on whether it appears as a subject or an object. Even when individual components of a vector do not reveal whether something is a subject or an object, the vector as a whole may contain a regular structure that encodes such relationships.
The analysis method the team developed to find this structure is called "DISCOVER."
In DISCOVER, researchers first set candidate roles, such as subject, verb, and object, according to the task being analyzed. The method then trains vectors for elements and roles so that they reproduce the LLM's internal representations as faithfully as possible.
The resulting approximation is then used to check whether the model can still produce correct outputs. The distinctive feature is that it verifies not only whether the internal numbers look similar, but also whether the approximation can reproduce the same function as the original internal representation.
Seven LLMs Analyzed, With Effects on Output Tested in GPT-OSS
The team first analyzed seven LLMs whose weights are publicly available, including GPT-2-XL and GPT-OSS-20b, as well as Gemma, Llama, and Qwen models.
The first experiment focused on the internal representation corresponding to the period at the end of an English sentence or word list. The final token may aggregate information about everything the model has read so far.
The team approximated this internal representation with TPR and examined whether the original text could be reconstructed from it. The reconstruction used not the LLM itself but a separately trained readout model.
In the next experiment, they narrowed the target to GPT-OSS and replaced the internal vectors corresponding to the input with TPR-based approximations. With that substitution in place, they had GPT-OSS itself generate answers and checked whether it could solve the tasks as well as it did with the original internal representations.
The experiment across seven models tested only whether information about the text could be read out of the internal representations; GPT-OSS alone was used to test whether manipulating the representations affects output.
Organized by target and what each tests, the experiments in Section 5 and Sections 6–7 of the paper look like this:
| Experiment | Target and method | What it can verify |
|---|---|---|
| Analyzing end-of-sentence representations | Approximate the representation of the period in seven models with TPR, then reconstruct the original sentence or word list with a separate model | That the internal representations contain a structure from which word order and relationships can be read out |
| Replacing internal representations | Replace the representations of each token and layer for GPT-OSS's task inputs with approximations | That, on six limited tasks, answers can be produced from the approximations with accuracy close to that of the original representations |
| Editing part of the internal representations | Change the bindings between elements and roles inside GPT-OSS | That operations changing specific relationships affect the model's output |
This distinction matters.
Being able to reconstruct text from an internal representation is evidence that the model holds that information. If the model can still answer in the same way after its representation is replaced, that is stronger evidence that the representation can actually be used in information processing.
However, the researchers did not verify behavior after altering internal representations in all seven models.
The GPT-OSS experiments also targeted tasks with limited conditions, not free-form conversation.
In arithmetic, they handled multiplication and addition of small integers; in logic, syllogisms that derive a conclusion from premises. In programming, the model was asked to predict the results of Python list operations. The language tasks were three types: converting an English sentence to the passive voice, changing its tense, and turning it into a question, for a total of six tasks.
The parts replaced with approximations were the vectors corresponding to the portion of the input that changes from task to task; the shared instructions and example portions were left as they were. The process of generating the answer is also still handled by the original GPT-OSS.
In other words, this study does not reproduce the workings of an entire LLM with a single formula. It examines how far specific information inside a model can be reproduced and manipulated using structured numerical representations.
Editing Internal Representations Yields 90.3% Average Accuracy, but That Is Not the Same as Reasoning Accuracy
The team also tested whether rewriting internal representations could steer GPT-OSS's answers in an intended direction.
Consider the sentence "The smart doctor helped the lawyer." In the original, the modifier "smart" is bound to "doctor." The team removed the component linking the doctor to "smart" from the internal representation and added a component linking it to the lawyer instead.
The model then answered as if it had been given a sentence meaning "The doctor helped the smart lawyer."
The key point is that the input sentence itself was not changed. The numbers representing relationships between words were manipulated inside the model while the original text stayed the same.
Moreover, this operation is not as simple as flipping a single neuron. Following the structure obtained from the analysis, it adjusts vectors spread across multiple tokens and layers.
In fact, when only the positions where the target word appears were edited, swapping words worked relatively well, but performance dropped sharply for operations that changed roles within the sentence. This result suggests that information about the roles words play may be distributed across multiple positions.
Across 31 types of intervention experiments on GPT-OSS, an average accuracy of 90.3% was reported. This came from preparing 100 examples for each condition, training the analysis model five times, and evaluating the results.
However, this 90.3% is the average proportion of cases, on predefined tasks, in which the model produced the expected correct answer after its internal representations were edited. It does not mean GPT-OSS's reasoning ability as a whole reached 90.3% accuracy.
The supplementary materials contain a result that makes this difference clearer.
In an experiment swapping the roles of nouns in a syllogism, accuracy after the intervention was 42.2%, while the answer-match rate with directly inputting the modified sentence reached 90.2%.
These two figures measure different things.
The 42.2% is the share of cases in which the model gave a logically correct answer after its internal representation was changed. The 90.2% is the share in which it gave the same answer as when the sentence the internal edit was meant to reproduce was supplied as ordinary input.
In fact, even when the modified sentence was entered normally, GPT-OSS's accuracy was only 52.0%.
In other words, editing internal representations may reproduce the model's behavior, including the errors it already makes.
These numbers compare results for the same intervention condition, "r-noun_swap," listed in Tables 4 and 5 of Supplement S2, converted to percentages. Both are averages over five evaluations using 100 examples, and the variability for both 42.2% and 90.2% is listed as ±2.6 points. This value corresponds to two standard deviations and is not a confidence interval.
Being able to manipulate internal representations to steer answers in an intended direction is strong evidence for understanding how LLMs work.
But that alone does not make answers more accurate.
To correct a model's wrong answers, it is necessary to clarify why the model went wrong in the first place, and how the handling of information or the computational process behind the error should be fixed.
Being able to control a model's behavior and getting it to generate accurate, reliable answers are separate problems.
Finding Symbolic Structure Does Not Prove General Reasoning Ability in LLMs
This is not the first study by McCoy and colleagues to use TPR.
In a paper accepted at ICLR in 2019, the team approximated the internal representations of recurrent neural networks (RNNs) with TPR.
However, in the four natural-language models analyzed then, representations that ignored word order could approximate internal states fairly well, so the benefit of explicitly incorporating sentence structure was limited.
This result also shows that not every neural network necessarily has the same kind of complex symbolic structure.
In a 2020 BlackboxNLP paper, the team also analyzed an RNN that converts natural-language instructions into sequences of actions, and showed that editing its internal representations changed the output as predicted.
The key advance of the current study is extending this kind of analysis to multiple models with different architectures and to modern LLMs.
The team also found that, under many conditions, internal representations could be approximated even for element-role combinations that were not used to train DISCOVER.
For example, they excluded examples in which a particular word is the subject from the analysis model's training and checked whether appropriate internal representations could still be built for that combination.
This suggests the analysis model may be capturing regularities that bind elements to roles, rather than simply memorizing combinations it saw during training.
Note, however, that what was excluded from training was DISCOVER's analysis model, not the corresponding examples from the original LLM's training data.
Therefore, this result does not prove that the LLM itself can solve unfamiliar problems. In fact, on some tasks such as arithmetic, cases were reported in which generalization to combinations not included in training did not work well.
In addition, the candidate roles DISCOVER searches over were set in advance by the researchers in this study.
That means that even if an LLM's internal representations can be approximated along the structure the team assumed, that is not necessarily the only way to explain how the model works internally.
Much also remains unknown about how LLMs acquire such structures during training and what computations they use them for in actual reasoning.
The study also did not directly examine the human brain, nor did it show that AI and humans reason through the same mechanism. Determining whether similar structures exist in the human brain would require separate experiments on brain activity.
The challenge ahead is whether the analysis method can be applied not only to tightly controlled experiments like these but also to free-form text and complex problems.
It will also matter whether, beyond identifying the internal representations associated with wrong answers, correcting that information can improve accuracy on problems the model has never encountered.
If such validation advances, techniques for analyzing and editing the internal structure of LLMs could become a means not just of explaining AI behavior but of tracing the causes of errors and building more reliable AI.