The University of Tokyo and the Japan Science and Technology Agency (JST) announced on November 6, 2026 a new theory that analyzes the generation process of large language models (LLMs) and similar systems from the perspective of stochastic thermodynamics. The paper is a single-author work by Professor Takahiro Sagawa of the Department of Applied Physics, Graduate School of Engineering, the University of Tokyo. It presents a method for investigating how AI learns the order of events by comparing the probabilities with which a sequence, such as a passage of text, appears in the forward and reverse directions. Take, for example, the two sentences "The cup fell. The cup broke." The question is how the probability a language model assigns changes when the order of these sentences is swapped. The analysis targets irreversibility in the probabilistic structure of generated sequences, not heat from semiconductors or power consumption. Experiments using synthetic English text confirmed that probabilities differ depending on the order of events, but they did not prove that AI understands the causal laws of the real world.
What does the "reverse direction" of a generation process compare?
When an LLM generates text, it probabilistically selects the next output based on the words and other tokens it has already produced. This mechanism is called an "autoregressive generative model." The next output depends not only on the immediately preceding output but also on the earlier history. The new theory exploits a structural feature of autoregressive models: the history is condensed into an internal state, and output probabilities are computed from that state.
Professor Sagawa compares the "forward process," which generates a sequence in the normal order, with a "backward process," which handles the same components in reversed temporal order. He calculates the probability that a particular sequence appears in each process and defines "entropy production" from the logarithm of their ratio. This makes it possible to quantify, using the probabilities the model assigns to sequences, the asymmetry that arises when the direction of time is reversed.
The average entropy production equals the KL divergence between the path probability distributions of the forward and backward processes, and is always zero or greater. By contrast, the log probability ratio calculated for an individual sequence can be negative. In other words, the non-negativity that holds for the average does not hold for every text or sequence.
The thermodynamics referred to here concerns the probabilistic structure of generated sequences. JST's release also clearly distinguishes it from the physical heat produced by computers. The entropy production value therefore cannot be converted directly into power consumption or cooling load.
The theoretical contribution is not that it builds the first thermodynamics of history-dependent stochastic processes. Rather, it presents a framework that exploits the structure common to autoregressive generative models so that they can be analyzed in a unified way. The targets include not only Transformers but also recurrent neural networks (RNNs) and Kalman filters, and the framework is also applicable to state-space models and Mamba. For linear Gaussian models, the paper confirms that analytical expressions derived from the theory agree with Monte Carlo results, so this is not merely research that draws an analogy between event order in language and physics.
This framework also does not require enumerating every possible combination of past histories. Entropy production can be estimated from sequences generated as samples and the model's output probabilities. This avoids the problem of the number of history combinations growing exponentially, but it does not eliminate the inference computation of the LLM itself. The analysis also requires access to the probabilities the model assigns to outputs, so it cannot necessarily be applied as is to commercial APIs that return only generated text.
The paper was published on November 5 in the peer-reviewed journal PRX Intelligence. Its DOI is 10.1103/tv38-b23y, and the first version of the preprint was released on April 9 of the same year. The announcement is a joint release by the University of Tokyo and JST accompanying the publication of the research in a peer-reviewed journal.
Preserving grammar while reversing only sentence order
Simply rearranging the tokens that make up a text in reverse order destroys not only the sequence of events but also the grammar. As a result, even if the probability the model assigns drops sharply, it is hard to tell whether this reflects the model capturing event order or merely that the text became unnatural. In the LLM experiment, therefore, the word order within each sentence was left unchanged, and only the order of sentences in four-sentence passages was reversed. The two-sentence cup example at the start is a Japanese illustration to explain the mechanism; the actual experiment used four-sentence passages in English.
The texts for the experiment were created by five models: Claude Opus 4.6, Claude Fable 5, Gemini 3.1 Pro, GPT-5.4 Pro, and GPT-5.6 Pro. Each model was asked to generate 100 passages in which events proceed in a temporally and causally connected way and 100 passages that list mutually independent facts, giving each model a set of 200 passages. The paper labels the former "causal" and the latter "non-causal," but this classification rests on the instructions given when generating the texts and is not the result of formally verifying the causal relationships in each passage.
The evaluators that assigned probabilities to the resulting texts were two models, GPT-2 and Qwen3-4B-Base. GPT-2 is a pretrained base model with about 124 million parameters, and Qwen3-4B-Base is one with 4 billion parameters; neither is a chat model tuned for conversation. Both evaluators were given the same five sets of passages for comparison.
This experiment therefore does not grade the abilities of the five models that created the texts. Its purpose is to examine whether the difference in probability when sentence order is reversed appears consistently even when different evaluators are used. Reversal at the sentence level avoids the problem of broken grammar, but it cannot eliminate every factor that affects the connections between sentences.
What the five passage sets showed
The experiment compared the values obtained by dividing the stochastic entropy production, calculated from sentence-level reversal, by the number of tokens T in the passage. For the "causal" and "non-causal" groups, the researchers examined the difference in this value and evaluated its direction and size using the rank-biserial correlation coefficient (r). A positive r means that the metric tends to be higher in the "causal" passages, where events proceed in sequence. However, this is not a measure of the proportion of causal relationships understood correctly, nor a direct indicator of the model's causal reasoning ability.
The table below organizes the sentence-level reversal r values presented in Tables II and III of the original paper by the model that created the input text.
| Model that created the input text | r for GPT-2 | r for Qwen3-4B-Base |
|---|---|---|
| Claude Opus 4.6 | 0.4572 | 0.5692 |
| Claude Fable 5 | 0.2712 | 0.3772 |
| Gemini 3.1 Pro | 0.2624 | 0.4438 |
| GPT-5.4 Pro | 0.0934 | 0.3542 |
| GPT-5.6 Pro | 0.0936 | 0.1640 |
For each passage set, 100 "causal" and 100 "non-causal" passages were compared using a two-sided Mann–Whitney U test. The results show that r was positive for both GPT-2 and Qwen3-4B-Base in all five passage sets. In other words, in these synthetic texts, a difference in the same direction appeared between passages in which events proceed in order and passages that list independent facts, even when the evaluator was changed. In individual tests without correction for multiple comparisons, three sets were judged statistically significant for GPT-2 and five for Qwen3-4B-Base.
This significance calls for caution, however. The reported individual p-values were not corrected for multiple comparisons, and the largest p-value for Qwen3-4B-Base was 0.04525, close to the conventional significance level of 0.05. The result that all five sets were significant should therefore be interpreted as based on uncorrected individual tests. The paper also shows 95% bootstrap confidence intervals for each r estimate; point estimates of effect size alone are not enough to assess that uncertainty.
There are also constraints in how the samples were created. Each passage set was generated together from a single model response, so the passages within a group are not necessarily statistically independent. In addition, one fixed example passage per category is included in common across all five passage sets. The remaining 99 of each differ by creating model, but the five sets as a whole cannot be treated as completely independent samples.
The results were obtained within the same study by changing the evaluator and are not a replication by independent researchers. Furthermore, GPT-2 and Qwen3-4B-Base split text into tokens differently. The r values in the table express the difference between the "causal" and "non-causal" passage groups within each evaluator and are not a direct comparison of the absolute values of entropy production per token. Nor can the numbers or row order in the table be used to judge an overall ranking of the evaluators' or text-creating models' abilities.
Temporal order is not the same as causation
The fact that generation probability changes when the order of text is reversed is a separate matter from whether AI understands real-world cause and effect. The "causal/non-causal" classification in this experiment was determined by the instructions used when generating the texts. It is not an experiment in which the presence or absence of causal dependence was verified by formal criteria and only that condition was varied. The observed difference therefore cannot be interpreted directly as the result of learning causal relationships.
For example, one event happening before another, being more likely to be narrated in a particular order in text, and an earlier event being the cause of a later one are each different relationships. The probabilities a language model assigns to text reflect not only real-world causality but also temporal order and conventions of text composition. Even when sentences are reversed while preserving the word order inside them, the possibility remains that these factors are mixed together.
Professor Sagawa also explains in the paper that confirming differences through sentence-level reversal alone does not mean a metric for quantitatively evaluating general causal structure has been established. What was measured is only the difference in probabilities the model assigned to particular sets of synthetic passages. Further verification is needed to clarify how much of that difference stems from learning causal relationships.
There is also the question of whether the same relationship would be detected if the text were paraphrased without changing its meaning. If the metric remains stable when expressions and phrasing are changed, it becomes more likely that it captures information beyond the mere form of the text. To connect responses to order with representations of the world that AI has acquired internally, experiments that separate the influence of textual meaning from surface structure are essential.
Challenges remaining for analyzing "world models"
One direction the research aims for is quantitative analysis of the "world models" that AI acquires internally. A world model here refers to representations formed inside a model about the structure and changes of the real world. The new theory is positioned as a first step toward investigating such internal representations.
In theory, average entropy production can be decomposed into non-negative contributions corresponding to each time step. Each contribution can be further divided into two factors: "compression loss" and "model mismatch." Compression loss means that information needed to estimate the past is lost when the future-side history is condensed into the internal state. Model mismatch refers to the mismatch that arises when the output probabilities used for forward prediction are applied in the reverse direction. Neither refers to the general compression rate of training data or to errors in text generation.
However, no method for efficiently estimating these contributions has yet been established. Even though entropy production can in theory be decomposed into multiple factors, the LLM experiment did not separate and measure each contribution. Confirming that probabilistic differences exist between groups of synthetic passages and elucidating the internal mechanisms of the model that produce those differences need to be evaluated separately.
This research also builds a theory for analyzing existing generative models; it does not propose new training methods or demonstrate improvements in model performance. Nor has it been shown to deliver benefits such as lower power consumption or fewer hallucinations. What matters going forward is whether the metric derived from the theory can be developed into a method for diagnosing the properties of actual models.
The author has released the code and data used in the research, and the GitHub README describes token-level and sentence-level evaluation using GPT-2 as well as experiments on linear Gaussian models. The challenges ahead are whether the metric works on texts in which causation and temporal order are strictly controlled, whether similar trends are confirmed with independent samples and third-party replication, and whether the theoretical contributions can actually be estimated. If these are verified, the work could develop from observing the phenomenon that generation probability changes with the order of text into a method for quantitatively investigating what information AI retains in its internal state and what information it loses.