On October 29, 2026, Randall Balestriero and Yann LeCun posted a paper to arXiv that examines, through theory and experiments, the conditions under which the internal representations of separately trained image and language models come to have similar structures. A "good view"-style JEPA, which uses differently worded descriptions of the same scene as training pairs, reached a representation alignment score of 0.109 after training on 21.6 million samples. A next-token prediction model, which predicts how a sentence continues, scored 0.111 after training on 900 million samples. Comparing the published training endpoints, that is a gap of about 42 times in sample count.
The metric being compared here measures how similar the positional relationships among data points are in "representations," the numerical vectors that images and text are converted into. The key point of the study is that, rather than simply scaling up models, what you link together as observations of the same scene may greatly change how well image and language representations agree. However, the roughly 42x figure is a comparison of training sample counts. It does not mean that computation or training cost is 1/42. In addition, the theoretical guarantees rest on several conditions, and real models do not satisfy all of them.
Using differently worded descriptions of the same scene for training
The study by Randall Balestriero of Brown University and AMI Labs and Yann LeCun of New York University and AMI Labs takes as its starting point the Platonic Representation Hypothesis proposed in 2024. The hypothesis rests on the observation that as image models and language models are scaled up, the distance relationships among data in their learned representations grow increasingly similar. The idea is that a model learning from images and a model learning from text may both end up capturing a shared structure of the world. In this study, the authors derive sufficient conditions for such agreement to arise in an idealized learning model, then test whether representation alignment can also be improved in actual training. The paper has been released as a preprint, and no peer-reviewed venue has been confirmed.
The JEPA (Joint-Embedding Predictive Architecture) used in the study converts two corresponding observations into numerical vectors and pulls their representations closer together. It also includes a mechanism to prevent "representation collapse," in which every input is mapped to the same vector. The part that converts inputs into vectors is called an encoder, and the pairs of observations treated as corresponding during training are called "positive examples." For instance, if two people describe the same dog running across a lawn in different words, the representations of the sentences differ, but the content of the scene, such as the dog and its action, is shared. The authors define a pair of separate observations of the same scene as a "good view," and examples include different captions attached to the same image or different frames from the same video.
By contrast, if you mask part of a single sentence to create two inputs, they share not only the scene content but also that sentence's particular word order and phrasing. Likewise, when training pairs are made by cropping an image, the original image's shooting angle, lighting conditions, and so on remain. The authors call such pairs derived from a single observation "bad views," in light of the goal of aligning representations across different types of data. This does not mean that masking or cropping is an inferior training method; their usefulness depends on what you want the model to learn.
The authors further explain that training to predict the next token from the first half of a sentence can also be formulated as a kind of JEPA that uses the relationship between the first half of a document and its continuation. Because next-token prediction also learns information that does not correspond directly to images, such as sentence-specific word order, it may hinder agreement with image representations. However, the conditions of this formulation differ from those of the theorem guaranteeing representation agreement, so the authors also test the differences between training objectives experimentally.
The "42x" figure compares training sample counts
In the experiments, the authors used a metric called "mutual 10-NN" to measure how well image and language representations agree. For each data point, it finds the 10 nearest data points separately on the image side and the text side, then evaluates how much the neighbor relationships coincide. It can compare the internal structure of representations without needing to align the coordinate axes of the two representation spaces in advance. However, it is a different metric from image classification accuracy or text generation quality.
The first experiment used 29,000 training images from Flickr30k, each with five captions, for a total of 145,000 sentences. The image encoder and text encoder are trained separately, but the text-side JEPA is given groups of captions corresponding to the same image as positive examples. For the comparison next-token prediction model, the authors also set up a condition that gave it two sentences belonging to the same image, separating the effect of being exposed to multiple captions from the effect of a training objective that draws their representations closer.
For the larger-scale experiment, they used Recap-DataComp, which contains up to 940 million images. The training material consisted of the alt text for each image and two captions generated by the vision-language model LLaVA. In Section 4.2 of the paper, the authors compare endpoints reached when training the same 109-million-parameter BERT-style model on text derived from the same corpus, as follows.
| Item | Good-view JEPA | Next-token prediction |
|---|---|---|
| Parameters | 109 million | 109 million |
| Training samples | 21.6 million | 900 million |
| WIT alignment score | 0.109 | 0.111 |
| Text processed | 3 sentences per sample | Total tokens about 14x the JEPA's |
| Learning rate | 0.0001 | 0.0002 |
| Batch size | 256 | 2048 |
Dividing 900 million by 21.6 million gives about 41.7, so there is a gap of about 42x in training sample count at the points where similar alignment scores were recorded. However, because the JEPA processes three sentences per sample, the gap shrinks to about 14x when compared by the number of text tokens actually read. This token ratio is a figure published in the paper, but absolute counts are not given. Therefore, the claim that "it matched performance with 42 times less data" needs to make clear what is being counted as data.
More importantly, the paper does not evaluate checkpoints of the next-token prediction model earlier than the 900-million-sample mark. It cannot be ruled out that the model reached a similar alignment score before reaching 900 million samples, so these results alone do not support concluding that the minimum number of samples needed for equivalent alignment differs by 42x. The two models also differ in learning rate and batch size. Comparing GPU runtime, power consumption, training cost, and so on would require separately measuring each model's run conditions and processing speed. What has been confirmed is that applying different training methods to models of the same size produced a large difference in the sample counts at the published endpoints.
The theorem guaranteeing representation agreement has conditions
The "same representation up to rotation" that this theory addresses refers to a state in which the positional relationships among data correspond even if the coordinate axes of the vector spaces point in different directions. It is like rotating maps made from images and from text so that the arrangements of points overlap. The numerical outputs of the image encoder and the text encoder therefore do not need to match from the start.
However, the theorem rests on several premises. First, it assumes that the number of world states is finite and that the corresponding world state can be distinguished from each observation. Furthermore, for the positive examples learned on the image side and the text side, the distribution of occurrence of the underlying world states (marginal distribution p) and the distribution over pairs of world states linked as positive examples must match. In addition, the model must reach the optimal solution over the entire distribution, and the solution must be uniquely determined apart from differences due to rotation.
These conditions differ from real models, which use finite training data and finish training within limited computation time. Images contain objects that are not shown, and text contains situations that are not stated explicitly, so it is not always possible to fully identify the state of the world from observations alone. The authors themselves acknowledge that the actual experiments do not strictly satisfy the theorem's assumptions. This theorem therefore shows that representations agree if certain conditions are met; it is not a proof that all real image and language models converge to the same representation.
The range of what could be proved about uniqueness also varies. In the "whitened second-order statistics" setting, which imposes conditions on the variance and correlation of representations, the authors used spectral analysis based on matrix eigenvalues to prove uniqueness when the eigenvalues are sufficiently separated. In contrast, the LeJEPA used in the experiments imposes constraints on the whole distribution of representations through SIGReg, and uniqueness of the solution in this case has not been proved. The result that representation drift can be bounded within a certain range when the distribution of positive examples shifts between different observations likewise applies to solutions obtained through spectral analysis.
In addition, the predictors treated in the theory are limited to identity maps that output their input unchanged. World models with learned predictors, such as those that predict a future state conditioned on actions, are not covered. This theorem clarifies the conditions under which a shared structure of the world can be learned from different observations, but applying it to more general models will require further examination of conditions on observation methods and optimization.
Comparison with giant LLMs also involves differences in training data
Table 3 of the paper compares agreement with several language models, using representations from WIT-1024 and the image model DINOv2 ViT-B as the reference. The good-view JEPA's mutual 10-NN was 0.184, exceeding the 0.111 of the next-token prediction model of the same size. It also exceeded Qwen2.5-72B at 0.154 and Llama-3-70B at 0.151. These are results measuring how well neighbor relationships agree between the image side and the text side, and they do not mean that a small JEPA outperforms giant LLMs at language reasoning or coding.
The value of 0.184 is a result at the point after training on 900 million samples, so the amount of training differs from that behind the 0.109 obtained at 21.6 million samples mentioned earlier. The main figures in Table 3 are results from three random seeds that used the same effective weight for the SIGReg term, which prevents representation collapse. In contrast, a fourth seed with the SIGReg effective weight doubled dropped to 0.121, making the average across all four seeds 0.168. It is true that the main setting recorded a high value, but it should also be noted that results change depending on the training settings.
Another important condition is the origin of the text used for training. The large-scale experiment used alt text attached to web images as well as captions generated by LLaVA after viewing the images. LLaVA's image encoder uses CLIP ViT-L/14. In other words, even though the text encoder itself does not process pixels directly, it learns content derived from visual information through the AI-generated captions included in the training pairs. In fact, under the condition where paraphrases made from text alone were added to the alt text, the WIT alignment score was 0.060, whereas using captions regenerated from the images raised it to 0.107. Not only the training objective but also how the captions are produced affects the degree of representation agreement.
This difference changes what the comparison means depending on the comparison target. Experiments comparing JEPA and next-token prediction using the same generated text can examine the effect of the training objective on common text data. In comparisons with existing giant LLMs, however, both the training data content and the way image-derived information was incorporated differ. Additional controlled experiments would therefore be needed to attribute differences in alignment scores solely to a particular training objective.
The authors have also published the results of checks on evaluation data overlap. Of the 1,024 captions in WIT, at least 43 were contained in the alt text of the training data. When the nine models were evaluated with these removed, the alignment scores changed by at most 0.002, and the authors report that the effect is small. On the other hand, hyperparameters were tuned on the good-view JEPA side, and the same tuning was not applied to the control models. Unifying training conditions and re-verifying the results with independent evaluation data will be important for improving the reliability of the comparison.
Application to image captioning and remaining verification issues
What this work suggests is that, beyond model scale, which observations are linked during training is important for aligning representations across different types of data. However, theoretical guarantees and experimental results cannot be treated as the same thing. The conditions under which the theorem holds, the comparison of training objectives using the same text, the comparison with existing giant LLMs, and the application experiments that generate text from images each rest on different premises.
| What was tested | Information used for training/tuning | What was confirmed and conditions that remain |
|---|---|---|
| Theoretical theorem | Idealized shared positive-example and marginal distributions | Representations agree up to rotation if the conditions are met. Uniqueness and reaching the optimal solution are premises |
| Comparison of training objectives using the same text | Caption groups for the same image and AI-generated text | Differences in alignment score and sample count at published endpoints. Differences in training settings must be taken into account |
| Comparison with existing giant LLMs | Different training data and image-derived captions | Advantage recorded on a particular alignment metric. Additional verification under common conditions is needed |
| Image caption generation | About 30,000 pairs used to map image and text representations | Text generation succeeded under conditions. Tuning the mapping and selecting a candidate sentence are required |
As a practical application, the authors built a system that generates captions from images. It converts the representations from frozen DINOv2 or DINOv3 image encoders into the text-side representation space through a linear map, and combines them with language models such as Qwen2.5-1.5B that were fine-tuned on text alone. In the configuration that recorded the best CIDEr of 0.838 on COCO, the mapping was tuned using about 30,000 image-text pairs from Flickr, and the final caption was selected using the degree of agreement among 32 candidate sentences. CIDEr is a metric for evaluating the quality of generated captions, and 0.838 cannot be interpreted as 83.8% accuracy.
Also, when image and language representations have different dimensionalities, the linear map linking them falls outside the direct scope of the theorem's guarantee, which concerns rotating representation spaces of the same dimension. While a system can be built without jointly training encoders on images and text, obtaining the best captioning performance required tuning the mapping with corresponding image-text pairs.
The authors also ran an experiment that aligns representation spaces without using any corresponding image-text pairs. They split COCO train2014 images and text into two mutually non-corresponding groups to estimate the mapping, and recorded top-5 accuracy of 64.4±3.6% in ImageNet-100 zero-shot classification. Under the same conditions, Qwen3-8B re-run scored 47.6%, and CLIP's text encoder scored 68.7%. These results suggest that different representation spaces may be linked without using corresponding pairs, but they come from conditions different from the experiment that achieved the best captioning performance. They are also a separate evaluation from mutual 10-NN, which measures neighbor relationships between images and language.
The authors have published the experiment code, run arguments, reproduction steps, and a demo on GitHub, but no replication involving GPU retraining by an independent research team has been confirmed. Going forward, it will be necessary to clarify the effect that image-derived captions have on the results and to run comparisons with matched training settings and evaluation conditions. It will also be necessary to verify efficiency including the cost of collecting or generating multiple observations of the same scene. If the advantage is confirmed under such conditions, it could become a new design option for handling images and language efficiently by being smart about how training data is linked, rather than relying only on larger models.