Can causal relationships that no one has yet articulated be extracted from the sea of tens of millions of scientific papers? The attempt to connect fragments of accumulated knowledge to derive new hypotheses is called Literature-Based Discovery (LBD), and since Don R. Swanson's pioneering research in the 1980s, it has remained a major goal of computational science. However, engineering the extraction of biologically plausible hypotheses from an enormous pool of candidates—while going beyond the existing deductive bounds of knowledge—has proven extremely difficult.
A team of 24 researchers led by Tarek R. Besold (including researchers affiliated with Sony AI's Scientific Discovery team) submitted a paper on a new hypothesis prediction and explanation system called "Hakken" to the preprint server arXiv on September 3, 2026 (arXiv:2609.04494, cs.LG and cs.AI, 66 pages total, DOI pending registration). This research has not passed peer review at an academic journal and, at this stage, remains a self-reported result by the authors.
The authors' proposed concept of "knowledge prediction" refers to a framework that goes beyond the deductive hull of past knowledge to predict new relationships not currently stated explicitly in the literature, thereby expanding scientific understanding. The paper sets biomedicine as its domain of demonstration, using PubMed Central (PMC) open-access full-text articles, MEDLINE titles and abstracts, and a commercially licensed academic dataset for training. Some of the predictions generated by this system were actually advanced to wet-lab experimental validation using cells.
A Dual Mechanism Fusing Temporal Graph Structure and Linguistic Context for Prediction and Explanation
At the core of Hakken are two technical foundations: the relation prediction model "THiGERLLM," and "PHELInE," an explanation framework that helps scientists verify and evaluate the basis for its predictions.
- THiGERLLM (Proposed Method)
- THiGER (Prior Model)
データを表で見る
| THiGERLLM (Proposed Method) (%) | THiGER (Prior Model) (%) | |
|---|---|---|
| Macro F1 | 51.16 | 50.98 |
| Weighted F1 | 73.73 | 77.97 |
| Macro Recall | 60.72 | 46.13 |
| Macro Precision | 53.77 | 60.2 |
| nDCG mean | 90.3 | 92.19 |
THiGERLLM (Temporal Hierarchical Graph-based Encoder Representation with LLM) is structured to solve a multi-label relation prediction task for entity pairs (a subject and an object—for example, a specific drug and a target disease) extracted from a knowledge graph built from scientific literature. The model was designed with two cooperating branches. One is a "graph branch," which combines GraphSAGE-style neighborhood aggregation with a temporal encoder that captures the persistence and temporal change of concepts. The other is a "language branch," which interprets the semantic relationship between entity pairs from literature text.
The textual evidence fed into the language branch consists of sentences extracted from PMC open-access papers. A notable design choice is that instead of aggregating passages where the two entities co-occur in the same sentence, the system aggregates sentences in which the entities are not mentioned together (sentences where each entity appears independently). This allows the model to derive potential connections from separate contexts, rather than simply tracing relationships already stated together in the literature. The resulting graph embedding representations are fed into a large language model (LLM) as prefix tokens, together with literature-derived text. In addition, the model's confidence scores are statistically calibrated; according to the authors, they are adjusted so that among predicted events with probability $p$, roughly a fraction $p$ turn out to be correct in empirical frequency.
PHELInE (Predicted Hypothesis Elucidation with Literature-Inferred Explanations), on the other hand, is a model-agnostic explanation framework. It searches for relational paths connecting the predicted subject and object (in the experiments described below, paths of 2 to 4 hops), and tracks how the prediction score changes depending on whether such a path is present or removed. PHELInE does not directly access the model's internal parameters; instead, it estimates the "sufficiency" and "necessity" of explanatory paths through pseudo-retraining of a surrogate model. The resulting set of paths is ranked by degree of influence, then re-ranked to ensure diversity before being presented. This allows scientists to see not just a black-box numerical score, but which literature-derived pathways support a given prediction.
The authors hypothesize that THiGERLLM's performance improvement over the prior model, THiGER, stems from the addition of textual information and the semantic context provided by the language model.
The Wall of Evaluation: A Temporal Split Benchmark Centered on 2020 and Incomplete Negative Examples
To assess the validity of the prediction model, the authors adopted a strict temporal split protocol. To prevent information leakage in machine learning, relationship triples first observed before 2020 were used as training data, while relationships first reported in papers after 2020 were held out as test data. This design tests whether "future discoveries" can be predicted using only past data.
Comparison baselines included random prediction, the knowledge graph embedding method ComplEx, k-nearest neighbors (KNN), a multi-label MLP, a rule-based classifier, the temporal embedding method tNodeEmbed, and the authors' own prior model, THiGER.
In the evaluation results at the 2020 cutoff, THiGERLLM recorded a macro F1 score of 51.16%, slightly surpassing THiGER's 50.98%. However, it did not outperform the prior model on every metric. For weighted F1, which weights the average by frequency of occurrence in the data, THiGERLLM scored 73.73% compared to THiGER's 77.97%, meaning the prior model performed better here. Similarly, for nDCGmean, which measures ranking quality, THiGERLLM scored 90.30% versus THiGER's 92.19%.
Looking more closely at the metrics reveals differences in the models' behavior. For macro recall, THiGERLLM achieved 60.72%, a 14.59-point increase over THiGER's 46.13%. Meanwhile, macro precision came in at 53.77%, down from THiGER's 60.20%. The authors interpret this result as reflecting an improved ability to broadly capture a wider variety of relationship types, including rare relations that appear infrequently.
An important caveat in this evaluation, which the authors themselves note, must not be overlooked. The precision calculation in this benchmark is based on the assumption that "any prediction not currently observed in the literature is entirely wrong (a true negative)." In reality, however, it is highly likely that many biologically real relationships are buried within the unobserved data, simply because they have not yet been experimentally tested or reported. Therefore, the reported absolute values for precision and recall should be understood as representing a lower bound on the model's true performance.
The authors also present a retrospective consistency evaluation showing that the model's outputs remain consistent and informative over long historical periods. However, it should be noted that these are evaluations conducted within a benchmark environment designed by the authors themselves, and do not constitute an objective standard verified through independent third-party replication.
Three Hypotheses Selected from 1.5 Million Candidates, and Mixed Results in Wet-Lab Validation
A superior computational score is not equivalent to the intended molecular behavior actually occurring in a living system. To test Hakken's practical utility, the research team collaborated with biomedical experts to run a hypothesis validation cycle targeting aging-related genes.
For a subset of aging-related genes selected by experts, Hakken generated a total of 1,543,297 hypotheses exceeding a confidence threshold. From this pool of over 1.5 million candidates, a hypothesis selection pipeline extracted those possessing mechanistic plausibility. Ultimately, biomedical experts used subjective judgment—taking into account academic novelty and experimental feasibility—to select three candidates to advance to laboratory validation.
| Hypothesis Under Validation (Entity Pair) | Relationship Predicted by the Model | Prediction Confidence | Experimental Measurement Result | Validation Outcome |
|---|---|---|---|---|
| TP53 and BAMBI | TP53 influences the expression of BAMBI | 0.85391 | Following induction of P53 protein, BAMBI expression levels rose continuously, coinciding with the timing of peak P53 expression | Supported |
| RAF1 and TNF | RAF1 decreases the expression of TNF | 0.83114 | After administration of a RAF1 inhibitor, increased ERK phosphorylation (dimerization and transactivation) was accompanied by a consistent increase in TNF mRNA | Supported |
| SOAT1 and STAT3 | SOAT1 influences the transcriptional activity of STAT3 | 0.84183 | HepG2 cells were loaded with LDL to stimulate SOAT1, but no changes were observed in STAT3 at either the transcript or protein level | Not supported |
The experiments were conducted at an independent contract research organization (CRO) specializing in drug discovery, under the supervision of aging genetics experts.
For the first hypothesis—that "TP53 influences the expression of BAMBI" (confidence 0.85391)—inducing P53 within cells led to a clear increase in BAMBI expression in both the initial screening and the confirmatory assay. This increase was synchronized with the timing at which P53 protein reached its maximum expression.
The second hypothesis—that "RAF1 decreases (suppresses) the expression of TNF" (confidence 0.83114)—was also supported by experiments in which cells were treated with a RAF1 inhibitor. Following administration of the inhibitor, increased ERK phosphorylation, indicative of RAF1 dimerization and transactivation, was observed, and in step with this, TNF mRNA expression consistently increased. The authors position these two cases as instances in which previously unrecorded, novel interactions in biomedicine were confirmed.
However, the third hypothesis—that "SOAT1 influences the transcriptional activity of STAT3" (confidence 0.84183)—was not supported. Although HepG2 cells were stimulated with low-density lipoprotein (LDL) loading to activate SOAT1, no significant changes were detected in either transcript levels or protein expression levels. At least under this particular experimental design and cell system, the predicted interaction was not reproduced.
This outcome underscores a fact that demands caution in interpreting the model's outputs. Among the confidence scores, the highest, TP53 and BAMBI (approximately 0.854), succeeded; the second highest, SOAT1 and STAT3 (approximately 0.842), failed; and the third, RAF1 and TNF (approximately 0.831), succeeded. The magnitude of the confidence score did not simply rank-order the success or failure of the experiments.
The Limits of the Demonstrated Molecular Interactions and Remaining Challenges for Scientific Discovery Systems
Regarding the two experimentally supported relationships, the authors discuss implications for drug discovery and drug repurposing (redeveloping existing drugs for new uses), but these remain conditional hypotheses at best.
For example, regarding the relationship between TP53 and BAMBI, the directionality of regulation within a living organism has not been determined by this experiment alone. The authors present two parallel interpretations: if a context exists in which TP53 suppresses BAMBI, then P53 may be enhancing the tumor-suppressive effect of TGF- signaling; conversely, if TP53 activates BAMBI, it may be acting as a brake to restrain excessive TGF- activity. Similarly, regarding the interaction between RAF1 and TNF, the authors discuss the possibility that it could lead to new therapeutic approaches for cancer, autoimmune diseases, and chronic inflammatory conditions—but this remains a proposed mechanistic model, not a demonstrated therapeutic effect.
In interpreting the study as a whole, excessive statistical extrapolation should be strictly avoided. The result that two of three hypotheses were confirmed experimentally (an approximately 66.7% success rate) is merely a figure derived from a tiny sample of just three candidates, selected by human experts based on interest and testability, out of a filtering process applied to a vast pool of 1.5 million candidates. This cannot be treated as a measure of the overall accuracy or hit rate of the system as a whole.
Furthermore, the findings obtained constitute limited evidence based on in vitro assays using specific cultured cell lines. Whether these effects would function similarly in vivo or in a human clinical setting, or whether they hold universally across other tissues or different experimental systems, has not been shown at all.
Approaches that train on past literature as supervisory data always carry the risk of literature bias and potential information leakage. The very framework of predicting future trends from past descriptions is inherently prone to being pulled toward the research trends and interests of specific research communities contained within the training data. Whether Hakken can truly serve as a foundation for deriving genuinely unknown scientific laws will have to await results beyond this limited preprint's findings—specifically, replication by different research teams and the outcomes of larger-scale, systematic blind testing.