Gregory Kurtzer, who launched Rocky Linux, has released OpenWALDO, a project for collaboratively building AI training data. As of August 13, 2026, the live index had reached 202,834,581,119 reference tokens. That's roughly a 21% increase in about 8 hours from the approximately 167.3 billion tokens The Register confirmed the previous day. However, the real value of this project lies not in the race to accumulate more data, but in a mechanism that lets third parties trace which data, under what conditions, was used for which model.
From 167.3 Billion to 202.8 Billion: An 8-Hour Jump in the Public Corpus
As of 00:49 UTC on August 13, 2026, the public index listed 47 corpora, 949 Parquet shards, and 97,741,086 documents. The encoded bytes reported in status.json amount to approximately 256.6 billion bytes. Each shard records document counts and token counts. It also logs provenance and licensing claims, with SHA-256 hashes fixing identity. The live tally points to Git commit 6f07b0b.
The substance of this increase can be traced through Git history. The Register recorded approximately 167.3 billion tokens at 16:57 UTC on August 12. Seven hours and fifty-one minutes later, OpenWALDO added 35,503,670,607 tokens of PubMed Central open-access papers. Nearly the entire increase of roughly 35.5 billion tokens is explained by this single commit.
The breakdown by category is as follows.
| Category | Corpora | Reference Tokens |
|---|---|---|
| science | 5 | 84,417,683,415 |
| core | 17 | 75,282,070,378 |
| government | 4 | 19,129,654,689 |
| code | 5 | 12,725,616,820 |
| community | 10 | 6,509,147,414 |
| law | 1 | 4,621,071,246 |
| post-train | 4 | 117,173,399 |
| math | 1 | 32,163,758 |
Scientific papers and general-purpose materials account for the vast majority of the total, while the post-train group used for instruction tuning and similar purposes makes up only about 0.0578%. While volume is growing rapidly, the current content is heavily skewed toward pre-training text. A public corpus does not by itself determine a model's conversational performance.
What's New Isn't the Data Volume, It's the AI BOM
OpenWALDO doesn't cram large volumes of data into a Git repository. Git holds descriptions and provenance of the corpora. License claims and transformation methods are also recorded, with counts and object references managed within the same history. The actual body data is stored as normalized Parquet files in external lookaside storage, identified by a SHA-256 computed from the content. This keeps the changes in Git—which describe "what the data means"—at a size humans can review.
When a user selects a corpus, WALDO fixes the index revision and target manifest. It resolves source and license, then assembles a Bill of Materials (BOM) that bundles per-shard counts and hashes. This record doesn't stop at the training plan—it flows into an ORIGIN-BOM that records the starting weights, a RUN-BOM that records the execution environment, loss, and checkpoints, and a BOM that records the distributed artifacts. It's a design that brings the SBOM concept—tracking dependent components in software—into the training data and model lifecycle.
Contributions require a Developer Certificate of Origin (DCO) sign-off. Who brought in which claim remains on the public record, and errors can be corrected as ordinary Git changes. This makes accountability far clearer than an anonymous file dump.
Even so, what a hash proves is identity and integrity. OpenWALDO itself states plainly that it doesn't guarantee that license claims are legally accurate, that upstream Git repositories will continue distributing the same commit in the future, or that a trainer actually consumed all the selected data. Nor does DCO automate the review of redistribution rights. It's a mechanism that aligns the entry point for audits; legal review and quality assessment remain human tasks.
A Starting Point Where 83.13% Comes From Common Pile
Aggregating the live index by source URL shows that 24 corpora, totaling 168,606,550,174 tokens, originate from Common Pile. That's 83.13% of the total. Many of the top entries—the largest, peS2o, the second largest, PubMed, along with Stack Exchange, Wikimedia, and pre-1929 books—incorporate Common Pile's filtered releases.
Common Pile v0.1 is a collection of public-domain and openly licensed documents exceeding 8TB, presented at NeurIPS in 2025. It contains more than 30 sources and even released checkpoints of a 7B model trained on 1 trillion and 2 trillion tokens. OpenWALDO has converted part of this pre-existing asset into a new auditable format, making it selectable from the same index alongside other government documents, code, and mailing lists. Right now, its distinctiveness lies less in the volume of newly collected text than in having aligned existing data into a common evidentiary format.
The scale, too, is still developing. Ai2's Dolma is a public corpus of 3 trillion tokens, and OpenWALDO's current figure is about 6.76% of that. Qwen3 was pre-trained on roughly 36 trillion tokens, putting OpenWALDO at about 0.563% of that scale. However, OpenWALDO's new PubMed manifest counts reference tokens using cl100k_base, while models use different tokenizers and read the same data multiple times through weighting and repetition. These ratios are only rough indicators of order-of-magnitude difference, not a measure of training capability.
Simply increasing the amount of data isn't enough on its own. What determines model performance is how many times each domain is read, how duplicates and low-quality documents are filtered out, and what supplements instruction tuning and preference learning. OpenWALDO's post-train group stands at only 117,173,399 tokens, and SFT and preference training remain unimplemented even in WALDO's official roadmap. Transparency in public data has advanced, but whether it can be converted into a competitive model remains to be measured.
A Transparency Infrastructure Emerging After EU Enforcement Began
On August 2, 2026, the European Commission activated its enforcement authority over General-Purpose AI (GPAI) model providers. OpenWALDO's announcement came just ten days later. Under Article 53 of the AI Act, GPAI providers must publish a summary of training content using a designated template. Even models released as open source are not exempt from the obligations around copyright policy and training content summaries.
The summary covers the type and scale of data and addresses both public and private datasets. This includes web crawls and user data. Synthetic data and processing methods must also be documented. For models undergoing continued training, the summary must be updated at least every 6 months, or sooner if the content changes significantly. Penalties for violations can reach up to 3% of the previous year's global turnover or €15 million, whichever is higher. A BOM that separately tracks a dynamically growing corpus from actual training runs fits well with this kind of update workflow.
WALDO has implemented machine-readable disclosure JSON aimed at the EU and checks for missing items. But the feature to generate the European Commission's official Word template is not yet complete, and the project's own development documentation cautions that output alone doesn't constitute legal compliance. While the timing of the announcement overlaps with regulatory demand, CIQ has not publicly stated EU regulation as the reason for starting the project. Regulatory compliance is a plausible use case, but there's no basis to declare it the founding motivation.
The Open Source Initiative's (OSI) Open Source AI Definition 1.0 also requires providing sufficient data information, training and execution code, and model parameters in a form suitable for modification. OpenWALDO can serve as a component supporting data and training records. However, with no published model parameters currently available, OpenWALDO itself is not, at this stage, a complete open-source AI system.
Three Gaps to Fill Before It Can Produce a Model
The first gap is the absence of a working model and evaluation results. The Register asked CIQ whether any model had been trained on the OpenWALDO corpus, but received no response. The official site likewise shows no published models, benchmarks, or measured training costs. Even if the BOM chain links up correctly, the quality of the data mixture and model performance still need to be verified separately.
The second is computational scale. WALDO can run training on Apple Silicon's MLX, PyTorch on Linux, and single-node TorchTitan. However, it does not yet support common Hugging Face tokenizers and architectures, multi-node execution, generation in PyTorch, supervised fine-tuning (SFT), preference learning, or a formal release package. For a community to run pre-training at the scale of tens of trillions of tokens, beyond sharing data, there needs to be a mechanism for who bears the cost of large-scale compute and how failed training runs get reproduced.
The third is the institutional design of the community itself. WALDO's CLI code is licensed under Apache-2.0, but as of commit 6f07b0b on August 13, 2026, the root of waldo-index has no LICENSE file, and the GitHub API returns null for its license. While individual corpora record 51 distinct license expressions, the terms for reusing the index's own descriptions and metadata should be made explicit separately. Given how recently this was released—the CLI has 2 stars, the index has 1 star, and neither has been forked yet—it's still impossible to judge whether this will move from CIQ-led development to sustained outside contribution.
What will determine OpenWALDO's success is not how fast the figure of 202.8 billion tokens closes in on 36 trillion. It's whether a third party can use the same BOM to extract the same data selection, retrain a model in a different computing environment, and feed the resulting differences back into the public Git history. If that loop starts working, training data will shift from being each company's private inventory to becoming infrastructure the community can improve together.