On October 19, 2026, OpenAI published new evaluation results on how well AI can develop software in programming languages with relatively little code and documentation available for training, such as COBOL and Fortran. In MultiLangBench, which asks models to implement a SQLite-compatible database engine in 20 languages, GPT-6 Astra reached an average test pass rate of 87.4%, up sharply from GPT-5.6 Sol's 26.8%. A separate evaluation, in which models migrate existing COBOL programs to other languages, also showed marked improvement, but it tests whether the new program can return results identical to the original. To understand these results, it helps to distinguish between how well a model can implement new software and how well it can rewrite an existing system while preserving its behavior.
87.4% average across 20 languages in the SQLite implementation test
MultiLangBench asks models to build a SQLite-compatible database engine from scratch in a specified programming language. SQLite is a lightweight database engine that stores data in a file, and the model is responsible for the whole codebase, from design to state management, error handling, and consistency with existing functionality. What OpenAI calls "low-resource languages" are languages with less source code and technical documentation available for AI training than major languages such as Python and JavaScript. COBOL, Fortran, Visual Basic, and others are still used in important business systems today, but a shortage of training data has long been a challenge for AI-assisted development in them.
According to OpenAI's evaluation, GPT-6 Astra, run at maximum reasoning effort, outperformed GPT-5.6 Sol in all 20 target languages. COBOL's test pass rate rose from 10.9% to 72.1%, and Fortran's from 21.4% to 83.5%. OCaml improved from 25.6% to 85.8%, and Visual Basic from 27.5% to 89.4%. Nineteen of the 20 languages exceeded 83%, while COBOL stayed at 72.1%, a reminder that the average alone hides differences in performance between languages.
GPT-6 Luna, a lower-priced model, also recorded an average test pass rate of 40.3% at maximum reasoning effort, beating GPT-5.6 Sol in all 20 languages. Its estimated average API cost per sample was $0.77, about 78% lower than GPT-5.6 Sol's $3.48. In other words, the results present two options: Luna, which costs less while still beating the previous model, and Astra, which achieves a higher pass rate. However, this cost comparison is between Luna and GPT-5.6 Sol and does not show the cost of Astra, which achieved the 87.4%. These figures are also benchmark estimates, and the same cost-effectiveness is not guaranteed in real development projects.
The test examines the database that was actually modified, not the generated code
According to the MultiLangBench README published by OpenAI on GitHub, the evaluation gives the model the official SQLite 3.40.1 documentation and has it implement a compatible engine using that as a reference. What sets it apart is that scoring is based not on explanations of the generated code or how it looks, but on whether the program correctly operated on an actual database file. Because the file created by the candidate program is read by genuine SQLite, behavior can be verified against a common standard even when the implementation languages differ.
Specifically, the candidate program is first given an initial database file and SQL statements. It processes the SQL and rewrites the database file, and the modified file is then opened with SQLite 3.40.1 and checked by running verification SQL. If the result matches the expected string exactly, the test passes. This approach can distinguish programs that merely accept SQL statements without actually processing them, and implementations that display correct results but fail to update the database file properly. The explainer site explainx.ai also notes that verifying the files left by the candidate program with genuine SQLite makes it possible to evaluate even the on-disk file format. That site's commentary, however, is an explanation of the method rather than an independent replication of the results.
The evaluation environment also has restrictions to prevent passing tests by illegitimate means. The README requires removing not only existing SQLite libraries and executables but also SQLite implementations embedded in other programs. Internet access is not allowed, and an implementation in a language other than the specified one, or one mixing multiple languages, is scored at a 0% test pass rate. A test also fails if the program crashes or does not finish within 60 seconds.
When interpreting the published scores, though, the scope of the tests used also matters. MultiLangBench has converted about 24,000 of SQLite's roughly 51,000 existing TCL tests for evaluation. Because running all of them would be costly, however, the README recommends using 272 tests whose names begin with 00, 01, or 02 for normal evaluations. Using the full set is permitted, but the test set used must then be stated. OpenAI's announcement does not specify which test set was used to calculate the published scores, so it cannot be assumed that the 87.4% figure comes from the recommended 272 tests. In any future third-party replication or comparison with other models, the test set and the conditions for calculating API cost will need to be aligned, along with the target languages.
COBOL migration demands output identical to the original
The other evaluation examines whether models can rewrite existing COBOL programs in C++, Java, Kotlin, Python, or Rust while preserving the original behavior. According to OpenAI, GPT-6.1 Sol recorded an 86.6% success rate at low reasoning effort, far above GPT-5.6 Sol's 5.4% under the same conditions. API cost was also held to 61% of the older model's, a 39% reduction. At higher reasoning effort, the success rate reached 97.9%. However, 86.6% and 97.9% were measured at different reasoning-effort settings; 97.9% was not achieved at low reasoning effort.
This evaluation judges success differently from MultiLangBench. MultiLangBench aggregates pass rates across individual tests, whereas the COBOL migration evaluation requires externally observable output to match byte for byte across all hidden tests. A migration counts as successful only when a program passes every test. So even if some inputs return the same results as the original, a migration is not accepted as successful if other inputs produce different results.
OpenAI cites COBOL's fixed-point arithmetic as an example of why such strict verification is needed. In COBOL, extra decimal places in a calculation result may be truncated unless rounding is explicitly specified. If this is replaced with the common rounding behavior of the target language, the program itself may run normally while its results differ from the original system's. In financial transactions and accounting, such small differences can lead to business problems. Generating code that merely compiles, in other words, does not amount to correctly migrating an existing system.
These details of processing are especially important in old business systems. When the original developers have already left and design documents and specifications are not sufficiently preserved, business rules may be recorded only in the program itself. The handling of missing records, exception processing, and number rounding may not be mere implementation conveniences but specifications that reflect years of operational practice. Migrating from COBOL to another language requires not just replacing old syntax with new syntax but accurately reading these implicit specifications.
For real-world deployment, per-language performance and behavioral reproducibility matter
Both evaluations OpenAI published measure AI programming ability, but the tasks and passing conditions differ. The differences can be summarized as follows.
| Evaluation | Task given to the model | Pass criterion | Cautions when reading results |
|---|---|---|---|
| MultiLangBench | Implement a SQLite-compatible engine from scratch in a specified language | Pass rate on individual tests that verify the modified database | Distinguish the 20-language average from individual language scores, and check which test set was used |
| COBOL migration | Rewrite an existing COBOL program in another language | Share of migrations whose output exactly matches the original program on all hidden tests | Align reasoning effort, and check that the original processing results are preserved |
A high score on MultiLangBench does not mean existing COBOL systems can be migrated at the same success rate. Conversely, a high 97.9% success rate in the COBOL migration evaluation does not prove that every business system can be migrated safely. All of these figures come from OpenAI's own evaluations, and whether equivalent results can be obtained in real enterprise systems needs to be verified separately. OpenAI itself explains that production deployment requires expert review, as well as testing that takes business rules and dependencies on external systems into account.
For companies considering AI-assisted development or system migration, performance on the languages and code they actually use matters more than average benchmark scores. In migrations in particular, it is necessary to confirm that the same inputs produce the same results before and after, and to verify down to rounding, the handling of missing records, and behavior when exceptions occur. These results suggest that AI is increasingly likely to handle complex development work even in languages it has traditionally struggled with. But turning that capability into updates of real business systems will continue to depend not on the volume of generated code or test pass rates alone, but on how faithfully the behavior of the existing system can be reproduced.