OpenAI's latest model, GPT-6 Astra, has achieved an 87.4% average pass rate across 20 programming languages, marking a significant leap from its predecessor. The improvements are particularly notable in languages critical to banking, scientific computing, and essential services.
TL;DR
- OpenAI's GPT-6 Astra model achieves an 87.4% average pass rate across 20 programming languages, with substantial gains in COBOL, Fortran, and Visual Basic.
- GPT-6.1 Sol reaches 97.9% accuracy in migrating COBOL programs to modern languages, though deployment still requires expert review.
- OpenAI releases MultiLangBench evaluation tools on GitHub, inviting developers to test and contribute.
What happened
OpenAI has reported that its GPT-6 Astra model achieved an 87.4% average test pass rate across 20 programming languages, a significant improvement from the 26.8% pass rate of GPT-5.6 Sol. The evaluations were conducted using MultiLangBench, a new benchmark developed by OpenAI to assess the models' ability to build working software in various languages. The results show substantial gains in languages like COBOL (72.1%), Fortran (83.5%), and Visual Basic (89.4%), which are critical to banking, scientific computing, and essential services.
In a separate evaluation, OpenAI's GPT-6.1 Sol model demonstrated a 97.9% accuracy rate in migrating COBOL programs to modern languages like C++, Java, Kotlin, Python, and Rust. This migration process ensures that the underlying behavior of the programs remains unchanged, which is crucial for financial systems and other critical applications. The model achieved 86.6% accuracy at low reasoning effort and 97.9% at higher reasoning effort, reducing estimated API costs by 39%.
OpenAI also tested GPT-6 Luna, a lower-cost alternative, which achieved a 40.3% average pass rate at an estimated cost of $0.77 per sample, compared to $3.48 for GPT-6 Astra. Despite the cost reduction, Luna's performance remains considerably below that of Astra. OpenAI has made the MultiLangBench evaluation setup, test suite, and language-level results available on GitHub, allowing developers to inspect and contribute to the tests.
Why it matters
These advancements are particularly important for industries relying on legacy programming languages like COBOL, Fortran, and Visual Basic. These languages, despite their age, are still widely used in critical infrastructure such as online banking systems, weather forecasting, and nuclear reactor safety analysis. The ability of AI models to write and modernize software in these languages can significantly reduce the risk of system failures and improve operational efficiency.
For developers and startups, the release of MultiLangBench and the availability of GPT-6 models through OpenAI's API provide new tools and capabilities for coding tasks. The improvements in coding accuracy and cost efficiency can enhance productivity and reduce development costs, making AI-powered coding solutions more accessible and practical for a wider range of applications.
Investors may see these advancements as a positive indicator of OpenAI's continued innovation and market leadership in AI-powered coding solutions. The company's focus on improving performance in less commonly used but critically important programming languages demonstrates its commitment to addressing real-world challenges and expanding the applicability of its models.
Key facts
- GPT-6 Astra achieved an 87.4% average pass rate across 20 programming languages, compared to 26.8% for GPT-5.6 Sol.
- COBOL migration accuracy reached 97.9% with GPT-6.1 Sol at higher reasoning effort.
- GPT-6 Luna achieved a 40.3% average pass rate at an estimated cost of $0.77 per sample.
- MultiLangBench evaluation tools are available on GitHub for developers to test and contribute.
- COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions.
- Fortran is used in applications including weather forecasting and nuclear reactor safety analysis.
- GPT-6 Astra exceeded 83% pass rate in 19 of the 20 languages tested, with COBOL remaining the most difficult.
- OpenAI continues to seek developer and enterprise feedback on languages and workflows requiring further improvement.
Context
The advancements in AI-generated software and COBOL migration highlight the growing importance of AI in modernizing critical infrastructure. As industries increasingly rely on legacy systems, the ability of AI models to write and modernize software in less commonly used programming languages becomes crucial. OpenAI's focus on these areas demonstrates its commitment to addressing real-world challenges and expanding the applicability of its models.
The release of MultiLangBench and the availability of GPT-6 models through OpenAI's API provide new tools and capabilities for developers and startups. These advancements can enhance productivity, reduce development costs, and make AI-powered coding solutions more accessible and practical for a wider range of applications. For investors, these developments may indicate OpenAI's continued innovation and market leadership in AI-powered coding solutions.
