OpenAI's latest models, GPT-6 Astra and GPT-6 Luna, have significantly improved their capabilities in low-resource programming languages, achieving substantial gains in accuracy and cost-efficiency. This advancement could aid in maintaining and modernizing critical infrastructure that relies on these languages.
TL;DR
- OpenAI's GPT-6 Astra achieves 87.4% accuracy in building SQLite from scratch across 20 low-resource languages, a significant improvement from GPT-5.6 Sol's 26.8%.
- GPT-6.1 Sol demonstrates 86.6% accuracy in COBOL migration at low reasoning effort, compared to GPT-5.6 Sol's 5.4%, while being 39% more cost-efficient.
- These advancements could help teams maintain and modernize essential services that depend on low-resource programming languages.
What happened
OpenAI has developed new evaluations to measure progress in low-resource programming languages like COBOL, Fortran, and Visual Basic. These languages support critical infrastructure, such as online banking systems and weather forecasting, but have less available code and documentation for models to learn from.
The MultiLangBench evaluation tests a model's ability to build a SQLite-compatible database engine from scratch across 20 different programming languages. GPT-6 Astra achieved an average test pass rate of 87.4%, up from 26.8% for GPT-5.6 Sol, with improvements in every language tested. GPT-6 Luna also outperformed GPT-5.6 Sol, reaching 40.3% at approximately 78% lower estimated API cost.
The COBOL migration evaluation tests a model's ability to migrate existing programs to another language while preserving their behavior. GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared to GPT-5.6 Sol's 5.4%, while costing 39% less. At higher reasoning effort, accuracy reached 97.9%.
Why it matters
These advancements could help teams maintain and modernize the software that essential services depend on, as maintaining and modernizing this software can be difficult due to incomplete documentation and the unavailability of original developers.
For developers and startups, these improvements mean better tools for working with a wide range of programming languages, which could lead to more efficient and cost-effective software development.
For investors, this demonstrates OpenAI's continued innovation and commitment to improving its models, which could translate to better products and services for customers.
Key facts
- GPT-6 Astra achieved an average test pass rate of 87.4% on MultiLangBench, up from 26.8% for GPT-5.6 Sol.
- GPT-6 Luna reached an average test pass rate of 40.3% on MultiLangBench at approximately 78% lower estimated API cost than GPT-5.6 Sol.
- GPT-6.1 Sol achieved 86.6% accuracy on the COBOL migration evaluation at low reasoning effort, compared to GPT-5.6 Sol's 5.4%, while costing 39% less.
- At higher reasoning effort, GPT-6.1 Sol's accuracy on the COBOL migration evaluation reached 97.9%.
- MultiLangBench tests a model's ability to build a SQLite-compatible database engine from scratch across 20 different programming languages.
- The COBOL migration evaluation tests a model's ability to migrate existing programs to another language while preserving their behavior.
- OpenAI has released the MultiLangBench evaluation setup and test suite on GitHub for developers to inspect and review performance in the languages they use.
Context
Low-resource programming languages like COBOL, Fortran, and Visual Basic support critical infrastructure but have less available code and documentation for models to learn from. Improving AI's ability to work in these languages could help teams maintain and modernize the software that essential services depend on.
OpenAI's latest models, GPT-6 Astra and GPT-6 Luna, have shown significant improvements in accuracy and cost-efficiency in these low-resource programming languages. These advancements could lead to better tools for developers and startups, as well as demonstrate OpenAI's continued innovation for investors.
