According to VentureBeat, the Terminal-Bench team has launched version 2.0 of their AI agent benchmark alongside Harbor, a completely new framework for testing agents in containers. The updated benchmark includes 89 rigorously validated tasks that are substantially more difficult and reliable than version 1.0, which saw rapid adoption after its May 2025 release. Early results from the new leaderboard show OpenAI’s GPT-5 powered Codex CLI leading with a 49.6% success rate, though no agent has broken the 50% barrier yet. Harbor enables developers to run thousands of containerized evaluations across cloud providers and integrates with both open-source and proprietary training pipelines. The dual release aims to standardize agent evaluation across the AI ecosystem, with researchers already integrating these tools into their workflows.
Why this matters
Here’s the thing about AI agents – everyone’s building them, but nobody’s had a great way to test them in realistic conditions. Terminal-Bench 1.0 was a good start, but it had problems. Some tasks were poorly specified, others broke when external services changed. Basically, it was the wild west of agent evaluation.
Now with version 2.0, they’ve manually validated every single task for hours. They’ve removed unstable dependencies and made everything more reproducible. The co-creator Alex Shaw noted on X that even though TB2.0 is harder, the top scores are similar to version 1.0 because the task quality is just that much better. That tells you something about how messy the previous benchmark was.
Harbor changes everything
But the real game-changer might be Harbor. This framework lets researchers run agents in containers at massive scale – we’re talking thousands of simultaneous evaluations. The Terminal-Bench team used it internally to run tens of thousands of rollouts while creating the new benchmark.
What’s brilliant about Harbor is its flexibility. You can test any container-installable agent, run reinforcement learning pipelines, even create custom benchmarks. It works with major cloud providers and integrates directly with Terminal-Bench 2.0. This is exactly the kind of infrastructure the field needs as we move beyond simple chat interfaces to actual autonomous systems.
The competitive landscape
Looking at the early leaderboard results is fascinating. GPT-5 is out in front, but barely. Claude Sonnet 4.5-based agents are right there too. The fact that nobody’s cracked 50% success tells you how challenging these tasks really are.
We’re seeing active competition across all the major platforms, but nobody’s running away with it. That clustering at the top suggests we’re hitting a plateau where incremental improvements matter. For companies building serious automation tools, having reliable testing infrastructure like this is becoming critical. When you’re deploying systems that need to work reliably in production environments, you can’t rely on flaky benchmarks.
What comes next
The team has a detailed preprint in the works covering their verification process and methodology. They’re clearly aiming to establish Terminal-Bench 2.0 as the new standard, and given how quickly version 1.0 was adopted, they’ll probably succeed.
As AI agents become more integrated into development workflows and operational environments, tools like these become essential. They’re building the foundation for a unified evaluation stack that could eventually become as standard as unit testing is for software today. The command line might seem old-school, but it’s where the real work gets done – and now we have a proper way to measure how well our AI assistants can handle it.
