IBM Research released the Open Agent Leaderboard, a new evaluation platform that scores Agent systems (model + harness + tools) rather than just the underlying model. The standout finding: the same model can produce dramatically different scores depending on the Agent harness, with gaps up to 30 points on the same benchmark.
The methodology: the leaderboard evaluates "Agent systems" — a model (e.g., Llama-3-70B) + an Agent harness (e.g., LangChain, AutoGen, custom) + a set of tools. The same model is run on the same task with different harnesses, and the score variation is measured. The result: the harness contributes 15-30 points of variation, more than the model-to-model variation in many cases.
The benchmark: on a set of 10 Agent tasks (web navigation, code execution, data analysis, etc.), Llama-3-70B with the "best" harness scores 68.4, with the "worst" harness scores 38.2. The gap is 30 points — much larger than the gap between Llama-3-70B and GPT-4 (which is 15 points).
The analysis: the variation comes from three sources — (1) the harness's prompt template (different harnesses have different system prompts, which significantly affects the model's behavior); (2) the tool integration (some harnesses have better error recovery); (3) the memory management (some harnesses use long-term memory, some don't).
The bigger takeaway: "model evaluation" is no longer sufficient — we need "Agent system evaluation." The Open Agent Leaderboard is a significant step in this direction, and the finding that "harness matters more than model" has big implications for the industry. For the industry, this means Agent vendors should focus on harness quality, not just model quality.