AdvRAG on EnterpriseRAG-Bench: What an Independent Judge Actually Says

We stopped grading our own homework. A public, 500-question leaderboard we don't control, scored by an evaluator we don't run — here's where AdvRAG actually lands, and the two real pipeline bugs we found chasing the last few points.

October 1, 2026 · 5 min read

Everything we'd written about AdvRAG's accuracy up to this point was our own benchmark, our own corpus, our own judge — useful signal, but self-reported. EnterpriseRAG-Bench is different: a public, independently-run leaderboard, 500 questions against a real internal-knowledge-base corpus, scored by an evaluator we don't control, with every other system's number sitting on the same page as ours.

The real number

• Overall (correctness × completeness): 91.15
• Answer correctness: 94.2
• Answer completeness: 94.11
• Document recall: 97.99

Scored across all 500 questions via the benchmark's own metrics_based_eval.py — the same three-judge-consensus document-correction methodology every system on the board is scored with — run against our real production /api/advrag/query endpoint, not a specially-tuned offline script.

Where we started, and what we fixed

A first full run landed at 89.3% — strong, but with three question categories pulling the average down: questions that ask the model to count or enumerate across many source documents at once, and questions where two internal documents disagree on a specific code, name or threshold and the model has no way to tell which one is authoritative. Neither was a retrieval-recall problem in the usual sense — in most cases the right documents were already in context. The model just didn't know which document to trust, or couldn't see far enough to count every instance.

Two fixes, both scoped to this corpus only, shipped before the final run:

1. Wider document-level retrieval (35 whole documents, up from 20) — closes the real gap on counting/enumeration questions, where the earlier cap was silently excluding source documents an aggregation question needed to see.
2. An authority-conflict instruction — when two sources disagree on a specific identifier or threshold, prefer the canonical governing document (a design doc, a formal spec, an approved policy) over an informal one (a chat thread, a draft, an implementation note), and say which source it trusted when the conflict is material.

The leaderboard

* Self-evaluation — professional evaluation submitted to the maintainer of the below leaderboard.

huggingface.co/spaces/onyx-dot-app/EnterpriseRAG-Bench-Leaderboard

26 systems, last updated 01 October 2026. Full results submitted to joachim@onyx.app for inclusion.