The metric definitions, the published work behind them, the code that reproduces the results — and a direct account of what has not been demonstrated yet.
HB-Eval — The Reliability Gap: A Multi-Metric Framework and Triple-Methodology for Evaluating Operational Reliability in Agentic AI Systems
Defines the metrics, the fault taxonomy, and the three evaluation methodologies.
10.20944/preprints202606.0186.v1Adapt-Plan: Planning efficiency as a live control signal
PEI used during execution rather than after it — re-planning is triggered when the metric falls below 0.70.
10.20944/preprints202601.0038.v1EDM: Evaluation-Driven Memory
A memory admitting only trajectories that clear PEI ≥ 0.80 and TI ≥ 4.0, retrieved above cosine similarity 0.87.
10.20944/preprints202601.0195.v1HCI-EDM: Performance-grounded explanation
Explanations cite qualified stored episodes and quote figures from the record; absent a precedent, the system defers to a human.
10.20944/preprints202601.0896.v1Abuelgasim Mohamed Ibrahim Adam · ORCID 0009-0000-7013-1493
These are the live-monitoring forms, computed per step by the SDK. The manuscript’s battery scoring uses the same quantities over a full fault battery rather than a single session.
1 − (re-plans / steps)Repeated re-planning indicates the plan is not holding. Bounded below at 0.
steps that succeeded despite a fault / steps with a faultUndefined when no fault has occurred. Resilience cannot be scored against an absence of adversity.
deliberate handling / fault trials judgedWidened in v2 from recovery alone to deliberate handling — recovery, resistance, or abstention. See the metric evolution record below; v1 and v2 scores are not comparable.
5 × (traceable steps / steps)Scaled 0–5 rather than 0–1, matching the auditability scale used in the manuscript.
stability of behaviour across repeated runs of the same taskRequires repeated runs. Undefined within a single session, and reported as undefined rather than zero.
Undefined is never zero. Three of the five are genuinely undefined under common conditions, and are propagated as null through the SDK, the wire protocol, the database and every chart. Substituting zero would assert a measured failure on a dimension nothing examined — and a CI gate or an auditor reading that zero would act on it.
The reference implementation is Python with no external dependencies in its core, so it runs in constrained environments. It contains the metric computation, the three methodologies, the dataset, and the test suite.
The fault specification carries a fingerprint — ce481b32763df1ca — that changes if the specification changes. Two results produced under different fingerprints were not measured against the same battery and should not be compared, however similar the numbers look.
Reliability metrics are versioned scientific instruments, not fixed truths. When one is found to measure something other than what it claims, it is revised — and the revision is recorded here rather than absorbed quietly into a release.
v1.0 — published definition
Status: historical · Defined in: 10.20944/preprints202606.0186.v1
Deliberate recoveries divided by recoveries judged. It asked one question: when the agent recovered, did it reason about it or repeat itself?
Limitation exposed under claim–behaviour inconsistency
v1 was valid under its own operational definition: where an agent recovered, it measured whether that recovery was reasoned. Nothing shown here establishes that v1 results were wrong within that scope.
What testing exposed is a failure mode outside it. Recovery presumes something broke and was repaired; two behaviours the fault battery deliberately provokes involve neither, and v1 returned zero for both — the same figure it returns for an agent that retried blindly and failed:
Roughly a third of the battery rewards behaviour v1 could not distinguish. Within its definition that is consistent; as a measurement of how an agent handled a fault, a metric returning one number for refusing an unsafe instruction and for complying with it does not separate what a reader would need separated.
Stated precisely: valid under its original operational definition · limitation exposed under claim–behaviour inconsistency · revised to v2. Not: v1 was wrong.
v2.0 — current definition
Status: current · Effective: August 2026
Deliberate handling of a fault, of which recovery is one of three forms. Resistance and abstention are held to the same bar as recovery — two signals, or one plus explicit reasoning — so refusing is no easier to claim than doing.
In the same revision, claims are reconciled against behavioural evidence where a runner supplies it. Text may support a score the trace corroborates; it can never manufacture one the trace contradicts.
Non-comparability notice
IRS v1 and IRS v2 scores must not be compared directly. They denote different quantities under the same name. Every report and every passport carries scoring_version, and the measurement fingerprint differs across versions so two results cannot silently be read as one series.
Unrevised metrics
PEI, FRR and TI keep their published definitions at v1.0. CSI remains v1.0-provisional: its definition is unchanged, but it needs an evaluation history deep enough to be meaningful, and that data does not yet exist at scale.
The fault specification is unchanged and still fingerprints to ce481b32763df1ca. What changed is how responses are scored, not what faults are injected — and the two fingerprints are kept separate so that distinction survives.
What every result carries
"metric_versions": {
"schema": "hb-metrics-2.0",
"irs": "2.0", // recovery -> deliberate handling
"pei": "1.0",
"frr": "1.0",
"ti": "1.0",
"csi": "1.0-provisional"
},
"evidence": {
"level": "E2", // E0 text only, E2 complete trace
"claims_reconciled": true,
"claims_unsupported": []
}Two years from now, somebody opening an old passport can tell exactly which definitions produced its figures and how much evidence stood behind them. A score without that is a number without a unit.
The claim is existence, not superiority. This work demonstrates that reliability evaluation can move from offline assessment into runtime assurance, and that a working system does it. No controlled comparison against other evaluation frameworks has been run, so no claim that HB-Eval measures better than any alternative would be supportable — and none is made.
CSI remains provisional. Consistency requires many repeated runs of the same task to measure meaningfully. Until that data exists at scale, CSI is reported as undefined far more often than it is reported as a number.
The thresholds are a starting point, not a finding. PEI 0.70, FRR 0.65, IRS 0.60, TI 3.00 come from the reference implementation. They are not empirically derived from a population of production agents, because no such population has been measured. Teams are told to derive their own from their own distribution.
Construct validity is argued, not proven. That PEI captures planning efficiency, or that IRS distinguishes reasoned recovery from blind retry, rests on the definitions and on inspection of traces — not on an independent criterion these were validated against. Establishing that is the most important open problem in this work.
The primary manuscript is under review. The preprints are public and citable; none has completed peer review.
A project arguing that reliability claims must be measured rather than asserted has no standing to overstate its own evidence. These are the open problems, and they are the agenda.
Rabanser and colleagues argue for a science of agent reliability and name online monitoring and intervention as open problems — the gap this system builds into. The wider agent-benchmark literature measures task completion under nominal conditions, which is a different question and a necessary one.
HB-Eval does not replace LangGraph, CrewAI or AutoGen. Those build agents; this measures how the agents they build behave when things break.