Operational reliability for agentic AI
Benchmarks measure whether an agent completed its work. Almost nothing measures how it behaved while things were going wrong — and that is where production incidents come from. HB-Eval measures it, watches it live, and can stop a run before the damage lands.
The demo needs no account. 500 evaluations a month on the free plan.
An agent can retry the same failing call three times, lose its reasoning trail, recover by accident, and still deliver the right answer. Pass/fail scores it a success. The next time the same weakness appears, it will not be so lucky.
What benchmarks measure
Task completed
Nominal performance, under conditions where nothing broke.
What nobody measures
Behaviour under fault
Whether recovery was reasoned, whether the plan held, whether anyone can reconstruct what happened.
Each answers a question a completion score cannot.
PEI
Planning Efficiency
Is the plan holding, or is the agent redoing it?
Repeated re-planning means the agent is thrashing, not thinking.
FRR
Failure Resilience
Of the steps where a fault was present, how many still completed?
Undefined until a fault actually happens — resilience cannot be scored against nothing.
IRS
Intentional Recovery
Was the recovery reasoned, or a blind retry?
The difference between an agent that adapts and one that repeats itself faster.
TI
Traceability
Can each decision be followed afterwards?
An agent nobody can audit is an agent nobody should deploy.
CSI
Consistency Stability
Does the same task produce the same behaviour across runs?
Needs repeated runs, so it stays undefined inside a single session.
Undefined is never reported as zero
A metric that was never measured shows as a dash, all the way to the wire. Reporting it as 0.00 would claim a measured failure on a dimension nothing examined — and a chart, a gate or an auditor reading that zero would draw a confident, wrong conclusion.
Wrap your existing loop. Nothing about the agent changes, and the instrumentation costs about 0.002 ms per step.
pip install hb-eval-sdk==2.9.0
from hb_eval_sdk import HBEvalClient
client = HBEvalClient(api_key=..., aes_key=..., signing_secret=...)
with client.monitor(
agent_id="support-agent",
halt_policy={"metric": "frr", "below": 0.5, "for_steps": 3},
) as session:
for step in my_agent.run(task):
session.record_step(
action=step.name,
success=step.ok,
had_fault=step.faulted,
)
if session.should_halt:
break # cooperative: your loop decides how to stopEvery item links somewhere you can use it now.
Zero-setup instrumentation
One import derives reliability signals from OpenTelemetry spans you already emit. No new instrumentation code.
How derivation worksLive monitoring
Metrics computed per step, streamed while the agent runs. Timeline, replay, and comparison between runs.
Open monitoringSafe Halt
Stop a run when resilience collapses. Cooperative, sustained rather than instant, and off unless you configure it.
How halting worksPolicy engine
Retry, escalate or halt on a metric breach — with three independent limits so a policy cannot loop forever.
Policy referenceFault Injection Studio
Six fault types across six domains. Preview the exact text your agent would receive before running anything.
Open the studioCI reliability gate
Gate a pull request on reliability, comparing against your own baseline rather than an absolute floor.
Set up the gateAlerting
Slack, PagerDuty or a plain webhook. Twenty breaches raise one alert, not twenty.
Configure alertsObservatory
Aggregate reliability across contributed runs. Anonymous at write time, withheld until five independent accounts.
View the dataVerified evaluation
HB-Eval calls your agent itself, so the result cannot be shaped by the agent. Five runs a month on the free plan — enough to compare against a self-reported one.
How verification differsWorkspaces and audit export
Roles for a team, and a one-call export of every automated decision your agents made — including the alerts that failed to deliver.
Audit exportHow it fits together
Where your data goes, what leaves your machine, and what we can see — drawn from the implementation, not an intended design.
See the architectureStatus and retention
Health checked rather than asserted, with the last automatic data cleanup published so the policy is verifiable.
Service statusOne adapter call. Your agent keeps its own control flow — HB-Eval measures it rather than replacing it.
LangChain
adapt_langchain_agent(agent_executor)LangGraph
adapt_langgraph_agent(compiled_graph)CrewAI
adapt_crewai_agent(crew_agent)Not using a framework? The SDK works around any loop, and an OpenTelemetry-instrumented agent needs one import and no calls at all.
Beyond scoring
Most systems remember everything and explain with confident guesses. HB-Eval does the opposite: it remembers only what proved reliable, and explains only what it can ground in a qualified record.
Quality-governed memory
Every run is judged before it is remembered. A trajectory enters qualified memory only when it clears PEI ≥ 0.80 and TI ≥ 4.0.
A memory that stores everything eventually recalls its own worst behaviour as precedent. The gate is what stops an agent learning from the runs it should forget.
Performance-grounded explanations
Explanations cite stored episodes above a similarity threshold and quote figures from the record, not from a model. Where no precedent exists, the system says so and defers to a human.
Saying “I have no comparable case” is a useful answer. A fluent explanation with no evidence behind it is not, and is harder to catch.
Both layers are live in production and implement published work — EDM and HCI-EDM. Papers, definitions and open problems
HB-EVAL AGENT PASSPORT
Every halt decision with the policy that caused it, every alert and whether it was delivered, the five metrics over 30, 60 and 90 days — signed with Ed25519 and published at a link.
The signature is checked in the reader’s own browser against a public key we publish. An auditor, a customer or a regulator verifies it without asking us and without our permission — which is the only thing that makes such a record worth more than a claim.
The free plan is not a trial. It runs the full battery, all five metrics, live monitoring and signed passports, with no time limit.
Free
$0
Pro
Coming soon
Why five verified runs are free. On every other path the behavioural evidence comes from your own agent’s runner. On the verified path HB-Eval calls your agent itself, so the result cannot be shaped by the agent. That difference is the strongest thing the paid tier offers and the hardest to convey in a sentence — five runs is enough to see it for yourself.
The framework, the metric definitions and the fault methodology are described in a manuscript under review, with three companion preprints on adaptive planning, evaluation-driven memory, and performance-grounded explanation.
Two minutes, no account, no keys. The same agent with and without a reliability policy.