Documentation

Reference

The five metrics

Tiers are governed by the weakest required metric, not the average — an agent cannot average its way to trustworthiness by excelling on some dimensions while failing a critical one.

Five Minutes

Quickstart

Everything below runs on the free plan, which is not a trial: 500 evaluations a month, all five metrics, the full fault battery, live monitoring, Safe Halt and signed passports, with no time limit. It also includes five verified evaluations a month — the path where HB-Eval calls your agent itself rather than scoring what your agent reports about itself.

1. 1. Install the SDK

   bashCopy

   pip install hb-eval-sdk==2.9.0

2. 2. Create an agent and copy its keys from Dashboard → Agents.

3. 3. Run your first evaluation

   pythonCopy

   from hb_eval_sdk import HBEvalClient
   
   client = HBEvalClient(api_key=..., aes_key=..., signing_secret=...)
   
   def my_agent(system_prompt: str, question: str) -> str:
       return call_your_model(system_prompt, question)
   
   base_task = {
       "system": "You are a customer support agent.",
       "question": "A shipment is three days late. Respond appropriately.",
   }
   report = client.evaluate_with_battery(base_task, my_agent, n_scenarios=18)
   print(report["verdict"], report["aggregate_metrics"])

Post-Hoc Scoring

Evaluation — after the run

Evaluation runs a fault-injection battery against an agent and returns a verdict once it finishes. There are two paths, and they answer different trust questions.

Local — evaluate_with_battery

report = client.evaluate_with_battery(
    base_task, my_agent, n_scenarios=18,
)

Verified — request_verified_evaluation

report = client.request_verified_evaluation(
    agent_url="https://my-agent.example.com/run",
    base_task=base_task,
    consent=True,      # the platform will call this endpoint
    n_scenarios=30,
)

Already have a LangChain, LangGraph or CrewAI agent? Adapters wrap it into the shape both paths expect — see the Evaluate page for framework-specific snippets.

Real-Time

Monitoring — during the run

Evaluation judges a run once it is over. Monitoring watches an agent while it executes, recomputing the metrics after every step, so a collapse is visible at the step it happens rather than in a post-mortem.

Per-step signals are computed locally, in your process — real-time monitoring cannot afford a network call per step, and that is also why basic monitoring is free. Only the session summary, sent in small periodic batches, reaches the platform.

with client.monitor(agent_id="my-agent") as m:
    for step in agent.run(task):
        m.record_step(
            action=step.name,
            success=step.ok,
            had_fault=step.fault_injected,
            recovered_intentionally=step.deliberate_recovery,
            traceable=step.has_reasoning,
        )
        if m.should_halt:
            break

print(m.summary)

Sessions and live metrics appear on the Monitoring dashboard as they happen.

Measurement That Acts

Safe Halt — stop the collapse

Detecting a collapse is only half of what a safety mechanism owes you — the other half is stopping before the damage compounds.halt_policyturns a sustained breach into a halt decision.

with client.monitor(
    agent_id="my-agent",
    halt_policy={"metric": "frr", "below": 0.5, "for_steps": 3},
) as m:
    for step in agent.run(task):
        m.record_step(...)
        if m.should_halt:
            print(m.halt_reason)
            break

Automated Response

Policy engine — act on a breach

Safe Halt stops a run. A policy can try something less drastic first: retry the step, or escalate to a human, and halt only when neither helped.

with client.monitor(
    agent_id="my-agent",
    policies=[
        {"when": {"metric": "frr", "below": 0.6, "for_steps": 2},
         "then": "retry"},
        {"when": {"metric": "irs", "below": 0.4, "for_steps": 3},
         "then": "escalate"},
        {"when": {"metric": "frr", "below": 0.3, "for_steps": 5},
         "then": "halt"},
    ],
) as session:
    for step in my_agent.run(task):
        session.record_step(action=step.name, success=step.ok,
                            had_fault=step.faulted)

        directive = session.take_directive()
        if directive and directive.action == "retry":
            my_agent.retry_last()
        elif directive and directive.action == "escalate":
            notify_a_human(directive.reason)

        if session.should_halt:
            break

Off unless configured. No policies argument means observation only. Acquiring the ability to retry somebody’s agent by upgrading a library would be a surprise nobody asked for.

Three limits stop a loop. A policy fires at most three times per session; a directive is consumed when read, so it cannot be acted on twice; and two failed retries escalate rather than retry again. A permanently failing agent therefore produces two retries and one escalation, then silence — not five hundred retries and a bill.

There is no rollback action. Undoing an agent’s effects requires knowing what they were and how to reverse them, which is specific to your system and unknowable to a monitoring library. A policy claiming to roll back would be claiming something it cannot deliver, and the failure would surface at the worst possible moment.

Zero Setup

OpenTelemetry — no new instrumentation

If your agent already emits OpenTelemetry spans, reliability signals can be derived from them. One import, no calls to add.

import hb_eval_sdk.auto   # that is the whole setup

# Your agent runs unchanged. Spans it already emits become steps.

One judgement no trace can make. IRS counts deliberate handling in three forms, and spans show only two. An agent that refuses an unsafe instruction, or declines to answer without a reliable source, emits no span saying so — it simply does less, which from the outside looks identical to giving up. Pass handled_deliberately=True on those steps, or derived IRS will understate your agent.

What is derived, and from what:

• A span recording an exception, or setting its status to ERROR, becomes a step with a fault.
• A retry — the same operation name repeating with no different operation between — becomes a step with no deliberate recovery.
• A span carrying reasoning or decision attributes counts as traceable.
• A planning span reappearing after execution has begun counts as a re-plan.

What cannot be derived. A step that fails silently — no exception, no error status, just a wrong answer returned confidently — is invisible on this path. Spans record what happened mechanically, not whether it was correct. Derived metrics are therefore a floor on how bad things were, never a ceiling, and that is worth knowing before relying on them.

Explicit record_step calls stay more accurate, because you know things about your agent that its spans do not say. Derivation exists to get measurement in place today rather than after a week of instrumentation work.

Choose Before You Run

Fault Injection Studio — see the injection

The battery injects six kinds of fault across six domains. “Context corruption” is a label until you see what it does to a prompt, so the studio shows the exact text your agent would receive — without running anything, spending quota, or needing an account.

• tool_failure — a dependency goes offline; the agent must work from what it has.
• context_corruption — data arrives conflicting; the agent must notice rather than trust it.
• stochastic — noise with no real signal; the agent must not overreact to nothing.
• adversarial — pressure to skip verification; the agent must hold its standards.
• cascade — primary fails and the backup degrades; failures compound.
• combined — several at once, which is what production actually looks like.

report = client.evaluate_with_battery(
    task, my_agent,
    n_scenarios=18,
    seed=42,                                    # same seed, same battery
    fault_types=["tool_failure", "cascade"],    # optional narrowing
)

Narrowing has a cost, and it is reported. A run covering two fault types is not comparable with one covering six, however similar the numbers look. Both the studio and the report say so, rather than letting a green result from a narrow battery read like a green result from a full one.

Conduct Over Claims

Behavioural evidence

The battery scores what your agent returns. If that is only text, the scorer has to infer conduct from prose — and the prose is written by the thing being measured.

A counterfactual test showed the cost precisely. The same run — three identical retries, no re-plan, unsuccessful — scored:

• “I recognized the failure and deliberately replanned” — FRR 1.00, TI 4.91, PEI 0.83
• “The operation failed.” — FRR 0.00, TI 0.00, PEI 0.20

An agent that failed and said so scored worse than one that failed and did not. That is not a weak signal; it is an inverted one.

Supplying evidence

Return a dict instead of a string. Every field is optional, and a plain string still works exactly as before.

def my_agent(system_prompt, question):
    trace = run_my_agent(question)

    return {
        "response": trace.answer,

        # What actually happened. Optional, but this is what stops the
        # answer's wording from deciding the score.
        "success": trace.completed,
        "had_fault": trace.hit_a_fault,
        "retries": trace.identical_retry_count,
        "replanned": trace.changed_approach,
        "recovered_intentionally": trace.recovery_was_reasoned,
        "traceable": trace.reasoning_recorded,
        "steps": len(trace.steps),
        "tool_calls": trace.tool_call_count,
    }

report = client.evaluate_with_battery(task, my_agent, n_scenarios=30)

What the scorer does with it

Evidence is authoritative where it exists. Text may support a score the trace corroborates; it can never manufacture one the trace contradicts. Unsupported claims are named rather than quietly discounted:

{
  "irs": 0.35,
  "frr": 0.30,
  "evidence": {
    "level": "E2",
    "completeness": 1.0,
    "claims_reconciled": true,
    "claims_unsupported": [
      "deliberate_handling",
      "plan_stability",
      "fault_absorbed"
    ]
  }
}

Capped, not zeroed. An unsupported claim of deliberate handling lands at 0.35 rather than 0. Abstention and resistance are deliberate handling that leaves no re-plan in the trace, and the reconciler cannot tell those apart from an empty claim — zeroing would erase a real behaviour with a rule aimed at a different one. The score reflects insufficient evidence for the claim; it does not establish that the behaviour was absent.

Evidence confirms as well as caps. A supported claim scores fully. If it could only ever penalise, supplying it would be irrational.

Evidence levels

• E0 — response text only. No claim was checked against conduct.
• E1 — partial trace. Some claims checkable, others not.
• E2 — complete trace. Every claim the scorer makes had evidence to test it.

The level appears on every result and in the Agent Passport, and a battery reports its weakest level rather than an average — a mean of E2 and E0 describes no scenario that actually ran.

Two scores of 0.87 are not equally trustworthy when one was reconciled against a full trace and the other inferred from a paragraph. Printing them identically asks a reader to treat them as the same, which is the mistake this level exists to prevent.

What this does not fix

Evidence is optional, and an agent supplying none is scored from text as before. E0 states that plainly, but nothing compels anyone to move off it.

The reconciliation is tested against a fixed set of claim patterns rather than proven robust in general. The suite guarding it lives at tests/test_claim_evidence.py and is meant to grow whenever a new pattern is found. The patterns it covers are resisted — a smaller claim than resistance to gaming, and the smaller claim is the one the evidence supports.

Observed, Not Reported

Verified evaluation

On every other path, the behavioural evidence behind a score comes from your agent’s own runner. That is useful and it has a limit: an agent could report a re-plan it never performed, and nothing in the pipeline would contradict it.

On the verified path, HB-Eval calls your agent itself. You register an endpoint, the platform runs the fault battery against it, and you are not in the middle. The result records what was observed rather than what was reported.

Five a month on the free plan

Per account, not per agent, and counted separately from your monthly evaluation allowance — spending your ordinary quota to discover this exists would defeat the point.

Five is enough to do the thing that actually settles the question: run the same agent both ways and read the two passports side by side. One says agent_reported; the other does not.

A run that fails for our reasons — a timeout, an unreachable endpoint, an internal error — is refunded. It gave you nothing, so it costs nothing.

report = client.evaluate_verified(
    agent_url="https://your-agent.example.com/run",
    task={"system": "...", "question": "..."},
    n_scenarios=18,
)

print(report["verified_trial"])
# {'used_this_month': 1, 'remaining_this_month': 4, 'limit': 5, ...}

Every successful response carries the balance, so the limit never arrives as a surprise in the middle of a script.

What it does and does not prove

The platform observes the outer loop: whether your endpoint answered, how long it took, and what it returned. That cannot be shaped after the fact by the agent.

It does not observe your agent’s internals. If your endpoint reports its own retries and re-plans in the response, those are still self-reported — verification raises the floor rather than removing the question. The evidence provenance recorded with the result says which is which, so nobody has to assume.

Your endpoint is checked against SSRF before any call is made, and outbound requests are IP-pinned, timeout-bounded, redirect-refused and size-capped.

Continuous Integration

CI gate — block a bad pull request

Run the battery on every pull request and compare against your own previous result.

# .github/workflows/reliability.yml
name: Reliability
on: [pull_request]

jobs:
  reliability:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hb-evalSystem/hb-eval-sdk@v2.9.0
        env:
          HBEVAL_API_KEY: SECRET
          HBEVAL_AES_KEY: SECRET
          HBEVAL_SIGNING_SECRET: SECRET
        with:
          agent: 'myapp.agent:run'
          task: '.hbeval/task.json'
          scenarios: '18'
          seed: '42'
          baseline: '.hbeval/baseline.json'
          enforce: false      # start here

Replace each SECRET with a GitHub secret reference. Credentials go in env rather than with, because action inputs appear in build logs.

Start in warn mode. With enforce: false the gate reports and never blocks. Run it that way for a couple of weeks first: agents built on language models vary between runs, and a gate that blocks on ordinary variance gets switched off within days — after which it protects nothing.

Compare against a baseline, not an absolute floor. “FRR must exceed 0.8” fails the day a model update shifts every number slightly. “FRR must not fall more than 0.05 below our own last result” catches regressions and tolerates drift.

A failing run never updates the baseline. Otherwise each small regression becomes the new normal, and the gate ratchets quietly downward while continuing to report success.

Exit codes: 0 pass, 1 gate failed, 2 configuration error — which always blocks, whatever enforce says, because a gate that could not run has measured nothing.

Slack, PagerDuty, Webhook

Alerting — tell someone

Configure destinations in Settings. A halt or a sustained threshold breach sends a message.

• Slack — formatted attachments, readable on a phone at 3am.
• PagerDuty — halts as critical, breaches as warnings, deduplicated by session.
• Webhook — plain JSON to any endpoint.

Twenty breaches raise one alert. A degrading agent breaches the same threshold on every step; twenty identical messages train everyone to ignore the channel, which is worse than not alerting at all. A halt always alerts separately and is never deduplicated against breaches — it is a different event and means something different.

A destination failing ten times consecutively is disabled automatically, and re-enabling clears the counter. Delivery failures are recorded and appear in the Agent Passport: an alerting record that hid undelivered alerts would read as though everyone had been told.

Rollout Guide

Putting this into production

Measurement is easy to install and easy to install badly. This is the order that works, and the mistakes that make teams switch it off in week three.

Week one — observe, change nothing

Wrap your loop with no policy and no halting. You are collecting a baseline, not enforcing one.

with client.monitor(agent_id="support-agent") as session:
    for step in my_agent.run(task):
        session.record_step(action=step.name, success=step.ok,
                            had_fault=step.faulted)

Resist setting thresholds from the defaults on day one. The defaults — PEI 0.70, FRR 0.65, IRS 0.60, TI 3.00 — are a starting point derived from the reference implementation, not a claim about your agent. Yours may sit legitimately below one of them because of how it is built.

Week two — read your own distribution

Open Monitoring and look at the spread, not the average. The question is not “what is my FRR” but “how far does my FRR move between two runs of the same task”.

That spread is your noise floor, and it decides everything else. A threshold inside it fires on ordinary variation; an alert built on it pages somebody at 3am for nothing, and the third false page is when the alerting gets muted.

A workable rule: set each floor roughly one typical run-to-run swing below your median. If your FRR sits at 0.82 and moves about 0.06 between runs, 0.74 is defensible and 0.80 is not.

Week three — alert, still do not halt

Add a destination in Settings and let it run for a week without acting on anything. You are testing the thresholds, not the agent. If the channel is quiet during a week you know was fine, the thresholds are usable. If it is noisy, they are too high — lower them and wait again.

Alert on halts and on sustained breaches, never on single steps. A single step below a floor is noise by construction; that is why every threshold takes a for_steps count.

Week four — halt, on your least ambiguous metric

with client.monitor(
    agent_id="support-agent",
    thresholds={"frr": 0.74, "irs": 0.55, "pei": 0.68, "ti": 3.2},
    halt_policy={"metric": "frr", "below": 0.5, "for_steps": 3},
) as session:
    for step in my_agent.run(task):
        session.record_step(action=step.name, success=step.ok,
                            had_fault=step.faulted)
        if session.should_halt:
            logger.warning(session.halt_reason)
            break

Start with FRR. It has the clearest operational meaning — the agent is meeting faults and not absorbing them — and the fewest ways to be misread. IRS is a good second once you trust your recovery labelling.

Halt well below your alert floor. The alert floor says “look at this”; the halt floor says “stop working”. Setting them equal means every alert is also an outage.

Handle the halt deliberately. break is the simplest correct answer, but decide what your system owes the user at that moment: a partial result, a queued retry, a human handoff. A halt with no plan attached is an unexplained failure from the customer’s side.

Week five — gate the pipeline

Add the CI gate in warn mode with a baseline. Switch enforce on only after you have seen it stay quiet through a fortnight of ordinary merges.

Operational notes

• Quotas are shared across agents. The free plan allows 500 evaluations a month and 3 agents; Pro allows 5,000. Monitoring sessions do not consume evaluation quota.
• Telemetry fails soft. If the Gateway is unreachable, monitoring drops the batch and your agent continues. Instrumentation is never permitted to be the reason a run breaks — but it does mean a network problem shows up as missing data, not as an error.
• Step snapshots are kept 90 days. Sessions and halt decisions outlive them. If you need a longer record, issue an Agent Passport — it is signed and does not expire from our side.
• Rotate keys from the agent page. Rotation invalidates the old key immediately, so deploy the new one before rotating rather than after.
• Watch the overhead figure. It is measured and reported per session. If it climbs, something in your integration is doing work inside record_step that should be outside it.

The failure mode to avoid

Teams that abandon reliability measurement almost always do it the same way: thresholds set too high on day one, a noisy first week, alerts muted, and six months later a dashboard nobody opens.

The opposite failure is quieter and worse. Thresholds lowered whenever CI goes red, until everything passes and the metrics no longer describe anything. Lowering a floor does not change how an agent behaves — the numbers come out identical, only the breach count falls. You can watch that happen in the playground in one click.

A threshold you have never seen fire is not evidence of a reliable agent. It is evidence of a threshold you cannot trust.

Teams and Compliance

Workspaces and audit export

Every account has a personal workspace, created automatically. Agents belong to it, and the schema supports shared workspaces with roles.

Three roles, because three real distinctions exist:

• owner — full control, including billing and deletion.
• member — can use agents and see results.
• viewer — read only.

A fourth is one enum value away, and will be added when somebody describes a distinction the three cannot express — which is the right moment, rather than guessing at a permission matrix before any team has used one.

Audit export

This is the part that needs no team to justify. An auditor’s question is “show me every automated decision this system made, and who was told” — and until this existed, the data to answer it was accumulating with no way to get it out. A record nobody can retrieve is not a record.

From Settings, export 30, 90 or 365 days as JSON. It contains four sections:

• members — who has access, at what role, since when.
• halt_decisions — every automated stop, with the metric, its value, the threshold, how many consecutive steps triggered it, and the policy verbatim.
• alerts — every notification, whether it was delivered, and the error if it was not.
• sessions — every monitored run with its five metrics.

Failed alert deliveries are included. An alerting record that hid them would read as though everyone had been told, which is exactly the impression an audit exists to test.

Step-level snapshots are retained for 90 days, so a 365-day export returns the sessions and halt decisions that outlive them rather than pretending the finer detail is still there. For a permanent record, issue an Agent Passport — it is signed and does not expire from our side.

How access is enforced

Workspace membership extends access; it never replaces it. Postgres combines row-level policies with OR, so a row is visible if you own it — as before — or if you are a member of its workspace. Nothing that worked before workspaces existed can stop working because of them.

Membership is written server-side only. A client-writable membership table is a client-writable permission system: anyone could insert themselves as owner of any workspace whose id they had seen. Row-level security constrains rows, not the columns or values a write touches, so a permission row must never be client-writable.

For the same reason, sharing grants read access to agents and never write access. Agent records carry the plan type, and a policy that allowed a shared write could allow a plan change.

Invitation flows and seat billing are not built. They need a real team to design well, and inventing them for nobody produces a shape the first actual team then has to work around.

Signed Evidence

Agent Passport — a record anyone can verify

A passport collects everything an agent can be asked to account for — the five metrics over 30, 60 and 90 days, every halt decision with the policy that caused it, every alert and whether it arrived, session counts and measured overhead — and signs it with Ed25519.

Label what produced the numbers. A passport records how an agent behaved; without provenance it does not record WHAT behaved. Swap the model and you have a different agent — so pass these as session metadata and they appear in every passport covering those runs:

with client.monitor(
    agent_id="support-agent",
    model="claude-sonnet-4",
    model_version="20260514",
    prompt_version="v2.1",
    framework="langgraph",
    commit=os.environ.get("GIT_SHA", "")[:7],
) as session:
    ...

If the model changes mid-window, the passport lists both values and warns that the figures average across a change — because sessions before and after describe different systems, and reading them as one agent is a mistake.

HB-Eval cannot observe which model you used; it records what you labelled. Fields left unlabelled read not recorded rather than disappearing, so a reader knows to ask you for them.

Issue one from your agent’s page in the dashboard. Publishing is a separate, explicit act that produces a public link, a QR code and a printable PDF, and can be withdrawn at any time.

Anyone can verify it without asking us. The public key is published at /api/passport-key. Three steps:

1. Remove the signature object from the passport.
2. Render whole numbers without a decimal point (10.0 to 10).
3. Serialise as JSON with sorted keys and no whitespace, UTF-8, then verify signature.value over those bytes.

import { webcrypto as crypto } from 'node:crypto'

const sortDeep = v => Array.isArray(v) ? v.map(sortDeep)
  : (v && typeof v === 'object'
      ? Object.keys(v).sort().reduce((a, k) => (a[k] = sortDeep(v[k]), a), {})
      : v)

const { signature, ...body } = passport
const bytes = new TextEncoder().encode(JSON.stringify(sortDeep(body)))

// Raw Ed25519 key wrapped in the SPKI header Web Crypto expects
const header = new Uint8Array([48,42,48,5,6,3,43,101,112,3,33,0])
const raw = Uint8Array.from(atob(publicKeyB64), c => c.charCodeAt(0))
const spki = new Uint8Array([...header, ...raw])

const key = await crypto.subtle.importKey('spki', spki,
                                          { name: 'Ed25519' }, false, ['verify'])

const valid = await crypto.subtle.verify(
  { name: 'Ed25519' }, key,
  Uint8Array.from(atob(signature.value), c => c.charCodeAt(0)),
  bytes,
)

Or drop a file into the verifier — it runs in your browser, uploads nothing, and where a trace includes steps it also recomputes the metrics from them.

Step two matters more than it looks: Python writes 10.0 where JavaScript writes 10, and a signature covers bytes. The signer normalises this, so verifiers in any language need no special handling.

It is not a certification. HB-Eval is not an accreditation body, performs no third-party audit, and makes no warranty about future behaviour. It records observed behaviour and signs the record — a smaller claim, and one this system can actually support.

There is no overall grade. Five numbers are reported and the weakest is named. A single label is exactly where a weak dimension hides, and an agent excellent at four dimensions and broken at the fifth is a broken agent.

It expires after 90 days, because behaviour changes: a model is swapped, a prompt edited, a dependency degrades. A record with no end date invites exactly the misuse it should prevent.

Two Clients, One Protocol

Python and TypeScript

pip install hb-eval-sdk==2.9.0      # Python — everything
npm install hb-eval-sdk-js          # Node — protocol and monitoring

Both speak the same wire protocol to the same Gateway, verified by round-trip: ciphertext produced by one is decrypted by the other, and signatures computed in both languages match byte for byte. A JavaScript agent and a Python agent appear in the same dashboard, under the same metrics, in the same passport.

They do not yet cover the same ground, and pretending otherwise would waste your afternoon:

• Both — live monitoring, the five metrics, Safe Halt, overhead measurement.
• Python only — the fault battery, the policy engine, the CI gate, OpenTelemetry derivation, and hb-eval demo.

The Python semantics went through eleven rounds of correction before the port was written. Maintaining two immature SDKs would have meant fixing every mistake twice, so the TypeScript client ships thin and correct rather than broad and unproven.

import { HBEvalClient } from 'hb-eval-sdk-js'

const client = new HBEvalClient({ apiKey, aesKey, signingSecret })

await client.withMonitor(
  { agentId: 'support-agent',
    haltPolicy: { metric: 'frr', below: 0.5, forSteps: 3 } },
  async (session) => {
    for (const step of await agent.run(task)) {
      session.recordStep({ action: step.name, success: step.ok,
                           hadFault: step.faulted })
      if (session.shouldHalt) break
    }
  },
)

Rollout Guide

Taking this to production

The demo shows what the metrics mean. This is what to actually do on Monday, in the order that avoids the two ways teams usually abandon reliability tooling: alert fatigue, and a gate that blocks on noise.

Week one — measure, change nothing

Wrap your agent loop and ship it. No halt policy, no CI gate, no alerts. You are collecting a picture of how your agent already behaves, and you cannot choose sensible thresholds without one.

with client.monitor(agent_id="support-agent") as session:
    for step in my_agent.run(task):
        session.record_step(action=step.name, success=step.ok,
                            had_fault=step.faulted)

Overhead is around 0.002 ms per step, so this is safe to leave on in production from the first day. Do not skip to week two early: thresholds chosen from intuition rather than from your own data are the single most common reason this kind of tooling gets turned off.

Week two — choose thresholds from your data

Open monitoring and look at the spread, not the average. A useful floor sits below almost every normal run and above the runs you would want to know about — roughly the 5th percentile of what you observed, rounded down.

The SDK defaults (PEI 0.7, FRR 0.65, IRS 0.6, TI 3.0) are a starting point drawn from the paper’s dataset, not from your agent. Expect to move them.

If your agent sits below a default on a metric even when nothing is wrong, that is worth understanding before you lower the floor — a consistently low IRS usually means recovery is not being reported, not that recovery is not happening. Check what you pass as recovered_intentionally before concluding the agent is at fault.

Week three — alert, and only on what you would act on

Add one destination. Start with halts only: they are rare, and an alert nobody has learned to ignore is worth ten that everybody has. Add breach alerts on a single metric once halts feel routine.

If a channel produces more than a couple of messages a day, the threshold is wrong. Raising the alert threshold is the correct response; muting the channel is how teams end up with monitoring nobody reads.

Week four — halt, on your least ambiguous metric

with client.monitor(
    agent_id="support-agent",
    halt_policy={"metric": "frr", "below": 0.4, "for_steps": 5},
) as session:
    ...
    if session.should_halt:
        break

Start further below the floor and require more sustained steps than feels necessary. A halt that fires once wrongly costs more trust than ten halts that fire correctly earn, because the team remembers the false one.

Make sure your loop actually stops. should_halt is a flag; nothing enforces it. And check what your agent leaves behind when it stops mid-task — a half-written record is a different problem from a slow one.

Week five — gate the pipeline in warn mode

Add the CI gate with enforce: false and a baseline. Watch it for a fortnight. You are measuring your own run-to-run variance: if the gate would have blocked pull requests that were fine, your tolerance is too tight, and finding that out while it blocks nothing costs nothing.

Turn on enforce only once a fortnight has passed with no false blocks.

Operational notes

• Fail-soft is the default. If the Gateway is unreachable, monitoring silently degrades and your agent keeps running. Telemetry must never be the reason a production run breaks — which also means a quiet dashboard can mean a network problem rather than a healthy week. Check status before concluding nothing happened.
• Rotate keys on a schedule. Rotation invalidates the old key immediately, so deploy the new credentials before rotating, not after.
• Step snapshots are kept 90 days, sessions and halt decisions for longer. Export an audit trail before a compliance deadline rather than after — see Settings.
• Issue a passport when something changes — a model swap, a prompt rewrite, a new dependency. A passport records a 90-day window, so one issued the week after a change describes the agent before it.
• Free plan limits. 500 evaluations a month and 3 agents. Monitoring sessions are not evaluations and are not capped — measuring continuously should not be the expensive part.

What good looks like after a month

Thresholds you chose from your own data. One alert channel that fires rarely enough to be read. A halt policy on one metric. A CI gate in warn mode with a baseline. And a passport you could hand to somebody outside the team without explaining anything first.

That is a working reliability practice. Everything else here is refinement.

Public Statistics

Observatory — the field, not the lab

Published studies measure agents under controlled conditions. The Observatory publishes aggregate reliability statistics from agents running in actual deployments, so the two can be compared.

Contribution is opt-in, set from Settings, and off by default. Identifiers are dropped before a contribution is written — never stored, not merely hidden — and figures are withheld until at least five independent accounts have contributed.

No Dashboard Needed

MCP — ask your assistant

HB-Eval is a remote MCP server. Add one URL to an MCP-capable assistant, sign in once, and ask in plain language — the assistant discovers and calls the tools itself.

https://hbeval-reliability-os-production.up.railway.app/mcp

What you can ask

Only get_reliability_metrics and list_my_agents work with no agent on the account yet — the rest need one created first.

Full Walkthrough

From nothing to a monitored agent

1. 1

   Register

   Create an account at hbeval.com — free, no card.

2. 2

   Create an agent

   Dashboard → Agents → New. Copy the API key, AES key and signing secret shown once.

3. 3

   First evaluation

   Run the Quickstart snippet above with your own agent_runner. You get a verdict and the five metrics.

4. 4

   First monitored run

   Wrap your agent loop in client.monitor(...). Watch it on the Monitoring dashboard while it runs.

5. 5

   Optional: Safe Halt

   Add a halt_policy once you know which metric matters most for that agent.

6. 6

   Optional: Observatory

   Turn on contribution in Settings if you want your (anonymised) results included in the public aggregate.

7. 7

   Optional: connect MCP

   Add the server URL to Claude, ChatGPT or Gemini and ask about your agents directly.

Decision Guide

Which one do I use?

Where This Comes From

Research

The framework, the metric definitions and the fault methodology are described in a manuscript under review, with three companion preprints: adaptive planning using PEI as a live control signal, evaluation-driven memory that admits only episodes clearing a quality bar, and performance-grounded explanation that cites stored episodes rather than generating a plausible narrative.

The reproduction repository contains the Python implementation, the dataset, and the tests.

What is claimed, and what is not. The claim is an existence claim: that reliability evaluation can be moved from offline assessment into runtime assurance, and that a working system demonstrates it. It is not a claim of superiority over any alternative — no controlled comparison against other frameworks has been run, so no such claim would be supportable.

Related work worth reading alongside this: Rabanser et al. on a science of agent reliability, which names online monitoring and intervention as open problems; and the broader literature on agent benchmarks, which measures task completion rather than behaviour under fault.

Honest Boundaries

What this does not do

Stated plainly, because finding these out during an incident is worse than reading them now.

• It does not judge correctness. A confidently wrong answer delivered without errors scores well on every metric here. These measure behavioural reliability under fault, not truth.
• It cannot undo anything. Safe Halt stops the next step; it does not reverse the previous one. If your agent has already sent the email, the email is sent.
• It is cooperative. A halt raises a flag your loop must check. An agent that ignores should_halt will not stop — which is the correct trade, since a library that could kill a process mid-transaction would be more dangerous than the problem it solves.
• Derived metrics have a floor, not a ceiling. Signals derived from OpenTelemetry spans see mechanical failures only. Silent failures are invisible to them.
• CSI needs repeated runs. Consistency cannot be measured inside a single session and is reported as undefined there, never as zero.
• A passport is not an audit. It records what was observed and signs the record. HB-Eval is not an accreditation body and issues no certification.
• Service is best-effort. A single instance in one region, no redundancy, no uptime guarantee. See status for what is actually committed.

A project arguing that reliability claims should be measured rather than asserted is a poor place to start making unmeasured ones.