Reliability Lab

From its catalog listing
Architecture-aware reliability testing for agent repositories: discover real usage datasets, generate scenarios, review them independently, execute configured targets and inspect evidence in Hermes web and Desktop dashboards. Freeze audited benchmarks and compare regressions across code changes, with baseline references for native Hermes memory.
Good to know before installing
Sends source excerpts from the repository you point it at and sampled public dataset messages to your active Hermes model, queries huggingface.co and zenodo.org (HF_TOKEN only to Hugging Face), clones the supplied HTTPS repositories into profile plugin data, and starts a detached background worker per campaign; execution is off by default, and when enabled and approved through Hermes' prompt it runs generated evaluator tests, model-authored reference actions and the target agent (a cloned Hermes) on this host outside any sandbox, sending the target API key to your configured target base_url, while OpenHands/DeerFlow targets run in a network-disabled Docker image you build.
From its catalog listing.
How to install
- Install it. Hermes asks for any API keys it needs (this one needs
HF_TOKEN,RELIABILITY_TARGET_API_KEY), then asks whether to turn it on.hermes plugins install hermes-reliability - Said no? Turn it on later:
hermes plugins enable hermes-reliability - Restart Hermes so it loads the plugin.
Some plugins also need a sign-in or settings: see Full setup & settings. Memory plugins ask whether to become your memory provider instead. Steps follow the official plugins guide (opens in a new tab).
- Category
- Memory & notes
- Maintained by
- MadBonzz (Community)
- Needs
HF_TOKENRELIABILITY_TARGET_API_KEYAPI keys- Adds tools
reliability_startreliability_statusreliability_stopreliability_evidencereliability_historyreliability_freezereliability_comparereliability_memory- Adds hooks
pre_tool_call- Works on
- linux
- Needs Hermes
- 0.21.5 or newer
- Version
- 0.2.0
- In the catalog
- 2026-11-11
Known issues (from its catalog listing)
- The Hermes target adapter uses version-sensitive runtime interfaces in separate evaluation workers; compatibility warnings and tested revisions are documented.
- End-to-end workflows are verified with deterministic models. Live Command A target smoke and Muse scenario generation/preflight passed; a complete autonomous live campaign remains unverified.
Using the Hermes desktop app? See where plugins live

Full setup & settings ↗Source code ↗
More in memory & notes
New Hermes Agent updates every day on X. Follow @HermesWatcher