Model Drift Watch
Scheduled check that the model you use in Hermes has not quietly got worse: 34 fixed, exactly graded questions (reasoning, instruction following, Japanese, plus your own) run through Hermes cron, each run tested against the same model's own baseline with an exact per-question test and a same-day confirmation run, and a message only for a confirmed drop, a changed served model or a failed measurement. Not yet tried against a real LLM API (checked with a local mock model and real Hermes cron).
Good to know before installing
Each check calls your own model through ctx.llm on your bill, capped per check by the plugin (default 120,000 tokens / $0.50, no daily cap; the agent can start checks itself via the model_drift tool, and the scheduled agent turn is extra); price lookup goes through Hermes's pricing code (may fetch OpenRouter/models.dev prices); 10 optional code questions, off by default, run model-written Python in a limited child process (not a hostile-code sandbox) including in unattended cron runs when enabled; `schedule` creates one Hermes cron job that survives plugin removal (run `unschedule` first); writes run history to plugin-data/model-drift-watch; no telemetry.
From the plugin's catalog listing.
How to install
- Install it. Hermes asks for any API keys it needs, then asks whether to turn it on.
hermes plugins install model-drift-watch - Said no? Turn it on later:
hermes plugins enable model-drift-watch - Restart Hermes so it loads the plugin.
Some plugins also need a sign-in or settings: see Full setup & settings. Memory plugins ask whether to become your memory provider instead. Steps follow the official plugins guide (opens in a new tab).
- Category
- Languages & regions · Web, research & browsing · Automation & multi-agent
- Maintained by
- TakeshiTGAL (Community)
- Needs
- No API keys declared
- Adds tools
model_drift- Needs Hermes
- 0.21.5 or newer
- Version
- 0.1.0
- In the catalog
- 2026-10-04
Full setup & settings ↗Source code ↗
More in languages & regions
All languages & regions plugins
New Hermes Agent updates every day on X. Follow @HermesWatcher