A model name is only one setting
The selected model reasons about the task; its provider supplies access; enabled tools perform operations outside the conversation. An otherwise capable model cannot fetch an inbox merely because its provider authentication succeeds. Treat these as separate parts of the configuration and check which part a failure belongs to.
The official configuration guide documents resolved-value inspection through hermes config get and validation through hermes config check. Use those diagnostics before changing several settings at once. An invocation override, another profile or a running service with old settings can make a saved file an incomplete description of what actually executed.
Compare completed work
Use one bounded task with a known answer or inspectable artifact. A useful model trial might retrieve five issues, identify duplicates and cite each conclusion. Record whether it used the required tools, handled missing data, followed the output format and finished without repair. Measure total effort across retries, not just the first response time.
Keep the source window and permissions stable between trials. For research, inspect whether a model separates collection dates from publication dates. For automation, test a no-change case and an unavailable-source case. Choose the least expensive configuration that reliably meets the actual requirements; there is no universal best model for every workflow.
Scope and limits
This is a comparison method, not a current model ranking or a price estimate. Provider offerings, quotas and supported identifiers change. Consult the relevant provider's current terms before selecting a paid option. Redact credentials when retaining diagnostic output, and record the effective provider and model with the test result so a future comparison remains meaningful.
Primary sources
Spotted something wrong or out of date? Tell us on X at @HermesWatcher (opens in a new tab) and we will fix it.
New Hermes Agent updates every day on X. Follow @HermesWatcher