Open model accuracy is a harness problem

Open model accuracy can change without changing the model.

In their own Hermes Agent A/B test, a hosted Qwen-family model solved the same file task in both arms. With a file-type guard in the read tool, it used 26,000 tokens. Without the guard, it used 122,000 and its slowest run took more than ten minutes.

Same weights. Same prompt. Same answer. Different harness.

Qwen3.8-Max on the named-pipe task	Guard off	Guard on
Mean tokens	122,000	26,000
Mean model turns	9.3	5.0
Worst wall clock	618 seconds	115 seconds
Task score	1.00	1.00

Those are the repo’s own read-tool results, measured over three repetitions of the same file-only prompt. The guard did not raise the score. It removed the recovery spiral required to reach it.

That distinction matters once a model can act. Answer accuracy is whether the final answer is right. Agent reliability also asks whether the model selected a real tool, encoded its arguments correctly, received a useful result, kept the user’s request in view, and finished before its turn, token, or time budget ran out.

Hermes Agent cannot give a smaller model reasoning it does not have. What it can do is stop correct reasoning from being lost at the boundaries around the model.

Accuracy changes when the model can act

A language model produces tokens. An agent has to turn some of those tokens into side effects.

The harness is the code that makes that translation: it builds the prompt, decides which tools to expose, describes their arguments, parses calls, schedules them, runs them, returns observations, manages context, retries failures, and decides when a turn is over. It is the system layer behind the AI product.

Every arrow in that lifecycle is a place where a capable model can still fail.

✕

The last line is the important one. A frontier model may recover from a vague error in one extra turn. A smaller model may repeat the call, copy the error, try a nearby tool, and fill its context with failed attempts. Preventing the first avoidable failure therefore helps the smaller model more, even when both models eventually reach the same answer.

Turn one response into a work loop

That protection also changes what “answering” means. Hermes Agent keeps the model in a bounded loop: propose an action, run it, observe the result, and decide what comes next. A tool call advances the turn; completion comes later.

The system prompt reinforces the loop. It tells the model that a plan, stub, or single command is not a completed task; a result should be backed by real tool output. When a partial approach fails, the model should inspect the failure, check prerequisites, retrieve missing context, and try another route instead of guessing. Some of this guidance is applied more forcefully to model families including Qwen, DeepSeek, GLM, and Gemma.

Hermes also grounds that behavior in the workspace. It discovers project instruction files, detects manifests and package managers, and surfaces likely test, lint, build, and start commands. The smaller model is doing an open-book job rather than reconstructing the repository from memory.

These are still instructions, so they can be ignored. The optional coding stop guard goes further: after source edits, it can notice that fresh verification evidence is missing and give the model a bounded follow-up before accepting “done.” It does not run the checks itself, and it ships off by default, so the honest claim is that verification discipline is always prompted while mechanical enforcement is configurable.

Make the right action easier to express

That prevention starts before the model generates anything. Hermes Agent does not treat the tool registry as a static menu.

Tools can declare availability checks. The request only includes the definitions that pass for this session: configured services, granted permissions, enabled toolsets, and reachable capabilities. Nested schemas are then rebuilt from that real set. If web search is unavailable, a code-execution tool does not advertise it as an allowed nested action. If a browser description recommends a web tool that did not load, that recommendation is removed.

This sounds like housekeeping. It is accuracy work because every tool name in the prompt is a candidate the model may emit. A description that says “prefer web search” while web search is absent creates a failure the dispatcher can only reject.

The same principle handles very large plugin and MCP catalogs. When their schemas consume too much of the active context window, Hermes Agent can replace the full non-core catalog with search, describe, and call bridges. The model discovers a relevant capability before seeing its complete schema. The core tools stay visible, so this reduces choice pressure rather than eliminating it.

The fair reading of the broad core toolbox is that an always-available personal agent needs capabilities it can reach without a discovery round trip. That choice also leaves a substantial schema floor for smaller contexts. Progressive disclosure is therefore a partial answer, not a universal one, and the larger architecture study covers that cost in detail.

Provider compatibility is the other half of expression. Cloud APIs, strict provider subsets, and local llama.cpp servers do not accept exactly the same JSON Schema. Hermes Agent sanitizes malformed or unsupported shapes, gives top-level tool inputs a valid object form, rewrites illegal property keys, and applies provider-specific rules where required. If llama.cpp rejects regular-expression hints while compiling a decoding grammar, the harness can strip those hints and retry once.

One boundary is worth stating: “smaller” here refers to fewer parameters or lower inference cost. The repo’s own startup checks generally require at least 64,000 context tokens for tool workflows. The harness reduces pressure inside that window; it does not pretend a very short window can hold a long agent task.

None of this makes a wrong tool choice right. It makes the valid choices visible and expressible on the backend actually serving the model, without spending an extra model turn or adding user-visible latency.

Repair representation, never invent intent

Once a call exists, the hard question is how helpful the harness should be. I went in expecting one strict rule: malformed model output should always be refused. The code has a better distinction, but it took me a while to read because transport repair and live execution happen at different moments.

Old tool calls already stored in conversation history may need to be sent to a new provider on the next turn. Some local models leave raw control characters, trailing commas, or unclosed braces in that history. A strict API can reject the entire replay before the model gets another chance. Hermes Agent normalizes common syntax damage in that historical wire representation so the conversation can continue.

A newly proposed action meets a stricter boundary. Its arguments must parse into a JSON object. If they do not, the tool is not executed. The model receives a compact structured error, and a valid sibling call in the same batch may still proceed.

After the object is valid, the registered schema authorizes a narrow set of type corrections:

value "42"       + schema integer  -> 42
value "true"     + schema boolean  -> true
value "https://" + schema array    -> ["https://"]
value '["a","b"]' + schema array  -> ["a", "b"]

the same JSON-looking value + schema string -> unchanged
failed conversion                         -> original value

The harness also walks nested arrays and objects, because open-model incidents documented in the code include lists whose elements are themselves JSON-encoded strings. Parsing happens only where the corresponding schema position expects structure.

This is not permission to “fix” any plausible call. A schema may be incomplete, and wrapping a bare value still changes representation. The boundary is narrower: use declared types to remove encoding ambiguity, refuse anything that would require guessing the intended action, target, or missing content. That choice costs nothing on valid calls and avoids a whole retry cycle on recoverable ones.

Build tools that return a next move

That clean contract still depends on the tool it enters. The named pipe from the opening is the cleanest example.

A named pipe looks like a file in a directory listing. Reading it with no writer does not return empty or raise a useful error; it waits. Hermes Agent checks the resolved file type before reading on host-visible filesystems and refuses FIFOs, sockets, and device files with a plain-language explanation. The Qwen-family model no longer spends turns diagnosing a tool that appears frozen, and the user no longer waits through the same wall-clock stall.

The read path also has separate line and character budgets. If a large file or one enormous minified line crosses the character limit, the tool keeps complete lines where possible and returns the next offset to read. Binary and structured documents are identified before raw text handling. A failure becomes “continue here” or “use the appropriate tool,” not a blank result that forces the model to guess.

That pattern repeats across the tool layer. An edit that is already present reports that no write was needed. An ambiguous patch reports match locations. A case-sensitive search miss can surface evidence of case-insensitive matches. Large terminal output can be preserved outside the immediate result instead of disappearing in the truncated middle. These are small messages, but they turn a dead end into a constrained next decision.

Hermes Agent also segments batches around dependencies. Independent reads may overlap, while a read whose path conflicts with an earlier write stays behind that write. The scheduler preserves the model’s order across side-effect barriers, so speed does not create a race that changes the meaning of the plan. Parallel work lowers latency only where the calls commute.

The full tool layer is mapped in the component-by-component Hermes study. Its recurring design is simple: do deterministic work in code, then give the model the smallest useful observation about what remains.

Recover without teaching the failure

Some failures cannot be prevented because the response itself is incomplete. Hermes Agent diagnoses a truncated tool call and refuses to treat it as an action.

It discards the broken call, repeats the clean request, doubles the output allowance on successive attempts up to a cap, and stops after four retries. If the arguments are still incomplete, the tool does not run.

truncated tool call
  -> discard partial call
  -> retry same clean context with more output room
  -> at most four attempts
  -> still incomplete: refuse execution

Discarding the fragment matters because a malformed example in recent context is a strong pattern to copy. The next attempt sees the valid schema without the broken imitation.

The same restraint applies to a blank tool name after the model reads tool-call syntax as data. A misspelling gets the valid catalog; a blank name gets one short reminder not to copy the syntax. The full catalog would feed the imitation loop.

Those retries add latency and paid prefills. Bounded recovery buys a clean failure state; it does not make the attempts free.

Keep observations from becoming instructions

But failure can also arrive from the opposite direction. Web, browser, and external-server tools return text written by someone else, including instructions designed to pull the agent away from the user’s task.

Hermes Agent wraps longer results from these sources in a delimiter that marks them as external data. Embedded copies are neutralized case-insensitively, so a poisoned page cannot close the block early. The content stays readable while its authority changes.

This remains a prompt-level defense, not a mathematical guarantee. I could not verify a model-level A/B result for it in the checked-out evals. The deterministic claim is that source and trust are labeled consistently and a forged closing tag cannot escape that label.

Long sessions create a related problem: summarization can blur the instruction that started the work. The default compressor protects a recent tail and keeps at least the latest real user message there verbatim. Synthetic user-role scaffolding does not count as that anchor.

Compression loses detail and can add a noticeable stall. Hermes Agent calibrates its trigger from provider-reported usage and protects user intent across the cut; the companion architecture analysis covers the full trade.

Measure the recovery work, not only the answer

That brings the opening result into focus. A score of 1.00 in both arms says the special-file guard did not break the task and that the fixture’s outcome metric was saturated. It does not say the 122,000-token path was equally reliable in a deployment with a lower token cap, a shorter timeout, or impatient users.

The repo’s second A/B harness tests that broader recovery surface. It creates nine tasks that deliberately trigger production waste patterns: interpreter confusion, an already-applied patch, an ambiguous edit, search casing, hidden files, truncated output, directory changes, a blocked inline script, and a paginated large file. Success is checked from traces and filesystem state rather than the model’s own claim.

Across their own 108-run August batch, the maintainers report that the fixes cut model turns by 21 percent, tool calls by 29 percent, and wall clock by 23 percent on a Qwen3-Coder 30B checkpoint, while tool errors fell to zero. Claude Sonnet stayed at parity. The setup and caveats live with the core-toolset A/B harness.

The numbers need boundaries. Each cell used three repetitions. The tasks were designed to fire specific failure modes rather than sampled from ordinary user traffic. The first test used hosted Qwen3.8-Max, an open-family model rather than a local checkpoint. The second used an open checkpoint through a hosted route. Neither experiment proves the same gains for every small model, every backend, or a task where the model’s plan is simply wrong.

What they do show is a useful evaluation method: hold the model and prompt constant, swap the harness code, grade completed state mechanically, and count the turns, errors, tokens, and wall-clock latency required to get there. Weak models are a useful signal because their recovery budget is smaller. Frontier-model parity is expected when the stronger model already absorbs the induced error in one turn.

The next useful test would impose realistic budgets and sample ordinary work rather than traps. That would measure the claim users care about directly: how often does each arm finish correctly before it exhausts the resources a deployment will actually allow?

Hermes Agent optimizes for completed work with less recovery waste, not benchmark accuracy in the abstract. That is a narrower claim than “the harness makes small models smart,” and a much more actionable one.

The accuracy checklist

This checklist is the practical result. Ask five questions of any agent harness:

• Does the model see only tools and actions that can actually run in this session?
• Are formatting mistakes corrected only where a schema removes ambiguity?
• Can malformed or incomplete calls reach side effects?
• Does every tool failure return a bounded, useful next move?
• Can the latest real user instruction survive tool noise, retries, and compaction?

One “no” spends scarce model capacity on recovery. Hermes Agent still cannot rescue a wrong plan, a hallucinated fact, or a valid-looking call aimed at the wrong file. It can keep observable system failures from burying useful reasoning before the task is finished.

I’m Kaushal Prajapati, a Staff AI/ML Engineer. More posts, or say hi on LinkedIn.