Stas Katkov published a detailed look at how GitHits behaves inside his agent setup. He intercepted every call between his harness and our MCP server, benchmarked us against plain web fetching, and published the full trace.

It is amazing to get feedback at this level of detail, even when it isn’t what you hoped to read. We are a small team working on a large product surface, and it is easy to miss things. This one post surfaced a use case where GitHits does not yet provide value, showed us the user-facing cost of an indexing bug whose fix we were already landing, and checked what data GitHits received during the session. Thank you, Stas.

The short version

Stas asked his agent what changed in the last two releases of his rdoc-markdown gem. Through GitHits MCP, the session took 9.5 minutes and 91 tool calls. Through plain web fetching, it took about a minute. Both answers were correct. He removed GitHits and wrote up why.

We analyzed his published trace. The findings, condensed:

GitHits calls were in flight for 166 of the 591 seconds. During the remaining time, no GitHits call was active. The large number of calls came from a gap in our tool surface.
His session hit an indexing bug that made first-time indexing considerably slower. We had found it the week before and were already fixing it. The fix is now deployed and would have saved about a minute in this session.
The agent explicitly requested JSON output in 86 of the 91 calls, overriding our compact default. This increased the amount of context returned to the model, and we want to understand why the harness made that choice.
The trace shows what GitHits received: public repository references, tags, file paths and search patterns. No local workspace files or private source were sent.

The rest of this post looks at what happened and what we will change due to this feedback.

What actually happened in those 9.5 minutes

His gem had no useful changelog entries for these releases, and the packaged gem excludes the changelog file. The agent’s first changelog lookup therefore returned nothing. GitHits currently has no direct way to compare two repository versions, so the agent reconstructed the diff using code-navigation tools. A child agent listed files and ran 42 reads and 35 greps across three tags. Those reads and greps accounted for 77 of the 91 calls. Most of the remaining session time was spent in the model reasoning between calls.

Our MCP instructions recommend delegating multi-call navigation work to a child agent, and the harness followed that guidance. The agent produced the correct answer after doing far more work than the question required. GitHub’s comparison page provided the information directly, and Stas is right that fetching it was the better option for this repository and question.

Three calls requested versions we had never indexed. Two waited about 30 seconds and returned an indexing-in-progress result; one completed after just under a minute. This was slower than our normal first-index time because of a bug we had discovered the week before the post. When preparing a new version, the indexer read old indexes too aggressively and performed more work than necessary. The fix was already in progress and is now deployed. Based on the trace, the bug added about one minute to the session.

The 1.3 million recorded tokens cover the complete agent session, including repeated model context and more than one million cache-read tokens across 22 turns. They are OpenCode’s token accounting for the session rather than data transferred to GitHits. The 91 tool calls still created substantial context use. GitHits MCP defaults to compact text, but the agent explicitly requested JSON in 86 calls. We are investigating why the harness selected JSON so consistently and whether we should change our tool descriptions, defaults or both.

The CLI run

Stas also drove our CLI directly, without MCP. It used the same back-end and produced the correct answer. The agent spent a significant share of its calls reading help output and discovering the command syntax.

The CLI was designed primarily for people working in terminals, and this becomes apparent when an agent uses it. An agent should not need several --help calls to discover command groups, target syntax, output formats and follow-up commands.

The CLI can also return more data than an agent needs. Code reads are uncapped, JSON responses can be larger than the compact MCP format, and there is no field-selection option for narrowing structured output. These are areas where we need to make the interface more suitable for agent use.

The interesting part for us is that we shipped CLI-backed Skills for this workflow. They explain when and how to run GitHits commands and reduce the need for help and discovery calls. Stas ran the CLI as our installer left it, without those Skills being surfaced when they would have helped.

We need to make the Skills easier to discover and improve the base CLI at the same time. The work includes more consistent command syntax, compact machine-readable output, control over returned fields, sensible limits and responses that make the next operation clear.

The part we should have answered weeks ago

Before running this experiment, Stas had asked for visibility into what happens between his harness and our back-end. We acknowledged the request without offering a way to inspect those exchanges.

The GitHits CLI and local MCP server are open source, but reading their source does not provide an activity history for a particular session. Stas had to add an MCP proxy to collect that information himself. The missing visibility was clearly part of his frustration with GitHits.

Stas is right, you should not need to add a proxy to your harness to see what a tool does on your behalf. We had started planning an activity view before the post. The trace clarified what the view needs to contain: individual GitHits tool calls, their arguments, status, duration and responses.

The activity view should make this information available without additional tooling. Aggregate usage counts would not answer the original question.

The current diagnostic options are more limited. A stdio proxy such as mcpsnoop records the complete exchange between the harness and MCP server. GITHITS_TELEMETRY=1 prints local timing information, and search_status exposes indexing progress. We should have shared these options when the request was made.

What we are changing

Add a direct version-comparison capability for repositories without useful changelogs or release notes.
Investigate why the harness requested JSON in 86 of 91 calls and improve the guidance around compact output.
Review the delegation guidance that led to the large child-agent investigation.
Make the CLI more agent-friendly, including compact output, field selection, clearer discovery and sensible limits.
Make the CLI Skills easier to discover and install.
Continue the planned activity view with the request and response visibility identified in this session.
The indexing-reuse fix is deployed.

It is never particularly comfortable to read that someone removed your product after testing it carefully. The useful response is to understand what happened and make the changes the evidence supports. This post gave us enough detail to do that. If you have run your own experiments on GitHits, good or bad, we want to see them.

We appreciate brutally honest feedback. Thanks, Stas, for all the effort you’ve put in testing GitHits and reporting your findings.