[Canonical HTML page](https://githits.com/blog/computer-use-agents-app-source-hidden-rules/)

[Back to blog](/blog.md)

![A form field in a browser window linked to the line of application source code that defines its limit.](/_astro/cover.BQmTDNVa_Zo72kk.webp)

November 18, 2026 · 7 min read

# Computer-use agents pass more hidden-rule tasks when they can look up the app's source

A browser-driving agent ran GitLab and Magento tasks with rules the interface doesn't show, with and without GitHits lookups of the app's source.

An agent working in GitLab’s web interface is asked to add a masked CI/CD variable, a setting that hides a secret value in job logs. It types a value, clicks save, and gets a vague error. The form never says what a maskable value has to look like. The rule exists, but it lives in GitLab’s source code, in `app/models/concerns/ci/maskable.rb`, which sets a minimum length and a limited set of allowed characters.

Magento’s admin panel has the same kind of gap. The form for a new website shows no limit for the website code, but the database column defined in Magento’s `db_schema.xml` holds 32 characters. Other fields cut input off silently or change a value after saving it. A person learns these rules by trial and error. An agent that only sees the screen has to guess.

A computer-use agent is a model that operates software through its user interface, much as a person would. We tested whether such an agent does better on tasks like these when it can also look up the application’s source code, by running the same agent with and without GitHits. GitHits indexes open-source code and documentation so agents can search and read it.

## How we set it up

We used the self-hosted GitLab 15.7.5 and Magento 2.4.6 environments from WebArena, a public benchmark built from real web applications. Open-weight models, whose trained parameters (weights) are published, drove the browser: at each step the model saw a screenshot and clicked or typed into page elements by ID.

Each task ran twice under each of two conditions, with the same model and prompt. In one, the agent had only the browser. In the other, it could also look up the app’s source and documentation through GitHits, using search, grep (exact text matching) and read on the GitHits index of the same GitLab and Magento releases. Most tasks targeted a hidden rule: behavior the interface doesn’t show or misstates, but the source defines. The rest were controls, ordinary tasks whose answer is on screen.

We counted a task as solved only if both of its runs passed, then compared conditions with McNemar’s test. It looks only at tasks where one condition succeeded and the other didn’t, and asks how likely a split at least that lopsided would be if both conditions were equally good. A p-value of 0.01 means a split like that would happen by chance about once in a hundred tries.

## A blind task suite

We wrote the first tasks ourselves, which raises a fair question: did we pick rules GitHits happens to find? So an independent writer with no knowledge of any results wrote 55 task specs. The writer looked up some Magento rules through GitHits; GitLab rules came from the public source. A scripted reference solution confirmed that each task could be done through the interface and checked correctly, and 43 passed: 31 hidden-rule tasks and 12 controls. We froze that set and recorded its sha256 hash, a fingerprint that changes if the file does, before any model run.

On the blind set, with Qwen3.8-27B, the browser-only agent passed 32 of 86 runs and the agent with GitHits passed 51 of 86. GitHits solved 22 tasks in both runs and browser-only solved 14 (p=0.008). On the hidden-rule tasks alone, the split was 14 to 8 (p=0.03). Pairing individual runs of the same task, GitHits succeeded where browser-only failed 20 times, and the reverse happened once. The agent with GitHits called it in 82 of its 86 runs.

We also reran our original 30 hidden-rule tasks alongside the blind set as a holdout, a second check run at the same time with the same model. Browser-only passed 28 of 60 runs and GitHits 42 of 60, with 19 tasks solved in both runs against 10 (p=0.004). Across all 73 tasks, the totals were 60 of 146 runs against 93 of 146, and 41 tasks solved in both runs against 24 (p<0.001). Browser-only never solved a task in both runs that the GitHits agent missed.

![Share of runs passed with the browser only and with GitHits source lookup, for the blind suite, the holdout, three model families on the original 30 hidden-rule tasks, and two task sets with no significant difference.](/_astro/success-by-suite.oqXVGmK-_1zBn3y.webp)

## Other model families

We also ran the original 30 hidden-rule tasks with two more model families. In an earlier run on those tasks, Qwen3.8-27B went from 24 of 60 runs to 44 of 60, with 20 tasks solved in both runs against 8 (p=0.002). Nemotron 3.5 Super VL 120B went from 20 of 60 runs to 33 of 60, with 15 tasks against 7 (p=0.021). Muse Glimmer-30B started from a much lower floor and went from 10 of 55 scored runs to 21 of 57; a spend cap and infrastructure errors left some of its runs unscored. Pairing individual runs, GitHits succeeded where browser-only failed 12 times, and the reverse happened twice (p=0.013), but few tasks passed in both runs under either condition, so the both-runs count shows no clear difference for Muse.

## Where it didn’t help

The gain was limited to hidden-rule tasks. Our first 19 custom tasks, written so the source would be relevant and run once each, came out flat: browser-only passed 12 of 19 runs and GitHits 11 of 17 scored runs, after excluding runs cut off by shutdown or a harness error. Tasks that asked where a setting lives in the interface showed no gain either.

On long, multi-step forms in GitLab and Magento, GitHits passed 11 of 22 runs against 9 of 22, which isn’t significant. The hardest Magento forms failed in both conditions because the agent couldn’t operate their complex widgets, which knowing the rules doesn’t fix.

We also tried plain web search as the extra tool instead of GitHits. On the original hidden-rule tasks, the agent with web search passed 27 of 60 runs, against 24 for browser-only and 44 with GitHits.

## Limits of the comparison

We ran the agents in our own harness, not a model vendor’s built-in computer-use mode, and only on GitLab and Magento. The blind set ran on Qwen only. Across all 73 tasks, the agent with GitHits still passed under 65% of runs. These are our own task suites, not a standard leaderboard score from WebArena or OSWorld, another public computer-use benchmark, so the numbers can’t be compared with published leaderboard results.

Earlier work such as [CUA-SWE](https://arxiv.org/abs/2609.32600) and [CoAct-1](https://arxiv.org/abs/2508.03923) has combined interface control with code, but as far as we know no one has published a controlled test of interface control with and without source lookup.

When an application’s rules live in its code rather than on screen, letting the agent look up that code at the matching version raised its pass rate across three model families. It didn’t help with finding things in the interface or with operating difficult widgets.

## Try it

You can give your coding agents access to GitHits through MCP, or call the API from agents you build yourself.

* **CLI + MCP:** run `npx -y githits@latest init`. The setup detects supported coding tools and configures the ones you select through the Model Context Protocol (MCP), a standard that lets AI tools call external services. Follow the [setup guide](https://docs.githits.com/installation/automatic-setup) for installation and sign-in.
* **API:** follow the [API quickstart](https://docs.githits.com/api/quickstart) to add GitHits lookups to your own agent. The API is [available for evaluation](https://docs.githits.com/api/overview). The v1 contract is not yet finalized.

If you’re building computer-use agents that operate open-source applications, [book a call](https://calendly.com/d/dvxq-7dn-msq/meeting-with-githits-select-15-or-30-minutes) to discuss integrating the API.

[Agents](/blog/tag/agents.md) · [Engineering](/blog/tag/engineering.md) · [Open Source](/blog/tag/open-source.md)

Nathan Burg — Co-founder, CPO

Keep exploring

* [The Solution](/the-solution.md)
* [Documentation](https://docs.githits.com/)
* [Changelog](https://docs.githits.com/changelog/overview)

Get started

## GitHits in one command

Install GitHits or refresh an existing setup with the same command.

`npx -y githits@latest init`

Or connect directly

* [Claude connector](https://claude.ai/directory/githits)
* [ChatGPT plugin](https://chatgpt.com/plugins/plugin_asdk_app_6ae059be511481918e3573fb5885dbb8)
* [Cursor plugin](https://cursor.com/marketplace/githits)

- [Vercel connector](https://vercel.com/connect/githits)
- [Manus connector](https://manus.im/app#connectors/all/connector_8387945e-9687-4a71-86b4-8a220dbd5b6b)
