Scaling cloud agents without losing track of their dependencies

A cloud coding agent picks up a job: update a library, fix a build, answer a question about how a dependency behaves. It creates a sandbox, an isolated cloud machine that exists for one job and is deleted afterwards, checks out the job’s repository, and works there. When a fleet of agents runs thousands of these jobs a week, nobody prepares the sandboxes by hand. Each agent sets up its own machine, connects to what it needs, and finds the code it depends on.

The dependency code is the hard part. The job’s repository records the exact version of every dependency in its lockfile, the file a package manager writes so that every install gets the same versions. The agent has to read each dependency at that version. A release from a few weeks earlier or later can have different options, defaults, and files, and an agent that reads it is working from code the job never runs.

There are three common ways to give an agent that code. It can clone the dependency’s repository and check out the right release. It can start from a snapshot, a machine image prepared ahead of time with the dependencies already on it. Or it can clone only the job repository and read every other project through GitHits, which returns the source, documentation, and package metadata of a public open-source project at the version the request names.

We gave agents each of those strategies and let them do everything else themselves: create the sandbox, find the pinned version, read the code, answer, and delete the sandbox. Through GitHits, the agents read every dependency at the pinned version in all 192 runs. Agents that cloned the dependencies themselves read the pinned version in 94.8% of runs, and GitHits cost about the same as cloning or less per correct answer. Agents starting from a snapshot catalog read the pinned version in 66.1% of runs and paid more per correct answer than with GitHits.

Click to expand

Close

How the agents worked

Each job pairs a public repository with one of its dependencies, at three commits from the past year where the pinned version of that dependency is different. That gives 24 jobs across four package ecosystems.

The agent received the task, the job repository’s URL, and the commit. It was not told where the dependency’s source lives or which version the job uses, so it had to read the lockfile and work that out. The task asked for the pinned version and two values from the dependency’s own manifest at that version, for example a peer dependency range in package.json. The agent had three tools: create a sandbox, run a command in it, and delete it. The sandboxes ran on Daytona, and the agent was DeepSeek V4.1 Flash. Every job ran eight times with each strategy.

A run passed when the version and both values matched the dependency’s manifest at the pinned release. We also checked every command the agent ran and recorded which version of the dependency each read came from. A run counts as reading at the pinned version only if every dependency read in it was at that version.

The three strategies were:

• Clone. The agent could clone or fetch anything with git, in whatever way it chose. Every agent chose a blobless clone, which downloads the commit history without the old file contents and fetches files only when they are checked out. Some also made a shallow clone of just the latest commits. None made a full clone.
• Snapshot catalog. A separate maintenance agent built a catalog of snapshots, one for each state of each job repository’s lockfile over the previous six weeks, with the dependencies checked out at the versions that lockfile named. That came to 15 snapshots. The agent picked one to start from, or fell back to git when the catalog didn’t have the version. The maintenance agent’s sandbox time, model use, and snapshot builds are counted in this strategy’s cost.
• GitHits. The agent cloned only the job repository and read the dependency through the GitHits API with plain HTTP requests from inside the sandbox, naming the package and the version it had found in the lockfile.

What each strategy got right, and what it cost

Cost is the sandbox time at Daytona’s published per-second price, plus model tokens, plus snapshot upkeep for the catalog. GitHits API fees are not included. The median is the middle run, so half the runs took less time and half took more.

Every clone run reported the right answer, but in 10 of the 192 runs the agent read at least one dependency file at a release the job doesn’t use, 9 of them in the jobs that read the AI SDK (ai) from npm. The answers in those runs still matched, so the task passed, though the agent had been reading the wrong code. Through GitHits, each request carries the package name and the version, and no run read the wrong release. The gap is 5.2 percentage points, with a 95% interval from 0.5 to 10.9 points when we resample the jobs.

GitHits cost no more per correct answer than cloning. It came to 0.82 times the clone figure, with an interval from 0.63 to 1.04, so the true cost could still be about the same. The median run took 15 seconds with GitHits and 18 seconds with a clone.

Those clone figures come from the partial clones the agents chose. In an earlier experiment where we cloned the full history of four related repositories into each sandbox, the sandbox time cost about 16 times as much as the same jobs through GitHits.

Why the snapshots fell behind

A snapshot only helps when someone has already built an image with the version the job needs. The catalog covered six weeks and the jobs reach back a year, so 14 of the 24 jobs pinned an older version. When the catalog had the job’s version, 95.0% of snapshot runs read it and 98.8% passed. Otherwise 45.5% read it and 67.0% passed, mostly because the agent started from the newest snapshot without checking the lockfile.

Each lockfile change means another image to build and store. The 15 snapshots cost about $0.05 during the experiment, which is included above, and a week of storage, which is not, would add about $3.86. GitHits has nothing to build or store, because the version is part of every request, and per correct answer it cost 0.59 times as much as the catalog (0.43 to 0.77).

The snapshot runs for 21 jobs came from an earlier run that day. On the original 24 jobs, before we replaced one pair (pydantic-ai reading anthropic) whose job-repository clones stalled, GitHits cost 1.12 times the clone figure (0.78 to 1.62) and again read every dependency at the pinned version.

What this means for agent fleets

When agents set up their own sandboxes, let each one clone only the repository its job changes and read every dependency through GitHits. The agent finds the version in the lockfile and names it in each request, so it reads the code the job actually runs. In this experiment that held in all 192 runs, at about the same cost per correct answer as the agents’ own clones or less, with no snapshot catalog to keep current. We ran it on public open-source projects.

Try it

You can give your coding agents access to GitHits through MCP, or call the API from the agents in your fleet.

• CLI + MCP: run npx -y githits@latest init. The setup detects supported coding tools and configures the ones you select through the Model Context Protocol (MCP), a standard that lets AI tools call external services. Follow the setup guide for installation and sign-in.
• API: follow the API quickstart to add GitHits reads to the agents in your fleet. The API is available for evaluation. The v1 contract is not yet finalized.

If you’re running agents that set up their own sandboxes, book a call to discuss integrating the API.