--- title: "A Coding Agent With Four Tools Outperformed Its Bigger Rivals" description: "Databricks tested AI coding harnesses on its own codebase. The most minimal one, Pi, had the highest pass rate at a fraction of the cost of rivals." dek: "Databricks benchmarked AI coding assistants against its own multi-million-line codebase and found that the software wrapped around the model, not just the model itself, decided what each task cost." published_at: "2026-08-06T18:53:41.616Z" updated_at: "2026-08-06T18:53:41.616Z" tags: - "AI" - "Business" - "Infrastructure" source_url: "https://earendil.com/posts/pi-autoresearch-and-databricks/" source_domain: "earendil.com" canonical: "https://hex37.com/coding-agent-four-tools-outperformed-bigger-rivals" --- A programmer who uses an [artificial intelligence](https://en.wikipedia.org/wiki/Artificial_intelligence) model to write software almost never speaks to the model directly. In between sits a program known as a harness. It runs in the terminal, takes the request, decides which files and error messages the model gets to see, hands it the ability to run commands, and feeds the results back. Claude Code, made by Anthropic, is one. Codex, made by OpenAI, is another. Pi, made by a company called Earendil, is a third. The natural assumption about such a tool is that a better one does more: more built-in abilities, more carefully written instructions, more machinery for planning and checking work. Pi is built on the opposite bet. It arrives with four tools and nothing else, and its standing instructions to the model fit in under 1,000 tokens — roughly a page and a half of text. Everything beyond that, you are expected to build yourself. That should be a handicap. In a benchmark run by [Databricks](https://en.wikipedia.org/wiki/Databricks), the data infrastructure company, it was not. Understanding why requires looking at what a harness actually sends. ## Everything in the window is billed A [large language model](https://en.wikipedia.org/wiki/Large_language_model) has no memory of previous requests. Each time the harness asks it to do something, it resends the whole picture: the system prompt, meaning the standing instructions that tell the model how to behave; a description of every tool it is allowed to use; and the entire conversation so far. All of that is priced by the token, the small chunks of text that AI services bill by. All of it also has to fit inside the context window, the fixed amount of material a model can hold in view at once. So a harness's built-in instructions and tools are never free. They are a fee charged on every single request, and they occupy space that would otherwise hold the code the programmer actually wants fixed. Because AI has made writing software cheap, the industry has been spending that cheapness on size. Bigger prompts, more layers of orchestration, more tools shipped by default. Each addition is easy to justify on its own. Together they raise the floor price of every exchange with the model, before the user has typed anything. ## The instructions models no longer need Much of that scaffolding was written for a reason. Earlier models genuinely needed to be told, at length, how to move around a file system and interpret what a terminal was telling them. That is no longer the situation. Frontier models handle terminal-style environments competently without being coached through them. Anthropic recently cut Claude Code's system prompt by roughly 80% — a vendor conceding that most of its own instructions had become surplus. Pi was designed around that premise rather than retrofitted to it. Four tools, a sub-1,000-token prompt, and a working assumption that most tasks need nothing more. ## The same model, twice the bill Databricks wanted to know which coding agents performed best on real work and how performance tracked with price. Public benchmarks have been ground down by overuse, so the company built its own from tasks its engineers actually do inside a codebase running to millions of lines. The useful thing about the study is that it separated the model from the harness. Run the same model, at the same reasoning effort, through different harnesses, and Databricks found the cost per task diverged by more than two times in some cases while the quality of the output stayed flat. Same answers, double the invoice. The mechanism was visible in the numbers. Pi sent around three times less context per turn than the alternatives. It held a tighter working set of files and finished tasks in fewer runs. Fewer turns is where the saving compounds: every turn resends everything, so a harness that reaches the answer in six exchanges instead of ten pays the standing fee four fewer times. Paired with Opus 4.8 at its highest reasoning setting, Pi recorded the highest overall pass rate in the benchmark at a significantly lower cost than either Claude Code or Codex. That inverts the usual expectation. An expensive model driven efficiently came out cheaper than cheaper configurations driven wastefully. Earendil reports the same pattern one level down, at the model itself: complex workflows on Haiku 4.5, the smaller and cheaper option, sometimes cost more than the same work on Sonnet 4.6, because the weaker model needed more attempts to get there. ## What people build when nothing is built for them Minimal is only workable if it is extensible. At Shopify, an engineer asked Pi to write an extension for something called autoresearch; Pi read its own extension documentation and built it. Autoresearch is an autonomous loop for optimisation: you name a change you want, it runs experiments, keeps what improves a measurable target and throws out what causes regressions. Shopify has reported unit tests running 300 times faster, a React component mounting 20% quicker, shorter build times across projects, and improvements to the package manager it uses. None of that shipped with Pi. That is the argument. A vendor cannot know which workflows matter to which team, so features shipped on a guess become permanent overhead for everyone who does not use them. Practitioners have measured that guessing problem directly. One built a small extension that offered to write down anything worth remembering into a properly scoped instructions file, and then tracked how often the offer was worth taking. Fewer than 5% of suggestions were accepted. The rest were one-off observations that would have sat in the context window forever, costing money on every request and teaching the model nothing. ## Where the austerity is felt The savings are not free. Users who find Pi's defaults too thin argue there is a floor of functionality any coding tool should provide without setup work — the complaint is that stock Pi resembles a text editor that will open a file but leaves you to add saving and searching yourself. Others report plainer friction: a program that advertises minimalism but takes a noticeable time to start, standard terminal key bindings that do not work, configuration files dumped into the home directory rather than the standard location for them. For those objections to be wrong, the setup cost would have to be smaller than the cost of carrying features you never use — and that is a judgment that changes with how much time a team has. What the Databricks numbers establish is narrower and harder to argue with. When computing is metered by the token, how well a tool manages context determines its cost more than how many features it has, because a feature nobody uses is pure expense. The unresolved part is durability. Model providers are beginning to encrypt the internal reasoning their models produce, and to build capabilities that only their own harnesses can reach. If that becomes the norm, third-party tools would be working with less information than the vendors' own — and efficiency might stop being enough.