We A/B tested a token-saving pi extension. It cost more

The pi coding agent has an extension called pi-rtk-optimizer. Its promise: rewrite shell commands to their rtk equivalents and compact noisy tool output to save context tokens. That sounded useful, so we put it to a real test by running identical tasks against two isolated copies of our environment, one with the extension and one without.

The result: extra compression was between 0 and 0.7 percent, and total cost went up by about 7%. This post is the measurement log.

The short version: pi already does this job itself

• Extra compression: 0-0.7%. The bash tool built into pi already truncates large output; whether the extension was loaded made almost no difference to what the model actually received
• On a realistic research task, total cost rose ~7%. Compacted output hid information, the agent re-ran commands to recover it, and the average turn count grew from 12 to 16
• Usage statistics are not persisted. /rtk stats only covers the current session, so nobody could have measured the long-term effect — there was no data
• The extension is abandoned. The latest npm version (0.9.0) pins pi to ^0.74-0.80, which excludes even our current 0.87.1

We assumed this extension was free insurance. The measurements say we were paying for it.

Background: a major pi upgrade forced an extension audit

The trigger was a routine version check.

pi --version
npm view @earendil-works/pi-coding-agent version

0.87.1        # installed
1.3.0         # latest (released 2026-10-15)

Going from 0.87.1 to 1.3.0 crosses a major version boundary, with several breaking changes to the extension API. Our environment loads 11 extension packages, so before updating anything we wanted to know which extensions actually pull their weight.

pi-rtk-optimizer was the first suspect, for three reasons.

• Its peerDependencies pin pi to ^0.74.0 || ^0.75.0 || ^0.78.0 || ^0.79.0 || ^0.80.0 — already out of range on our 0.87.1
• The npm latest tag has sat at 0.9.0 with no updates
• Its statistics command (/rtk stats) keeps numbers in memory only; the source (output-metrics.ts) writes nothing to disk

That last point matters most: no accumulated evidence of benefit existed anywhere.

Method: two isolated copies, one line apart

pi supports relocating its configuration directory with an environment variable.

PI_CODING_AGENT_DIR    # overrides the default ~/.pi/agent (official docs)

We copied the production directory twice and removed the extension from one copy.

# Copy the production agent dir (excluding 876MB of session history)
cp -r ~/.pi/agent ~/.pi-test-a/agent     # Group A: extension enabled
cp -r ~/.pi/agent ~/.pi-test-b/agent     # Group B: extension removed

# The only change in Group B: one line removed from settings.json
node -e "
const fs = require('fs');
const p = process.env.HOME + '/.pi-test-b/agent/settings.json';
const s = JSON.parse(fs.readFileSync(p, 'utf-8'));
s.packages = s.packages.filter(x => x !== 'npm:pi-rtk-optimizer');
fs.writeFileSync(p, JSON.stringify(s, null, 2));
"

Everything else — extensions, models, settings — stayed identical, so any difference is attributable to the extension alone.

Runs were headless, with the model pinned.

PI_CODING_AGENT_DIR=~/.pi-test-a/agent \
pi -p \
  --model open-design/deepseek-v4.1-flash \
  --session-dir <per-group directory> \
  --append-system-prompt "Never write, modify, or delete files. Read-only research only." \
  "<task text>"

One caveat: a compound research task is non-deterministic, so turn counts vary between runs (we return to this under *Limitations*). That is why we also measured the fully deterministic tasks 2 and 3.

Result 1: re-querying ate the savings

Task 1, three-run averages:

Individual runs:

Group A (enabled)
  run1: turns=17 promptTokens=425,225 outTokens=12,652 toolCalls=27 toolOutputChars=38,946
  run2: turns=15 promptTokens=323,350 outTokens=13,020 toolCalls=25 toolOutputChars=20,339
  run3: turns=17 promptTokens=497,741 outTokens=15,723 toolCalls=30 toolOutputChars=47,478

Group B (disabled)
  run1: turns=10 promptTokens=238,361 outTokens= 9,927 toolCalls=16 toolOutputChars=40,614
  run2: turns=16 promptTokens=352,099 outTokens=13,597 toolCalls=25 toolOutputChars=29,558
  run3: turns=11 promptTokens=445,201 outTokens=11,043 toolCalls=17 toolOutputChars=85,810

Tool output really was 32% smaller with the extension — the compaction works. But recovering the hidden information took extra command runs, turns climbed, and every extra turn re-sends the system prompt and conversation history. Prompt tokens and cost went up, not down.

Result 2: pi’s own truncation was already doing the work

Then we compared fully deterministic commands.

seq 1 30000 (raw output 163,893 chars)

The gap is 0.7%. Both groups clipped 163,893 chars down to roughly 12,000. That is the built-in bash tool limit doing its job (2,000 lines / 50KB, with the full output written to a side file).

seq 1 8000 (raw output 37,889 chars — under pi’s limit)

Identical to the character — a 0.0% difference. Even at mid-size outputs that pi’s limit never touches, the extension changed nothing.

Why it did not help

• Large outputs are already handled by pi. The 2,000-line/50KB truncation fires first, leaving no room for the extension’s pipeline
• Mid-size outputs never match a compaction pattern. ANSI stripping, test aggregation, build filtering, and git compaction only trigger on their specific patterns. Plain number sequences and file listings matched none
• Command rewriting compresses, but also obscures. Rewriting did fire (an ls came back with a [+85 hidden: rtk recall ...] marker). But when the hidden content mattered, the agent had to run more commands — turns and cost again
• The dependency range does not fit any current pi. Pinned to ^0.74-0.80, it excludes both 0.87.1 and 1.3.0, so there is no upstream fix coming

Limitations

• n=3 on the compound task, with visible variance. Non-deterministic model behavior is not fully controlled. The direction (more turns → more prompt tokens) was consistent across all three runs, though
• seq is not the extension’s home turf. Patterned outputs like test runner logs or git diffs might compact differently. The structural conflict with pi’s built-in truncation would remain either way
• One model only (deepseek-v4.1-flash). The method transfers: any model can be re-measured the same way

Summary

The extension that promised savings was not saving anything.

• Extra compression measured 0-0.7%; pi’s built-in truncation already does that job
• On a realistic task, extra re-querying pushed total cost up ~7%
• Per-session-only statistics meant nobody had ever measured the long-term effect
• The method is reusable: copy the agent dir, flip one line of settings.json, and A/B test any extension you suspect

As for upgrading pi itself to 1.3.0 — that is a separate decision. But for this extension, the measured answer was: it can go, today.