Five ways to drive a browser

What I learned

• Claude in ChromeChrome 154, my own profile
  extension

  Claude in Chrome costs about twice as much

  A median run took 26 seconds and 122k input tokens, against 13 seconds and 33k to 57k for the other four. It reaches for screenshots more often (a median of 4.5 images per realistic task), its fixed context is the largest at 16k tokens, and each tool call round-trips through the extension. It also could not read an HttpOnly cookie in any run. None of this measures what it is actually for: driving my own browser, where I am already signed in. For that it is still the only option here.

• Safari MCPSafari 27
  safaridriver --mcp

  Safari MCP is the leanest, with two gaps

  It was the cheapest surface overall and returned the most compact page reads, a median of 0.8 KB per probe. Two things are missing. A native confirm() left its click call hanging until the four-minute timeout, in all three dialog runs. And it has no press-move-release gesture, so on the kanban board the agent had to fake the drag by dispatching pointer events from page JavaScript. Only 70% of its passing interactions reached the page as real input (event.isTrusted), against 100% for every other surface.

• Orca browserChromium 150 (Electron)
  orca CLI through Bash

  Orca's browser is fragile around the host app

  A native confirm() dropped the Orca runtime connection, so every command after it failed; one of three runs got through with a workaround. set media print reported success without changing the media type. Its speed also depended on the Orca window itself: its viewport follows the panel size, and once the harness pre-opened a pinned tab and the workspace was on screen, its median run time halved.

• Playwright · Chrome Playwright · WebKit @playwright/mcp 0.0.83

  The interface matters more than the engine

  Playwright on Chrome and on WebKit finished within a second of each other on every tier. The bug hunt built around engine differences came out exactly as the engines dictate: both WebKit surfaces found three broken widgets, because two of them call Chromium-only APIs, and all three Chromium surfaces found one.

• Claude in Chrome Orca browser no print-media emulation

  The agents said when they took a shortcut

  Chrome and Orca have no way to emulate print media, so the agents improvised: one rewrote the page's print CSS and fired beforeprint, the other rendered a PDF to trigger the print handler. Both reached the right code and explained the workaround in their notes. The checker still fails those runs, because the task asked for real print media. They are the only five cases where an agent claimed success and failed.

The setup

What went wrong while measuring

The benchmark found bugs in itself before it found anything about the browsers, and I think those are worth listing:

• fixture

  Two fixture bugs. Before the full run, a scripted Playwright solution solved every task in both Chrome and WebKit. It caught a pop-up whose display: flex overrode the hidden attribute, which would have failed every agent, and a console page where two identical WebKit exceptions were reported as one, so the correct answer would have looked wrong.

• viewport

  A resized window. On the “make the viewport phone-width” probe, Claude in Chrome shrank my real Chrome window to 500 pixels and never restored it. The next four Chrome runs started narrow. The viewport beacon caught it; I discarded those runs, and the harness now restores window bounds and refuses runs that start narrower than 1700 pixels.

• runner

  A false rate limit. An API 529 Overloaded error looked like a usage limit to the first version of the runner, which then planned to sleep for four and a half hours.

• safety filter

  A safety filter. The HttpOnly-cookie probe tripped the API's cyber safeguard in 12 of its 15 runs. “Read this session cookie” looks a lot like cookie theft. The refused turn was retracted and re-run on an older model, so that probe mixes models; the other 658 runs used Opus 5.5 only.

The first 174 runs ran one at a time. For the rest I ran all five surfaces on the same task at the same moment, which was about five times faster and arguably fairer, since every surface saw the same machine load and the same API conditions. For four of the five, times on matching tasks moved by at most 6%.