A browser for agents

Some work lives on websites with no API: a supplier portal, a government form, a dashboard that only exports a CSV from a button. The browser capability gives an agent a browser for those sites. The agent can open pages, read them, take screenshots, download files and, if you allow it, click, fill in and submit forms.

The browser runs in the office, not in the agent's sandbox. The agent drives it through office tools (browser_open, browser_click, …), the office fetches every page itself, and nothing new opens in the sandbox's network allowlist. Logins stay with the office: the agent never sees a password or a cookie.

Optional The browser needs Playwright and Chromium on the office host. Without them the office runs as before and the browser tools answer "this office has no browser". officectl doctor tells you which is the case.

Install#

The office uses playwright-core (an optional dependency, installed with the office package) and a Chromium.

Ubuntu 24.04. As root, once:

sudo bash host-prereqs-ubuntu.sh --with-browser

This installs the libraries and fonts Chromium needs (libnss3, libgbm1, libasound2t64, libatk-bridge2.0-0t64, libcups2t64, libxkbcommon0, libxcomposite1, libxdamage1, libxrandr2, libpango-1.0-0, libcairo2, … and fonts-liberation, fonts-noto-color-emoji, fonts-noto-cjk) and an AppArmor profile that lets Chromium's own sandbox use user namespaces, which Ubuntu 24.04 restricts. The script is in the package (node_modules/@agent-office/platform/scripts/). Then, as the office user:

sudo -u office officectl browser install   # downloads Playwright's headless Chromium into <state>/browsers
sudo -u office officectl doctor            # "browser for agents: Chromium …"

macOS. As the office user: officectl browser install. Nothing else is needed.

A system Chromium. Set browser.executable in office.yaml to its path instead of running officectl browser install. Keep it updated: it renders untrusted pages.

Grant it#

In the agent's personnel file:

capabilities:
  - browser:
      sites: ["portal.example.com", "*.supplier.example"]
      actions: [click, fill, select, submit]
      approve: [submit]

• sites: the only hosts the browser may load anything from. *.supplier.example covers its subdomains, not supplier.example itself. Everything else is blocked: pages, frames, scripts, images, styles and the redirects between them.
• actions: the write actions the agent may take. Leave it out for a read-only browser (open, read, screenshot, download). login (filling a stored sign-in) needs fill.
• approve: click and/or submit, the actions that can send something. Each waits for a person on Needs you.
• subresources: allow-any-public: let pages load images, scripts and styles from any public host (GET only). Off by default; see what it costs.
• pages_per_run (30), minutes_per_run (10), download_mb (10, the most): per-run limits.

The full parameter list is in the capability reference. Like every capability, it changes by policy pull request.

What the agent sees#

browser_open returns the page as text, with every link, button and field numbered:

# Contact us
Write to us and we answer within a day.
Your name [#4 textbox "Your name" value=""] Topic [#5 select "Topic" options: "General"*, "Billing"]
[#6 textarea "Message" value=""] [#7 button "Send"]
- [#1 link "Orders" → https://portal.example.com/orders]

The text is framed as outside content: the agent is told to treat it as information, never as instructions. Links show their target, so the agent can follow them with browser_open (a read) rather than browser_click (a write).

Approvals#

With approve: [submit], a submit doesn't happen. The agent gets "Waiting for approval #N" and carries on; Needs you shows the request with the page URL and every field the agent filled on that page, with the values, so you see exactly what would be sent.

• Approve: the office opens the page again in the agent's browser (its sign-ins), fills in the same fields, does the submit, saves a screenshot of the result as an artifact and mails the agent the resulting page. If the page changed so a field can't be found, nothing is sent and the mail says why.
• Deny: nothing is sent; the agent gets a mail saying so.

With run.untrusted_writes: approve in the personnel file, every click and submit in a run started by outside content (a webhook, an email, a Slack message) waits for a person too, even when approve doesn't list it. Fills stay free in such runs: they only change the page in the office.

Graduated autonomy. Clicks and submits are the action class browser.write (graduated autonomy). In the personnel file's autonomy, browser.write: suggest records each click or submit for a person (who can do it, have the office do it once, or dismiss it), draft files the approval above, and act performs it, recorded so it can be flagged. The approve list and run.untrusted_writes hold the class at draft at most; the track record can earn promotion like any other class.

In eval runs the agent browses for real (reads, within its sites), but clicks, fills, selects and submits are recorded as dry runs and not performed, and screenshots and downloads aren't saved.

Sign the agent in#

People sign an agent in from the office host with officectl (as the office user). There is no web page for it yet.

Import cookies (works for SSO and two-factor logins). Sign in to the site in your own browser, export its cookies (a cookie-export extension's JSON, a Netscape cookies.txt, or a Playwright storage state), then:

sudo -u office officectl browser cookies ada cookies.json --site portal.example.com

Only cookies for the --site hosts are kept. Use an account made for the agent, not your own: the agent acts with whatever the cookies allow. Delete the exported file afterwards.

Store a sign-in (username and password sites):

printf '%s\n%s\n' "ada@example.com" "$PASSWORD" | sudo -u office officectl browser login ada portal.example.com

The values go to the encrypted secret store. The agent calls browser_login with the numbers of the password (and username) fields: the office fills them in only on an https page whose host is exactly portal.example.com, and only into a password-type field. The values are never in a tool result, a run event, an approval or the ledger. The agent then submits the form, which approve can gate as usual.

Forget: officectl browser forget ada drops the agent's cookies, local storage and stored sign-ins; --site portal.example.com drops only that sign-in. officectl browser status lists what each agent has (counts and hosts, never values).

Where it shows up#

• Run timeline: a Browser step for each page opened, click, fill, download and approval request, with screenshot thumbnails; blocked pages show as Blocked.
• Ledger: browser.navigate (host only), browser.action (action, host, element; never typed values), browser.blocked, browser.download, browser.screenshot, browser.login, browser.approval, and the CLI's browser.storage.imported / browser.login.set / browser.storage.cleared. See ledger actions.
• Artifacts: screenshots and downloads, in the agent's vault.

Limits and trade-offs#

• One page per run (pop-ups are closed). The browser closes when the run ends, after minutes_per_run, or when the office stops. At most browser.max_concurrent browsers run at once office-wide (default 2).
• Downloads started by a click are cancelled: the agent uses browser_download on the link.
• Many sites load scripts and fonts from a CDN. If a page needs them, list the CDN's host in sites (preferred), or set subresources: allow-any-public. The second lets page scripts send data out in GET URLs to any public host, so keep it off for sites where the agent types anything sensitive.
• Sites that save as you type (autosave, live search) see fills as they happen. approve gates clicks and submits; for such sites leave fill out of actions.
• An approval replays one page's steps. A flow that needs several approved steps needs several approvals.

How the browser is isolated, and what it can and can't reach, is on the security pages.