Files
pipelines/app/tool/capture

Capture kit

One generic browser driver for UI review. It works on the whole app, not one feature, so a renamed label never makes it go stale. Use it instead of writing new capture code, and instead of a per-feature tool/*_probe.mjs driver that no longer matches the app.

  • capture.mjs: the CLI. It serves a build on an OS-assigned loopback port (--build) or takes --url. It starts private headless Chrome (fresh profile, debugging port chosen by the OS) or attaches to one with --cdp http://127.0.0.1:<port>. Then it runs the steps and writes NN-<name>.png and report.json to --out. On exit, Ctrl-C or failure it removes Chrome, the profile and the server.
  • It needs no npm packages: Node 22+ (global WebSocket and fetch) and Google Chrome (CHROME_PATH overrides the macOS default).

The fixture app

tool/search_selection_probe_entry.dart is the real PipelinesApp running on an in-process fixture channel. It never contacts a server. It has run 7 (three visits plus the artifact release-report.txt), run 6, pipelines 24 and 22, and search hits for release. It also answers server_stats, so the Settings route can be captured before the server command is available. Build it once per change, from app/:

/Users/alan/development/flutter/bin/flutter build web \
  --target tool/search_selection_probe_entry.dart --output build/probe-app

URL parameters (the kit sets them from open= and the options): route=<app route>, q=<search>, platform=material|cupertino, theme=light|dark; --params (or a step's or script's params) adds fixture options such as chats=2&rich=1. Use route=/settings to open Settings and then click Server health; the fixture channel returns server_stats automatically. Without theme, the app follows the browser's colour scheme. --theme also sets that colour scheme through Chrome emulation.

Steps

You can pass steps on the command line as verb=value. With --script, they go in a JSON array or in { "size", "platform", "theme", "q", "steps": [...] }. In JSON, a step can also set wait (ms), timeout, name, ms and frames.

step effect
open=/runs/7[@WxH] load a route at a size (default --size, 880x1000) and turn on Flutter semantics
shot=name take a screenshot
click=<target> click the centre of the target, then wait --wait ms (default 1000)
waitfor=<target> wait up to 10 s for the target to appear
type=text type into the focused field (or the first visible one)
press=Enter press a key: Enter, Escape, Tab, Backspace, ArrowUp, ArrowDown, a letter or a digit, optionally after Control+, Shift+, Alt+ or Meta+ (press=Control+2)
scroll=600 scroll with the mouse wheel at the centre (JSON at sets a target, dx a horizontal delta, repeat and gap a run of ticks)
hover=<target> move the mouse onto the target without pressing (hover states)
drag=<target>>dx,dy press on the target, move by dx,dy in steps (12) moves gap (16) ms apart, release (JSON dx, dy, steps, gap; hold: true keeps the button down to capture mid-drag)
size=WxH resize without reloading
labels=name write the visible semantics labels, keys and rects to JSON
film=<target> screencast a click for --film-ms (900) and write the frames plus a contact sheet; film=press:Alt+t films a key chord (as press=) instead
wait=ms sleep
files=a.ttf[,b.ttf] answer the next file chooser the page opens with these files (paths relative to the working directory); put it before the click that opens the picker
swipe=<target>>dx,dy one-finger touch swipe (needs --touch true, which emulates a touch screen once semantics is on)
pinch=<target>>scale two-finger touch pinch, spreading (>1) or closing (<1) (needs --touch true)
tabs=name write the open tabs ({url, type}) to JSON, e.g. to see a link open a new tab
eval=<js> evaluate an expression in the page and log/record its value
expect=<js> fail the run (status 1) unless the expression is truthy; JSON name labels the check

Targets:

  • A bare Artifacts or label:Artifacts is a semantics label (aria-label or the node's text). An exact match beats a prefix match, and then the smallest visible node wins.
  • key:list-detail-body is a Semantics(identifier:). A Flutter Key is not visible in the DOM.
  • 120,340 or xy:120,340 is a point in CSS pixels.
  • eval:<js> is a point an expression in the page yields as [x, y], waited for while it yields null (e.g. eval:JSON.parse(chatScrollTargets()).table).

When a target is missing, the step fails. The kit lists the visible labels, saves NN-FAILED.png and exits with status 1.

An HTML artifact: --params html=1 adds release-dashboard.html to run 7. Its bytes are served from the build, so copy the fixture in after building: mkdir -p build/probe-app/artifacts/7/2 && cp tool/capture/fixtures/release-dashboard.html build/probe-app/artifacts/7/2/.

Recipes (run from app/ after building build/probe-app)

Capture a page at phone and fold widths, in Material and in Cupertino. Use 560 for phone: headless Chrome misrenders Flutter below about 500 px.

for platform in material cupertino; do
  for size in 560x1000 880x1000; do
    node tool/capture/capture.mjs --build build/probe-app \
      --out /tmp/review/runs-$platform-$size --platform $platform --size $size \
      open=/runs/7 shot=run7
  done
done

Add --theme dark (or light) to pin the theme.

Click through to a state:

node tool/capture/capture.mjs --build build/probe-app --out /tmp/review/artifact \
  open=/runs/7@880x1000 click=Artifacts waitfor=release-report.txt shot=artifacts \
  click=release-report.txt shot=artifact-open labels=artifact-open

To run the same flow as a script, use --script tool/capture/examples/run7-artifacts.json. Search with open=/search then type=release, or use --q release.

Film a transition. This writes the frames (fNNN_<ms>ms.png, relative to the click) to <out>/film-label-Pipelines/ and the sheet to NN-film-label-Pipelines-sheet.png:

node tool/capture/capture.mjs --build build/probe-app --out /tmp/review/film \
  --film-ms 900 open=/runs@560x1000 film=Pipelines

In a JSON script, { "film": "Pipelines", "name": "tab-switch", "ms": 900, "frames": 6 } sets the name, the duration and how many frames go on the sheet.

Find targets: run labels=now after any step and read NN-now.json.

Look at every PNG you cite. A capture that exits 0 is not a review.

Check a chat scroll: the full-app fixture exposes chatScrollMetrics() (the open transcript's {pixels, min, max, viewport} as JSON; tool/perf/chat_scroll_probe.dart), and ?chats=N with ?msgs=, ?rich=1, ?older=1, ?live=N and ?tasks=N (tool/perf/chat_scroll_fixture.dart) shape the transcripts. tool/capture/examples/chat-wheel-scroll.json wheels over each content kind and expects the offset to move:

node tool/capture/capture.mjs --build build/probe-app --out /tmp/review/chat-wheel \
  --script tool/capture/examples/chat-wheel-scroll.json

For a prompt delivered to the running reply, use --params chats=2&steered=1, open /chat/perf-chat-0, then type and send a prompt. Without steered=1 the fixture retains the older-server Sent acknowledgement.

Chat file explorer

Use --params 'chats=2&files=1' open=/chat/perf-chat-0 click=Files to open a populated explorer (the fixture always answers list_directory). files=1 redirects view/download URLs to same-origin static fixture files; production continues to use /files. After building the probe, copy the fixtures:

mkdir -p build/probe-app/capture-files
cp tool/capture/fixtures/files/* build/probe-app/capture-files/

The tree has an assets subfolder, a hidden folder, PNG/JPEG miniatures, text, JSON, CSV, PDF, archive, log and video entries. Image/text/table/archive files have bundled bytes; PDF and video entries demonstrate type labels and menus only. Use the real server for their external handlers and playback acceptance.

For mobile peek regression captures, chatPaneScrollMetrics() returns every retained transcript's {id,pixels,min,max,viewport,top,height} as a JSON array. Open a chat, use Add chat pane, then pick the other fixture chat. A wheel/swipe over the inactive pane must change its own offset without activating it; at its bottom pixels reaches max and its viewport equals the small visible height. Use film=key:chat-peek-<pane-id> to record the swap.

Chat header grouping: ?chats=2&header=1 supplies a model, permission mode and thinking levels (answering chat_effort, tool/perf/chat_effort_fixture.dart) to the fixture chat so the selector island and Files/More island can both be reviewed.