test(app): browser probe for search result selection

This commit is contained in:
2026-10-22 18:40:35 +01:00
parent 0acd7346bb
commit 4f0d2aab6f
3 changed files with 1255 additions and 0 deletions
+140
View File
@@ -0,0 +1,140 @@
# Search result-target selection: browser probe
Real-Chrome evidence for the PARITY `Search` row's missing capture: a field hit
and an artifact hit select the recorded run Summary section or visit, and a
retired field / unknown visit falls back to the plain Summary.
## Files
- `app/tool/search_selection_probe_entry.dart` — the normal `PipelinesApp`
composition over the shared `PipelinesClient`, with the socket replaced by an
in-process channel that answers `search`, `get_run`, `list_runs`,
`list_pipelines`, `get_pipeline` and `ping` with server-shaped fixtures.
`subscribe`/`unsubscribe` are silent like the real server. A `search` body
that is not exactly `{query, kind, limit: 0}` is answered with an error
envelope, so a wrong request shape fails the captures instead of passing.
No live server; nothing is contacted over the network.
- `app/tool/search_selection_probe.mjs` — CDP driver (same shape as
`changes_probe.mjs`): loopback static server with index fallback for route
paths, headless Chrome, Flutter semantics enabled, exact-label clicks, PNGs.
Run from `app/`:
```
flutter build web --target tool/search_selection_probe_entry.dart --output build/probe-search-selection
node tool/search_selection_probe.mjs --root build/probe-search-selection --out build/search-selection-probe
```
Exit code 0 means every assertion held and no defect was recorded; exit code 2
means the assertions held but at least one recorded state differs from the
expected one (listed under `defects` in the JSON report). A thrown assertion
also writes `failure.png` next to the captures.
## Fixture
Run 7 "Release train run" (`name: release-train`), `input {prompt: "ship it"}`,
`output {result: "ok"}`, steps in recorded order plan seq 5, build seq 2
(artifact `release-report.txt`), plan seq 1 — the sequences are deliberately
not the row indexes, as in `test/features/runs/ui/run_search_selection_test.dart`.
Search replies (`truncated: false`, `scanned_steps 3`, `scanned_bytes 2048`,
`cap 50`, `scan_bound_bytes 1048576`):
| query | hits |
| --- | --- |
| `release` | `pipeline:24`, `pipeline:22:v6`, `run:7`, `run:7:5` (plan visit 2), `run:7:5:output`, `/artifacts/7/2/release-report.txt` |
| `retired` | `run:7:5:evidence` (retired field) |
| `missing` | `/artifacts/7/9/missing.log` (sequence 9 is not recorded) |
| `first` | `run:7:1` (plan visit 1) |
## Captures (`app/build/search-selection-probe/`)
Light scheme is pinned through `Emulation.setEmulatedMedia`; one dark pass is
included. Phone is 390×844, desktop 1280×800.
| capture | state |
| --- | --- |
| `search-selection-material-phone-results.png` | Search panel, `release`, one row per kind |
| `search-selection-material-phone-artifact-visit.png` | artifact hit → Timeline, `build · visit 2` pane |
| `search-selection-material-phone-retired-summary.png` | retired field after the open visit → plain Summary |
| `search-selection-material-phone-unknown-visit-summary.png` | unknown-visit artifact after an open visit → plain Summary |
| `search-selection-material-phone-field-summary.png` | field hit (`output`) → Summary, Output expanded, Input collapsed |
| `search-selection-material-phone-retired-after-field-summary.png` | retired field after that field hit (defect 1 evidence) |
| `search-selection-material-phone-step-visit.png` | step hit `plan visit 1` (defect 2 evidence) |
| `search-selection-material-desktop-results.png` | desktop results |
| `search-selection-material-desktop-artifact-visit.png` | desktop artifact visit |
| `search-selection-material-desktop-retired-summary.png` | desktop plain Summary |
| `search-selection-material-desktop-field-summary.png` | desktop field selection |
| `search-selection-material-phone-dark-results.png` | dark results |
| `search-selection-material-phone-dark-field-summary.png` | dark field selection |
| `search-selection-cupertino-phone-results.png` | Cupertino results (`Kind: All kinds` control) |
| `search-selection-cupertino-phone-artifact-visit.png` | Cupertino artifact visit |
| `search-selection-cupertino-phone-retired-summary.png` | Cupertino plain Summary after retired field |
| `search-selection-cupertino-phone-unknown-visit-summary.png` | Cupertino plain Summary after unknown visit |
| `search-selection-cupertino-phone-field-summary.png` | Cupertino field selection |
| `search-selection-cupertino-phone-retired-after-field-summary.png` | Cupertino retired-after-field (defect 1 evidence) |
| `search-selection-cupertino-desktop-results.png` | Cupertino desktop results |
| `search-selection-cupertino-desktop-artifact-visit.png` | Cupertino desktop artifact visit |
| `search-selection-cupertino-desktop-field-summary.png` | Cupertino desktop field selection |
## Asserted strings (Flutter semantics, exact)
- Results, every host/width: row titles `Release train`, `pipeline 22 · v6`,
`run #7 · Release train`, `run #7 · plan visit 2`, `run #7 · step 5 · output`,
`release-report.txt`; row meta `Pipeline`, `Pipeline version`, `Run`, `Step`,
`Field`, `Artifact`; snippet `ship the release`; bounds
`6 results · cap 50 · scanned 3 steps · 2.0 KiB scanned · scan limit 1.0 MiB`;
Cupertino also `Kind: All kinds`.
- Every hit: `location.pathname === '/runs/7'`, title `Release train run`,
and the Search page's text gone.
- Field hit: Summary tile `Output` announces `Expanded` and `Input` announces
`Collapsed` (first line of the tile's `aria-description`); no
`Back to timeline`, no `build · visit 2`.
- Artifact hit: `build · visit 2` and `Back to timeline` present;
`Original prompt` (Summary-only text) absent.
- Retired field / unknown visit: `Original prompt` present, `Input` and
`Output` both `Collapsed`, no visit pane text.
- One-shot: after an artifact hit, `Back to list` → row `Release train run`
reopens `/runs/7` on the Summary with no `build · visit 2`.
- Fixture-side: a `search` body other than `{query, kind, limit: 0}` and a
`get_run` for any run but 7 reply with an error envelope; both would have
failed the label assertions above (the JSON report has `pageErrors: []`).
Semantics limitation: Flutter web renders a read-only `SelectableText` /
read-only `CupertinoTextField` as a `<textarea>` whose `value` stays empty, so
the section JSON (`"result": "ok"`) is only visible in the PNGs; the probe
asserts the section state on the tile instead and scrolls the body so the
expanded JSON is inside the frame.
## Defects found (lib not changed)
1. A retired-field hit after a field hit keeps the earlier section expanded.
Flow on one mounted run 7: `run:7:5:output` → Output `Expanded`; then
`run:7:5:evidence` → `output: "Expanded", input: "Collapsed"`, expected both
`Collapsed` (report entries `material phone: a retired field hit after an
output field hit …` and the Cupertino twin). The widget tests only cover
retired-after-artifact, where nothing was expanded. Cause:
`RunDetailTabs._selectSearchTarget` handles `RunSummaryTarget(section: null)`
with `case null: break;` and never collapses `_inputSection`/`_outputSection`
(`app/lib/features/runs/ui/run_detail_tabs.dart`); only a different run
collapses them. The browser's `searchFieldTab` selects the exclusive
Summary tab, so a retired hit never shows the Output text there
(`web/src/search.js`). Evidence:
`search-selection-{material,cupertino}-phone-retired-after-field-summary.png`.
2. A step hit opens the wrong visit when the stage has more than one visit.
`run:7:1` (`step.visit: 1`, sequence 1, plan's first visit) navigated to
`/runs/7/stage/plan?visit=1` and opened `plan · visit 5` (plan's second
visit); expected `plan · visit 1`. `_openSearchResult` in
`app/lib/application/pipelines_app.dart` passes the server's 1-based
`visit` straight into `AppRoute.run.stage(..., visit:)`, which
`app_route.dart` documents as zero-based (and `RunDetailTabs` clamps). The
browser resolves a step target to the zero-based index within the stage
(`resolveRunTarget` in `web/src/search.js`). Evidence:
`search-selection-material-phone-step-visit.png`, report entry
`material phone: a step hit for plan visit 1 opens plan · visit 1`.
## Not verified
Web only; Android/iOS are unverified. The pipeline and version hits are listed
and the fixture answers `list_pipelines`/`get_pipeline`, but their navigation
is not captured here (it is covered by the existing search wiring probe).
+646
View File
@@ -0,0 +1,646 @@
// Real-browser evidence for Search result-target selection.
//
// Usage (from app/):
// flutter build web --target tool/search_selection_probe_entry.dart \
// --output build/probe-search-selection
// node tool/search_selection_probe.mjs --root build/probe-search-selection \
// --out build/search-selection-probe
//
// Serves the fixture build over loopback (extension-less paths fall back to
// index.html so the app's real `/runs/7` and `/search` locations load), drives
// headless Chrome through the Chrome DevTools Protocol, enables Flutter
// semantics, types a query into the real search field, clicks result rows by
// their exact row title and asserts the run detail state the hit selects:
//
// field hit -> run Summary with the matched Output section expanded
// artifact hit -> the recorded visit pane (`build · visit 2`)
// retired field -> the plain Summary, no section expanded
// unknown visit -> the plain Summary, no visit pane
//
// Every reply comes from the in-app fixture channel in
// tool/search_selection_probe_entry.dart; no server is contacted.
import { createServer } from 'node:http';
import { mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises';
import { spawn } from 'node:child_process';
import { once } from 'node:events';
import { setTimeout as delay } from 'node:timers/promises';
import { extname, join, resolve, sep } from 'node:path';
import { tmpdir } from 'node:os';
const args = new Map();
for (let i = 2; i < process.argv.length; i++) {
const value = process.argv[i];
if (value.startsWith('--')) args.set(value.slice(2), process.argv[++i]);
}
const root = resolve(args.get('root') ?? 'build/probe-search-selection');
const out = resolve(args.get('out') ?? 'build/search-selection-probe');
const chromePath =
process.env.CHROME_PATH ?? '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome';
await mkdir(out, { recursive: true });
const mime = {
'.html': 'text/html; charset=utf-8',
'.js': 'text/javascript; charset=utf-8',
'.mjs': 'text/javascript; charset=utf-8',
'.json': 'application/json; charset=utf-8',
'.wasm': 'application/wasm',
'.png': 'image/png',
'.svg': 'image/svg+xml',
'.ico': 'image/x-icon',
'.ttf': 'font/ttf',
'.otf': 'font/otf',
'.woff': 'font/woff',
'.woff2': 'font/woff2',
'.bin': 'application/octet-stream',
'.frag': 'application/octet-stream',
};
const server = createServer(async (req, res) => {
try {
const pathname = decodeURIComponent(new URL(req.url, 'http://localhost').pathname);
let filePath = resolve(root, '.' + pathname);
if (filePath !== root && !filePath.startsWith(root + sep)) {
res.writeHead(403).end();
return;
}
// The app uses path URLs (`/runs/7`, `/search`); any route path without a
// file extension is the single page, as the Pipelines proxy serves it.
try {
if ((await stat(filePath)).isDirectory()) filePath = join(filePath, 'index.html');
} catch {
if (!extname(pathname)) filePath = join(root, 'index.html');
}
const file = await readFile(filePath);
const type = mime[extname(filePath)] ?? 'application/octet-stream';
res.writeHead(200, { 'content-type': type, 'cache-control': 'no-store' }).end(file);
} catch {
res.writeHead(404).end('not found');
}
});
server.listen(0, '127.0.0.1');
await once(server, 'listening');
const port = server.address().port;
const origin = `http://127.0.0.1:${port}`;
const profile = await mkdtemp(join(tmpdir(), 'x1-search-selection-chrome-'));
const chrome = spawn(
chromePath,
[
'--headless=new',
'--no-first-run',
'--no-default-browser-check',
'--disable-gpu',
'--remote-debugging-port=0',
`--user-data-dir=${profile}`,
'about:blank',
],
{ stdio: ['ignore', 'ignore', 'pipe'] },
);
let stderr = '';
chrome.stderr.on('data', (data) => {
stderr += data;
});
const waitFor = async (fn, what, attempts = 120) => {
for (let i = 0; i < attempts; i++) {
const value = await fn();
if (value) return value;
await delay(100);
}
throw new Error(`Timed out waiting for ${what}: ${stderr.slice(-1500)}`);
};
// The exact row titles the fixture's six `release` hits render, one per kind,
// and the row meta each one announces (tool/search_selection_probe_entry.dart).
const RELEASE_ROWS = [
{ title: 'Release train', meta: 'Pipeline' },
{ title: 'pipeline 22 · v6', meta: 'Pipeline version' },
{ title: 'run #7 · Release train', meta: 'Run' },
{ title: 'run #7 · plan visit 2', meta: 'Step' },
{ title: 'run #7 · step 5 · output', meta: 'Field' },
{ title: 'release-report.txt', meta: 'Artifact' },
];
const RELEASE_BOUNDS = '6 results · cap 50 · scanned 3 steps · 2.0 KiB scanned · scan limit 1.0 MiB';
const BUILD_VISIT = 'build · visit 2';
// Flutter web renders a read-only SelectableText as a <textarea> whose value
// stays empty, so the section's JSON is pixels only; the section state itself
// is announced on the tile ("Expanded …" / "Collapsed …" aria-description).
const EXPANDED = 'Expanded';
const COLLAPSED = 'Collapsed';
let ws;
let send;
let session;
const evidence = { origin, captures: [], checks: [], defects: [] };
try {
const endpoint = await waitFor(
() => stderr.match(/DevTools listening on (ws:\/\/[^\s]+)/)?.[1],
'Chrome DevTools',
);
ws = new WebSocket(endpoint);
await once(ws, 'open');
let id = 0;
const pending = new Map();
const pageErrors = [];
ws.addEventListener('message', (event) => {
const message = JSON.parse(event.data);
if (message.method === 'Runtime.exceptionThrown') {
pageErrors.push(JSON.stringify(message.params?.exceptionDetails?.exception?.description ?? message.params));
}
if (message.id && pending.has(message.id)) {
pending.get(message.id)(message);
pending.delete(message.id);
}
});
send = (method, params = {}, sessionId) =>
new Promise((resolveCall) => {
const callId = ++id;
pending.set(callId, resolveCall);
ws.send(JSON.stringify({ id: callId, method, params, ...(sessionId ? { sessionId } : {}) }));
});
const created = await send('Target.createTarget', { url: 'about:blank' });
if (!created.result) throw new Error(`Target creation failed: ${JSON.stringify(created)}`);
const target = created.result.targetId;
session = (await send('Target.attachToTarget', { targetId: target, flatten: true })).result
.sessionId;
await send('Page.enable', {}, session);
await send('Runtime.enable', {}, session);
// A headless target can stay unfocused, which makes the browser refuse the
// search field's autofocus rather than testing the app.
await send('Page.bringToFront', {}, session);
const evaluate = async (expression) => {
const response = await send(
'Runtime.evaluate',
{ expression, returnByValue: true, awaitPromise: true },
session,
);
if (response.error) throw new Error(`Runtime.evaluate failed: ${JSON.stringify(response)}`);
if (response.result?.exceptionDetails) {
throw new Error(JSON.stringify(response.result.exceptionDetails));
}
return response.result?.result?.value;
};
const labels = () =>
evaluate(
"[...document.querySelectorAll('flt-semantics')].map(x => x.getAttribute('aria-label') || (x.innerText || '').trim()).filter(Boolean)",
);
// Everything the semantics tree announces: labels, leaf text, and the value
// of read-only text controls (the Cupertino Summary sections render their
// JSON in a read-only CupertinoTextField, which is an <input>/<textarea>).
const text = () =>
evaluate(`(() => {
const chunks = [];
for (const el of document.querySelectorAll('flt-semantics, [aria-label], input, textarea')) {
const label = el.getAttribute('aria-label');
if (label) chunks.push(label);
if (!el.children.length && el.innerText) chunks.push(el.innerText);
if ((el.tagName === 'INPUT' || el.tagName === 'TEXTAREA') && el.value) chunks.push(el.value);
}
if (document.body?.innerText) chunks.push(document.body.innerText);
return [...new Set(chunks)].join('\\n');
})()`);
const path = () => evaluate('location.pathname');
const hasText = async (needle) => (await text()).includes(needle);
const assertText = async (needle, what = needle) => {
try {
await waitFor(async () => hasText(needle), `${what} (${JSON.stringify(needle)})`);
} catch (error) {
throw new Error(`${error.message}\n--- page text ---\n${await text()}`);
}
};
const assertNoText = async (needle, what = needle) => {
if (await hasText(needle)) {
throw new Error(`${what} unexpectedly present: ${JSON.stringify(needle)}`);
}
};
// Rows announce their title, then their meta and snippet lines; click the
// most specific (shortest) tappable node whose text starts with the title.
const clickPrefix = async (prefix) => {
const clicked = await evaluate(`(() => {
const wanted = ${JSON.stringify(prefix)};
const nodes = [...document.querySelectorAll('flt-semantics')];
const candidates = nodes.filter((x) =>
(x.getAttribute('aria-label') || '').startsWith(wanted)
|| (x.innerText || '').trim().startsWith(wanted));
if (candidates.length === 0) return false;
candidates.sort((a, b) => (a.innerText || a.getAttribute('aria-label') || '').length
- (b.innerText || b.getAttribute('aria-label') || '').length);
candidates[0].click();
return true;
})()`);
if (!clicked) throw new Error(`No semantics label starting ${JSON.stringify(prefix)}: ${JSON.stringify(await labels())}`);
await delay(450);
};
// The shell's Search action is a real button; a pointer event keeps user
// activation so the page may then focus the search field.
const tapExact = async (label) => {
const point = await evaluate(`(() => {
const wanted = ${JSON.stringify(label)};
const nodes = [...document.querySelectorAll('flt-semantics')];
const node = nodes.find(x => x.getAttribute('aria-label') === wanted)
|| nodes.find(x => (x.innerText || '').trim() === wanted);
if (!node) return null;
const rect = node.getBoundingClientRect();
return { x: rect.left + rect.width / 2, y: rect.top + rect.height / 2 };
})()`);
if (!point) throw new Error(`No semantics node for ${JSON.stringify(label)}: ${JSON.stringify(await labels())}`);
for (const type of ['mousePressed', 'mouseReleased']) {
await send('Input.dispatchMouseEvent', { type, x: point.x, y: point.y, button: 'left', clickCount: 1 }, session);
}
await delay(450);
};
const activate = async () => {
await waitFor(
() => evaluate("Boolean(document.querySelector('flt-semantics-placeholder'))"),
'Flutter semantics placeholder',
);
await evaluate("document.querySelector('flt-semantics-placeholder')?.click()");
await delay(900);
};
let viewport = { width: 390, height: 844 };
const setSize = async (width, height, scale = 1) => {
viewport = { width, height };
await send('Emulation.setDeviceMetricsOverride', {
width,
height,
deviceScaleFactor: scale,
mobile: width < 600,
}, session);
await delay(500);
};
const navigate = async (url, width, height) => {
await setSize(width, height);
const navigation = await send('Page.navigate', { url }, session);
if (navigation.error || navigation.result?.errorText) throw new Error(JSON.stringify(navigation));
await activate();
};
const capture = async (name) => {
await delay(350);
const data = (await send('Page.captureScreenshot', { format: 'png' }, session)).result.data;
const file = join(out, `${name}.png`);
await writeFile(file, Buffer.from(data, 'base64'));
evidence.captures.push(file);
return file;
};
// The Summary's Input/Output tiles: Material's ExpansionTile and the
// Cupertino expansion tile both announce their state as the first line of
// aria-description ("Expanded" / "Collapsed"); an expanded Cupertino tile
// merges with its read-only text child into one <textarea> node.
const tileState = (title) =>
evaluate(`(() => {
const wanted = ${JSON.stringify(title)};
const nodes = [...document.querySelectorAll('flt-semantics, textarea, input')];
const node = nodes.find((x) =>
(x.getAttribute('aria-label') || '').trim() === wanted
|| (x.children.length === 0 && (x.innerText || '').trim() === wanted));
if (!node) return null;
const description = node.getAttribute('aria-description') || '';
return description.split('\\n')[0].trim() || 'unknown';
})()`);
const waitForTile = async (title, state) => {
try {
await waitFor(async () => (await tileState(title)) === state, `the ${title} section to be ${state}`, 40);
} catch (error) {
throw new Error(`${error.message}\n--- tiles ---\nInput: ${await tileState('Input')}\nOutput: ${await tileState('Output')}\n--- page text ---\n${await text()}`);
}
};
// Scrolls the detail body to its end so an expanded section below the phone
// fold is inside the captured frame.
const scrollBodyToEnd = async () => {
for (let i = 0; i < 2; i++) {
await send('Input.dispatchMouseEvent', {
type: 'mouseWheel',
x: viewport.width / 2,
y: viewport.height * 0.6,
deltaX: 0,
deltaY: 3000,
}, session);
await delay(400);
}
};
const fieldFocused = () =>
evaluate(`(() => {
const active = document.activeElement;
return Boolean(active && (active.tagName === 'INPUT' || active.tagName === 'TEXTAREA'));
})()`);
const typeQuery = async (value) => {
if (!(await fieldFocused())) {
const focused = await evaluate(`(() => {
const nodes = [...document.querySelectorAll('input.flt-text-editing, textarea.flt-text-editing, input, textarea')];
const node = nodes.find((el) => el.checkVisibility?.() ?? true) || nodes.at(-1);
if (!node) return false; node.focus(); return true;
})()`);
if (!focused) throw new Error('Search field not exposed as a semantics input');
await delay(300);
}
const inserted = await send('Input.insertText', { text: value }, session);
if (inserted.error) throw new Error(JSON.stringify(inserted));
// 250ms debounce plus the fixture channel's 120ms reply.
await delay(900);
};
// Search is a transient page above the shell: the action opens /search and
// the Cupertino host mounts the field after its page transition.
const openSearch = async () => {
await tapExact('Search');
await waitFor(async () => (await path()) === '/search', 'the /search route');
await waitFor(fieldFocused, 'the focused search field', 60).catch(() => {});
};
const searchFor = async (query, expectRow) => {
await openSearch();
await typeQuery(query);
await assertText(expectRow, `the ${JSON.stringify(query)} result row`);
};
const assertReleaseResults = async (host) => {
const body = await text();
const missing = [];
for (const row of RELEASE_ROWS) {
if (!body.includes(row.title)) missing.push(row.title);
if (!body.includes(row.meta)) missing.push(`meta ${row.meta}`);
}
if (!body.includes(RELEASE_BOUNDS)) missing.push(RELEASE_BOUNDS);
if (!body.includes('ship the release')) missing.push('field snippet "ship the release"');
if (!body.includes('Kind')) missing.push('Kind control');
evidence.checks.push({
check: `${host}: the search panel lists one hit per kind for "release"`,
rows: RELEASE_ROWS.map((row) => row.title),
bounds: RELEASE_BOUNDS,
missing,
});
if (missing.length) throw new Error(`Search results missing ${JSON.stringify(missing)}`);
};
const assertRun7 = async () => {
await waitFor(async () => (await path()) === '/runs/7', 'the /runs/7 route from search');
await assertText('Release train run', 'the run detail title');
if (await hasText('Kind: All kinds') || await hasText('Search pipelines, runs and artifacts')) {
throw new Error('Search stayed open above the run detail');
}
};
const assertFieldSummary = async (host) => {
await assertRun7();
await waitForTile('Output', EXPANDED);
await waitForTile('Input', COLLAPSED);
await assertNoText('Back to timeline', 'a visit pane');
await assertNoText(BUILD_VISIT, 'a visit pane');
evidence.checks.push({
check: `${host}: a field hit opens the Summary with only Output expanded`,
path: await path(),
output: await tileState('Output'),
input: await tileState('Input'),
visitPane: false,
});
await scrollBodyToEnd();
};
const assertArtifactVisit = async (host) => {
await assertRun7();
await assertText(BUILD_VISIT, 'the recorded build visit');
await assertText('Back to timeline', 'the visit pane');
// The visit pane has its own Input/Output section strip; the Summary's
// prompt block is the text that only the Summary tab shows.
await assertNoText('Original prompt', 'the Summary tab');
evidence.checks.push({
check: `${host}: an artifact hit opens the visit that recorded it`,
path: await path(),
visit: BUILD_VISIT,
summaryShown: false,
});
};
const assertPlainSummary = async (host, what) => {
await assertRun7();
await assertText('Original prompt', 'the plain Summary');
await waitForTile('Input', COLLAPSED);
await waitForTile('Output', COLLAPSED);
await assertNoText(BUILD_VISIT, 'a visit pane');
await assertNoText('Back to timeline', 'a visit pane');
evidence.checks.push({
check: `${host}: ${what} selects the plain Summary`,
path: await path(),
input: await tileState('Input'),
output: await tileState('Output'),
});
await scrollBodyToEnd();
};
// A retired field selects the plain Summary; after an earlier field hit on
// the same mounted run, a section that hit expanded must not stay open, or
// the "plain" Summary still shows the previous match's section. The state
// is recorded rather than asserted so the probe can still capture every
// required state; a carried-over section is reported as a defect.
const recordRetiredAfterField = async (host) => {
await assertRun7();
await assertText('Original prompt', 'the Summary');
await assertNoText('Back to timeline', 'a visit pane');
const state = { input: await tileState('Input'), output: await tileState('Output') };
const check = {
check: `${host}: a retired field hit after an output field hit selects the plain Summary`,
path: await path(),
...state,
expected: { input: COLLAPSED, output: COLLAPSED },
};
evidence.checks.push(check);
if (state.input !== COLLAPSED || state.output !== COLLAPSED) {
evidence.defects.push({
...check,
note: 'The earlier field hit\'s expanded section is kept when the retired hit resolves to the plain Summary on the same mounted detail.',
});
}
await scrollBodyToEnd();
};
// The app follows the system colour scheme; pin it so captures are
// deterministic (light first, one dark pass below).
const emulateScheme = (value) => send('Emulation.setEmulatedMedia', {
features: [{ name: 'prefers-color-scheme', value }],
}, session);
await emulateScheme('light');
await navigate(`${origin}/`, 390, 844);
const contentType = await evaluate("fetch('/').then(r => r.headers.get('content-type'))");
evidence.checks.push({ check: 'index content-type is text/html', value: contentType });
if (!String(contentType).toLowerCase().startsWith('text/html')) {
throw new Error(`Root index is not text/html: ${contentType}`);
}
const routeType = await evaluate("fetch('/runs/7').then(r => r.headers.get('content-type'))");
if (!String(routeType).toLowerCase().startsWith('text/html')) {
throw new Error(`Route path is not served as the page: ${routeType}`);
}
// ---- Material host, phone ------------------------------------------------
await navigate(`${origin}/runs`, 390, 844);
await assertText('Release train run', 'the Runs list');
await searchFor('release', 'run #7 · step 5 · output');
await assertReleaseResults('material phone');
await capture('search-selection-material-phone-results');
// An artifact hit opens the visit that recorded it.
await clickPrefix('release-report.txt');
await assertArtifactVisit('material phone');
await capture('search-selection-material-phone-artifact-visit');
// The retired field replaces the open visit with the plain Summary.
await searchFor('retired', 'run #7 · step 5 · evidence');
await clickPrefix('run #7 · step 5 · evidence');
await assertPlainSummary('material phone', 'a retired field hit');
await capture('search-selection-material-phone-retired-summary');
// An artifact whose recorded visit the run never recorded (sequence 9).
await searchFor('release', 'release-report.txt');
await clickPrefix('release-report.txt');
await assertArtifactVisit('material phone (reopened)');
await searchFor('missing', 'missing.log');
await clickPrefix('missing.log');
await assertPlainSummary('material phone', 'an unknown-visit artifact hit');
await capture('search-selection-material-phone-unknown-visit-summary');
// A field hit opens the Summary at its matched section.
await searchFor('release', 'run #7 · step 5 · output');
await clickPrefix('run #7 · step 5 · output');
await assertFieldSummary('material phone');
await capture('search-selection-material-phone-field-summary');
// A retired hit after that field hit, on the same mounted detail.
await searchFor('retired', 'run #7 · step 5 · evidence');
await clickPrefix('run #7 · step 5 · evidence');
await recordRetiredAfterField('material phone');
await capture('search-selection-material-phone-retired-after-field-summary');
// Reopening the run from its list is an ordinary visit: nothing replays.
await searchFor('release', 'release-report.txt');
await clickPrefix('release-report.txt');
await assertArtifactVisit('material phone (before reopen)');
await tapExact('Back to list');
await waitFor(async () => (await path()) === '/runs', 'the Runs list');
await clickPrefix('Release train run');
await waitFor(async () => (await path()) === '/runs/7', 'the reopened run');
await assertText('Original prompt', 'the reopened Summary');
await assertNoText(BUILD_VISIT, 'a replayed visit pane');
evidence.checks.push({ check: 'material phone: a consumed selection does not replay on reopen', path: await path() });
// A step hit names the run's recorded visit by stage and server visit
// number. plan visit 1 is sequence 1; the pane must say `plan · visit 1`.
await searchFor('first', 'run #7 · plan visit 1');
await clickPrefix('run #7 · plan visit 1');
await waitFor(async () => (await path()).startsWith('/runs/7'), 'the step route');
await delay(600);
const stepPath = await path();
const stepSearch = await evaluate('location.search');
const stepBody = await text();
const stepVisit = stepBody.match(/plan · visit (\d+)/)?.[0] ?? null;
const stepCheck = {
check: 'material phone: a step hit for plan visit 1 opens plan · visit 1',
route: `${stepPath}${stepSearch}`,
opened: stepVisit,
expected: 'plan · visit 1',
};
evidence.checks.push(stepCheck);
if (stepVisit !== 'plan · visit 1') {
evidence.defects.push({
...stepCheck,
note: 'The hit carries the server visit number (1-based); the route query is zero-based.',
});
}
await capture('search-selection-material-phone-step-visit');
// ---- Material host, desktop ----------------------------------------------
await navigate(`${origin}/runs`, 1280, 800);
await assertText('Release train run', 'the Runs list');
await searchFor('release', 'run #7 · step 5 · output');
await assertReleaseResults('material desktop');
await capture('search-selection-material-desktop-results');
await clickPrefix('release-report.txt');
await assertArtifactVisit('material desktop');
await capture('search-selection-material-desktop-artifact-visit');
await searchFor('retired', 'run #7 · step 5 · evidence');
await clickPrefix('run #7 · step 5 · evidence');
await assertPlainSummary('material desktop', 'a retired field hit');
await capture('search-selection-material-desktop-retired-summary');
await searchFor('release', 'run #7 · step 5 · output');
await clickPrefix('run #7 · step 5 · output');
await assertFieldSummary('material desktop');
await capture('search-selection-material-desktop-field-summary');
// Dark scheme is the host theme, not a fixture flag: the app follows the
// system scheme, so emulate it for one results and one selection capture.
await emulateScheme('dark');
await navigate(`${origin}/runs`, 390, 844);
await assertText('Release train run', 'the Runs list');
await searchFor('release', 'run #7 · step 5 · output');
await capture('search-selection-material-phone-dark-results');
await clickPrefix('run #7 · step 5 · output');
await assertFieldSummary('material phone dark');
await capture('search-selection-material-phone-dark-field-summary');
await emulateScheme('light');
// ---- Cupertino host, phone -----------------------------------------------
await navigate(`${origin}/runs?platform=cupertino`, 390, 844);
await assertText('Release train run', 'the Runs list');
await searchFor('release', 'run #7 · step 5 · output');
await assertReleaseResults('cupertino phone');
await assertText('Kind: All kinds', 'the Cupertino kind control');
await capture('search-selection-cupertino-phone-results');
await clickPrefix('release-report.txt');
await assertArtifactVisit('cupertino phone');
await capture('search-selection-cupertino-phone-artifact-visit');
await searchFor('retired', 'run #7 · step 5 · evidence');
await clickPrefix('run #7 · step 5 · evidence');
await assertPlainSummary('cupertino phone', 'a retired field hit');
await capture('search-selection-cupertino-phone-retired-summary');
await searchFor('release', 'release-report.txt');
await clickPrefix('release-report.txt');
await assertArtifactVisit('cupertino phone (reopened)');
await searchFor('missing', 'missing.log');
await clickPrefix('missing.log');
await assertPlainSummary('cupertino phone', 'an unknown-visit artifact hit');
await capture('search-selection-cupertino-phone-unknown-visit-summary');
await searchFor('release', 'run #7 · step 5 · output');
await clickPrefix('run #7 · step 5 · output');
await assertFieldSummary('cupertino phone');
await capture('search-selection-cupertino-phone-field-summary');
await searchFor('retired', 'run #7 · step 5 · evidence');
await clickPrefix('run #7 · step 5 · evidence');
await recordRetiredAfterField('cupertino phone');
await capture('search-selection-cupertino-phone-retired-after-field-summary');
// ---- Cupertino host, desktop ---------------------------------------------
await navigate(`${origin}/runs?platform=cupertino`, 1280, 800);
await assertText('Release train run', 'the Runs list');
await searchFor('release', 'run #7 · step 5 · output');
await assertReleaseResults('cupertino desktop');
await capture('search-selection-cupertino-desktop-results');
await clickPrefix('release-report.txt');
await assertArtifactVisit('cupertino desktop');
await capture('search-selection-cupertino-desktop-artifact-visit');
await searchFor('release', 'run #7 · step 5 · output');
await clickPrefix('run #7 · step 5 · output');
await assertFieldSummary('cupertino desktop');
await capture('search-selection-cupertino-desktop-field-summary');
evidence.pageErrors = pageErrors;
console.log(JSON.stringify(evidence, null, 2));
if (evidence.defects.length) process.exitCode = 2;
} catch (error) {
// Keep the failing frame beside the captures so the failure is inspectable.
try {
const data = (await send('Page.captureScreenshot', { format: 'png' }, session)).result.data;
await writeFile(join(out, 'failure.png'), Buffer.from(data, 'base64'));
} catch {
/* The failure is reported below either way. */
}
console.log(JSON.stringify({ ...evidence, failure: String(error.message) }, null, 2));
throw error;
} finally {
ws?.close();
chrome.kill('SIGTERM');
await Promise.race([once(chrome, 'exit'), delay(3000)]);
server.close();
await rm(profile, { recursive: true, force: true });
}
+469
View File
@@ -0,0 +1,469 @@
// Standalone fixture entry for real-browser Search result-selection evidence.
//
// Build and probe (from app/):
// flutter build web --target tool/search_selection_probe_entry.dart \
// --output build/probe-search-selection
// node tool/search_selection_probe.mjs --root build/probe-search-selection \
// --out build/search-selection-probe
//
// This is the normal application composition (`PipelinesApp`) over the shared
// transport, with the socket replaced by an in-process channel that answers
// the real `search`, `get_run`, `list_runs`, `list_pipelines`, `get_pipeline`
// and `ping` requests with server-shaped fixtures. Nothing here opens a
// network connection or touches a live server; the fixtures exist only so the
// shell's search → run-detail selection can be rendered and captured.
import 'dart:async';
import 'dart:convert';
import 'package:pipelines_app/application/app_url_strategy.dart';
import 'package:pipelines_app/application/pipelines_app.dart';
import 'package:pipelines_app/application/server_endpoint.dart';
import 'package:pipelines_app/core/api_gateway.dart';
import 'package:pipelines_app/core/transport/pipelines_client.dart';
import 'package:flutter/widgets.dart';
import 'package:web_socket_channel/web_socket_channel.dart';
/// The run every run-scoped hit names. Its recorded sequences are deliberately
/// not its row indexes (plan 5, build 2, plan 1), exactly like the unit
/// fixture, so a selection has to resolve the recorded sequence.
JsonMap probeRunDetail() => {
'id': 7,
'title': 'Release train run',
'name': 'release-train',
'status': 'succeeded',
'created_at': '2026-10-20T08:12:00Z',
'finished_at': '2026-10-20T08:42:00Z',
'workspace': '/srv/probe/release',
'input': {'prompt': 'ship it'},
'output': {'result': 'ok'},
'steps': [
{
'stage': 'plan',
'sequence': 5,
'status': 'completed',
'outcome': 'success',
'target': '#success',
'started_at': '2026-10-20T08:30:00Z',
'finished_at': '2026-10-20T08:31:00Z',
'diagnostic': 'release plan accepted',
},
{
'stage': 'build',
'sequence': 2,
'status': 'completed',
'outcome': 'success',
'target': 'plan',
'started_at': '2026-10-20T08:16:00Z',
'finished_at': '2026-10-20T08:24:00Z',
'artifacts': [
{
'name': 'release-report.txt',
'content_type': 'text/plain',
'size_bytes': 12,
'url': '/artifacts/7/2/release-report.txt',
},
],
},
{
'stage': 'plan',
'sequence': 1,
'status': 'completed',
'outcome': 'success',
'target': 'build',
'started_at': '2026-10-20T08:12:00Z',
'finished_at': '2026-10-20T08:14:00Z',
},
],
};
List<JsonMap> probeRuns() => [
{
'id': 7,
'title': 'Release train run',
'name': 'release-train',
'status': 'succeeded',
'created_at': '2026-10-20T08:12:00Z',
'finished_at': '2026-10-20T08:42:00Z',
'pipeline_id': 24,
},
{
'id': 6,
'title': 'Nightly docs run',
'name': 'nightly-docs',
'status': 'failed',
'created_at': '2026-10-19T22:00:00Z',
'finished_at': '2026-10-19T22:03:00Z',
'pipeline_id': 22,
},
];
List<JsonMap> probePipelines() => [
{
'id': 24,
'name': 'Release train',
'version': 5,
'stages': 2,
'created_at': '2026-09-28T10:12:00',
'updated_at': '2026-10-20T09:18:00',
'scheduled': false,
'schedule_count': 0,
'enabled_schedule_count': 0,
},
{
'id': 22,
'name': 'Nightly documentation',
'version': 6,
'stages': 1,
'created_at': '2026-10-02T18:20:00',
'updated_at': '2026-10-18T16:05:00',
'scheduled': true,
'schedule_count': 1,
'enabled_schedule_count': 1,
},
];
const _releaseSource = 'name: Release train\n'
'stages:\n'
' plan:\n'
' type: model\n'
' output: plan\n'
' build:\n'
' type: model\n'
' output: release\n';
JsonMap probePipelineDetail(int id, int requestedVersion) {
final summary = probePipelines().firstWhere(
(pipeline) => pipeline['id'] == id,
orElse: () => probePipelines().first,
);
final latest = summary['version'] as int;
final version = requestedVersion > 0 ? requestedVersion : latest;
return {
...summary,
'version': version,
'latest_version': latest,
'version_created_at': '2026-10-18T16:05:00',
'content_hash': 'sha256:probe-$id-v$version',
'source': _releaseSource,
'versions': [
for (var v = 1; v <= latest; v++)
{'version': v, 'stages': summary['stages'], 'created_at': '2026-10-0${v}T10:00:00'},
],
};
}
/// One recorded hit per result kind, each with the text a server would have
/// matched (`haystack`). The payloads are server-shaped `search_results` rows:
/// the `target` tokens are the ones `internal/protocol/search.go` records.
///
/// `release` matches every kind once; `retired` matches only the retired-field
/// hit; `missing` matches only an artifact whose recorded visit (sequence 9)
/// the run never recorded; `first` matches only the plan visit 1 step hit.
const List<({String haystack, JsonMap hit})> probeHits = [
(
haystack: 'Release train pipeline definition',
hit: {
'kind': 'pipeline',
'target': 'pipeline:24',
'score': 3,
'recency': '2026-10-20T09:18:00Z',
'pipeline': {'id': 24, 'name': 'Release train', 'version': 5, 'stages': 2},
},
),
(
haystack: 'name: Nightly documentation release notes',
hit: {
'kind': 'version',
'target': 'pipeline:22:v6',
'score': 2,
'recency': '2026-10-18T16:05:00Z',
'version': {
'pipeline_id': 22,
'version': 6,
'content_hash': '9f2c1ad',
'created_at': '2026-10-18T16:05:00Z',
},
},
),
(
haystack: 'Release train run',
hit: {
'kind': 'run',
'target': 'run:7',
'score': 2,
'recency': '2026-10-20T08:42:00Z',
'run': {
'id': 7,
'pipeline_name': 'Release train',
'status': 'succeeded',
'workspace': '/srv/probe/release',
'created_at': '2026-10-20T08:12:00Z',
'finished_at': '2026-10-20T08:42:00Z',
},
},
),
(
haystack: 'release plan accepted',
hit: {
'kind': 'step',
'target': 'run:7:5',
'score': 1,
'recency': '2026-10-20T08:31:00Z',
'step': {
'run_id': 7,
'sequence': 5,
'stage': 'plan',
'visit': 2,
'outcome': 'success',
'target': '#success',
'diagnostic': 'release plan accepted',
},
},
),
(
haystack: 'ship the release',
hit: {
'kind': 'field',
'target': 'run:7:5:output',
'score': 1,
'matched_field': 'output',
'recency': '2026-10-20T08:31:00Z',
'field': {
'run_id': 7,
'sequence': 5,
'field': 'output',
'snippet': 'ship the release',
},
},
),
(
haystack: 'release-report.txt',
hit: {
'kind': 'artifact',
'target': '/artifacts/7/2/release-report.txt',
'score': 1,
'recency': '2026-10-20T08:24:00Z',
'artifact': {
'run_id': 7,
'sequence': 2,
'name': 'release-report.txt',
'content_type': 'text/plain',
'size_bytes': 12,
'url': '/artifacts/7/2/release-report.txt',
},
},
),
(
haystack: 'retired evidence note',
hit: {
'kind': 'field',
'target': 'run:7:5:evidence',
'score': 1,
'matched_field': 'evidence',
'recency': '2026-10-20T08:31:00Z',
'field': {
'run_id': 7,
'sequence': 5,
'field': 'evidence',
'snippet': 'retired evidence note',
},
},
),
(
haystack: 'missing.log',
hit: {
'kind': 'artifact',
'target': '/artifacts/7/9/missing.log',
'score': 1,
'recency': '2026-10-20T08:40:00Z',
'artifact': {
'run_id': 7,
'sequence': 9,
'name': 'missing.log',
'content_type': 'text/plain',
'size_bytes': 4096,
'url': '/artifacts/7/9/missing.log',
},
},
),
(
haystack: 'first plan visit',
hit: {
'kind': 'step',
'target': 'run:7:1',
'score': 1,
'recency': '2026-10-20T08:14:00Z',
'step': {
'run_id': 7,
'sequence': 1,
'stage': 'plan',
'visit': 1,
'outcome': 'success',
'target': 'build',
},
},
),
];
/// Answers one decoded request envelope with the server-shaped reply, or null
/// for a send-only request the real server does not acknowledge.
JsonMap? probeReply(JsonMap request) {
final id = request['id'];
final type = '${request['type']}';
JsonMap error(String message) => {'id': id, 'type': 'error', 'error': message};
switch (type) {
case 'subscribe':
case 'unsubscribe':
return null;
case 'ping':
return {
'id': id,
'type': 'pong',
'build': {'version': 'Search selection probe fixture'},
'time': '2026-10-20T09:00:00Z',
};
case 'list_runs':
return {'id': id, 'type': 'runs', 'runs': probeRuns(), 'usage': null};
case 'get_run':
if (request['run_id'] != 7) {
return error('Fixture run ${request['run_id']} is unavailable');
}
return {'id': id, 'type': 'run', 'run_detail': probeRunDetail()};
case 'list_pipelines':
return {'id': id, 'type': 'pipelines', 'pipelines': probePipelines()};
case 'get_pipeline':
final pipelineId = request['pipeline_id'];
if (pipelineId is! int ||
!probePipelines().any((pipeline) => pipeline['id'] == pipelineId)) {
return error('Fixture pipeline $pipelineId is unavailable');
}
final version = request['version'];
return {
'id': id,
'type': 'pipeline',
'detail': probePipelineDetail(pipelineId, version is int ? version : 0),
};
case 'search':
// The app must send exactly the browser's SearchRequest body; anything
// else is answered with a visible refusal so the capture cannot pass.
final search = request['search'];
if (search is! Map ||
search.length != 3 ||
search['query'] is! String ||
search['kind'] is! String ||
search['limit'] != 0) {
return error('Fixture refused a malformed search request: $search');
}
final query = (search['query'] as String).trim().toLowerCase();
final kind = search['kind'] as String;
final results = [
for (final entry in probeHits)
if ((kind.isEmpty || entry.hit['kind'] == kind) &&
(query.isEmpty || entry.haystack.toLowerCase().contains(query)))
entry.hit,
];
return {
'id': id,
'type': 'search_results',
'search_results': {
'query': search['query'],
'kind': kind,
'results': results,
'truncated': false,
'scanned_steps': 3,
'scanned_bytes': 2048,
'cap': 50,
'scan_bound_bytes': 1048576,
},
};
default:
return error('Fixture does not implement $type');
}
}
/// An in-process websocket channel: every frame the client sends is decoded
/// and answered asynchronously with [probeReply], as the server would over the
/// wire. `ready` completes at once so the shared client reports connected.
class _FixtureChannel implements WebSocketChannel {
_FixtureChannel();
final _incoming = StreamController<dynamic>();
late final _FixtureSink _sink = _FixtureSink(this);
@override
Stream<dynamic> get stream => _incoming.stream;
@override
WebSocketSink get sink => _sink;
@override
Future<void> get ready => Future<void>.value();
@override
String? get protocol => null;
@override
int? get closeCode => null;
@override
String? get closeReason => null;
void receive(dynamic data) {
if (data is! String) return;
final decoded = jsonDecode(data);
if (decoded is! Map<String, dynamic>) return;
final reply = probeReply(decoded);
if (reply == null) return;
Future<void>.delayed(const Duration(milliseconds: 120), () {
if (!_incoming.isClosed) _incoming.add(jsonEncode(reply));
});
}
Future<void> close() async {
if (!_incoming.isClosed) await _incoming.close();
}
@override
dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}
class _FixtureSink implements WebSocketSink {
_FixtureSink(this.channel);
final _FixtureChannel channel;
@override
void add(dynamic event) => channel.receive(event);
@override
void addError(Object error, [StackTrace? stackTrace]) => throw error;
@override
Future<void> addStream(Stream<dynamic> stream) async {
await for (final item in stream) {
add(item);
}
}
@override
Future<void> close([int? closeCode, String? closeReason]) => channel.close();
@override
Future<void> get done => channel._incoming.done;
@override
dynamic noSuchMethod(Invocation invocation) => super.noSuchMethod(invocation);
}
void main() {
final params = Uri.base.queryParameters;
final cupertino = params['platform'] == 'cupertino';
useAppUrlStrategy();
runApp(
PipelinesApp(
server: ServerEndpoint.resolve(base: Uri.base, web: true),
client: PipelinesClient(channelFactory: (_) => _FixtureChannel()),
cupertino: cupertino,
),
);
}