Use bounded verification wrappers in the app landing gate

This commit is contained in:
2026-11-05 14:03:00 +00:00
parent c7e57973a0
commit ea98a7ae2d
4 changed files with 176 additions and 47 deletions
@@ -0,0 +1,19 @@
# Verification gate and app workflow documentation
`tool/check_full.sh` now calls the bounded analyzer and memory-watchdog wrappers,
from the app directory regardless of the caller's working directory. It preserves
their actual nonzero statuses instead of accepting success-looking text from a
failed command. Wrapper summaries remain in `build/check/`; they name their own
full logs. Analysis failure stops before tests run, including a missing wrapper.
`pipelines/README.md` now directs app tasks to the registered Mobile workflow and
read-only inspection of its current version. The checked-in flutter-fix,
flutter-design and flutter-preview definitions are labelled historical, and the
README distinguishes code, checks, server landing, preview publication and the
installed web root. No pipeline definition was edited or published.
Checks: five Python orchestration fixtures pass; they cover successful wrapper
calls from the repository root, misleading success text on analysis/test failure,
a watchdog kill status, and a missing wrapper. `sh -n tool/check_full.sh` and
`git diff --check` pass. No real Flutter suite was run in this lane; the manager
runs integrated analysis and the full memory-watched suite after merging.
+53 -26
View File
@@ -1,31 +1,58 @@
# Flutter app pipelines
# Flutter app workflows
The published Pipelines workflows for `app/`, kept here so their history lives
with the code they work on. Publish a change with
`pipelines validate pipelines/<file>.md` then
`pipelines submit pipelines/<file>.md --name "<Name>"`.
Use the registered **Mobile** pipeline for one app behaviour, bug or visual
change. Its current flow prepares the worktree, implements the change, reviews
the diff, reviews rendered UI when affected, and commits a candidate for the
server's delivery checks and landing gate.
| Pipeline | File | One run does |
| --- | --- | --- |
| Flutter fix | `flutter-fix.md` | One behaviour or bug: Ground → Red → Code → Review → Verify → Commit |
| Flutter design | `flutter-design.md` | One visual defect on one screen: Capture → Compare → Code → Recapture → UIReview → Verify → Commit |
| Flutter preview | `flutter-preview.md` | Publish main to the preview: Gate → Publish → Smoke |
Inspect the live definition before a run; the server's registered version is
authoritative:
Running one item:
```sh
pipelines list
pipelines show Mobile
```
1. From the repository root, `git worktree add -b flutter/<slug> tmp/flutter-<slug> main`
and `flutter pub get` in its `app/`.
2. `pipelines run "Flutter fix" --workspace tmp/flutter-<slug>/app --prompt "<one behaviour, with what you already know>"`.
Flutter design takes the screen and its routes; Flutter preview runs in the main
checkout's `app/` and takes a short name for the rollback directory.
3. A run ends with a local commit on its branch. The manager reads the diff,
merges the branch into main with `--no-ff`, runs `sh tool/check_full.sh` on main,
and publishes with Flutter preview.
Run one item in its own repository-root worktree (not inside `app/`), keeping
other worktrees available for independent work:
Every stage runs on `openai/gpt-6-luna` (app/AGENTS.md). Stages share the repository's
own tools: `tool/check_full.sh` (analyze plus the full suite, passing only on
`All tests passed!`), `tool/shoot_real_app.mjs` (phone and desktop captures of the
real app against the loopback fixture) and `tool/publish_preview.sh` (build, then
swap `build/web` by rename with a rollback directory). No stage restarts a server,
uses the shared Pipelines server for captures, uses `git stash`, or adds AI
attribution to commits.
```sh
git worktree add -b flutter/<slug> tmp/flutter-<slug> main
cd tmp/flutter-<slug>
(cd app && /Users/alan/development/flutter/bin/flutter pub get)
pipelines run Mobile "<one observable change and its acceptance criteria>"
```
Use Luna for pipeline stages unless the user requests another model. Verify
the exact provider/model in `pipelines show Mobile`; do not copy model IDs from
the old files here. Put task-specific requirements in the run prompt. Creating
or publishing a workflow version requires an explicit pipeline-change request.
For evidence and recovery, use `pipelines inspect <run-id> --json`. A failed run
keeps its branch and edits; inspect those before retrying or taking over. A
successful model stage is not proof of landing: verify the delivery-check
results and main's ancestry after the server's landing gate finishes. **App
change** is the separate Caplens workflow; it is not the Flutter client workflow.
## Verification and delivery
From `app/`, use `tool/memwatch/full.sh test/<area>` and
`tool/analyze.sh <changed paths>` while editing. For integrated acceptance,
`sh tool/check_full.sh` runs bounded analysis followed by the full suite through
the memory watchdog. It fails on the actual command exit status and records
wrapper summaries under `build/check/`. Never run a bare `flutter test`.
Rendered changes also need interaction and inspected captures with
`tool/capture/` (see its README); test-device requirements are in `app/AGENTS.md`.
Code, checks, landing, publication and device installation are separate steps.
Publish only when requested, preserve a rollback, and do not restart shared
services over active work. `tool/publish_preview.sh` updates the checkout preview;
the installed web root uses the installation's delivery path. APK publication
uses `tool/publish_apk.sh` or the installation's `--apk` flow.
## Historical definitions
`flutter-fix.md`, `flutter-design.md` and `flutter-preview.md` preserve earlier
workflow definitions. They are not the current registered Mobile definition and
must not be submitted just to start or repair an ordinary run. In particular,
their names, model IDs, prompts and manual verification steps may be stale.
+24 -21
View File
@@ -1,25 +1,28 @@
#!/bin/sh
# Landing gate for app/ changes: analyze, then the full test suite.
#
# Usage (from app/): sh tool/check_full.sh
# Exits 0 only when `flutter analyze` reports no issues and the full suite
# prints `All tests passed!`. A piped `tail` cannot hide a failure here: the
# verdict comes from the suite's own final line, which is echoed with the
# totals. Logs go to build/check/ for the caller's evidence.
set -u
FLUTTER="${FLUTTER:-flutter}"
# Usage: sh app/tool/check_full.sh (or sh tool/check_full.sh from app/).
# The wrappers bound output and test memory. Trust their exit statuses, never
# success-looking text in a failed command's log. Keep their summaries in
# build/check/; each summary also names the underlying full log.
set -eu
cd "$(dirname "$0")/.."
mkdir -p build/check
"$FLUTTER" analyze > build/check/analyze.log 2>&1
if ! grep -q "No issues found" build/check/analyze.log; then
grep -E " • " build/check/analyze.log
echo "CHECK FAILED: flutter analyze reported issues (build/check/analyze.log)"
exit 1
fi
"$FLUTTER" test > build/check/test.log 2>&1
verdict=$(grep -E "All tests passed!|Some tests failed" build/check/test.log | tail -1)
if ! echo "$verdict" | grep -q "All tests passed!"; then
grep -E "\[E\]" build/check/test.log | head -20
echo "CHECK FAILED: $verdict (build/check/test.log)"
exit 1
fi
echo "CHECK PASSED: analyze clean; $verdict"
check() {
name=$1
log=$2
shift 2
if "$@" > "$log" 2>&1; then
cat "$log"
else
status=$?
cat "$log"
echo "CHECK FAILED: $name exited $status ($log)"
return "$status"
fi
}
check analyze build/check/analyze.log tool/analyze.sh
check test build/check/test.log tool/memwatch/full.sh test
echo "CHECK PASSED: analyze and memory-watched test suite passed"
+80
View File
@@ -0,0 +1,80 @@
"""Exercise the landing gate's orchestration without launching Flutter tests."""
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
class CheckFullTests(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix="check-full-")
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.app = self.root / "app"
self.tool = self.app / "tool"
(self.tool / "memwatch").mkdir(parents=True)
shutil.copyfile(Path(__file__).with_name("check_full.sh"),
self.tool / "check_full.sh")
self.wrapper("analyze.sh", "analyze", "ANALYZE")
self.wrapper("memwatch/full.sh", "memwatch", "TEST")
def wrapper(self, path, name, prefix):
script = self.tool / path
script.write_text(
"#!/bin/sh\n"
f'printf "%s\\n" "{name}:$*:$PWD" >> "$CALL_LOG"\n'
f'printf "%s\\n" "${{{prefix}_OUTPUT:-bounded summary}}"\n'
f'exit "${{{prefix}_STATUS:-0}}"\n')
script.chmod(0o755)
def run_gate(self, **overrides):
env = dict(os.environ, CALL_LOG=str(self.root / "calls"), **overrides)
result = subprocess.run(["sh", str(self.tool / "check_full.sh")],
cwd=self.root, env=env, capture_output=True,
text=True)
calls = self.root / "calls"
return result, calls.read_text().splitlines() if calls.exists() else []
def test_success_uses_both_wrappers_from_app_and_keeps_summaries(self):
result, calls = self.run_gate()
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(calls, [f"analyze::{self.app}",
f"memwatch:test:{self.app}"])
self.assertIn("CHECK PASSED", result.stdout)
for name in ("analyze", "test"):
self.assertEqual((self.app / f"build/check/{name}.log").read_text(),
"bounded summary\n")
def test_analysis_failure_cannot_be_hidden_by_success_text(self):
result, calls = self.run_gate(ANALYZE_STATUS="14",
ANALYZE_OUTPUT="No issues found")
self.assertEqual(result.returncode, 14)
self.assertEqual(len(calls), 1)
self.assertNotIn("CHECK PASSED", result.stdout)
self.assertFalse((self.app / "build/check/test.log").exists())
def test_test_failure_cannot_be_hidden_by_success_text(self):
result, calls = self.run_gate(TEST_STATUS="7",
TEST_OUTPUT="All tests passed!")
self.assertEqual(result.returncode, 7)
self.assertEqual(len(calls), 2)
self.assertIn("CHECK FAILED: test exited 7", result.stdout)
self.assertNotIn("CHECK PASSED", result.stdout)
def test_watchdog_kill_status_is_preserved(self):
result, _ = self.run_gate(TEST_STATUS="137", TEST_OUTPUT="KILLED")
self.assertEqual(result.returncode, 137)
self.assertNotIn("CHECK PASSED", result.stdout)
def test_missing_wrapper_fails_instead_of_claiming_success(self):
(self.tool / "memwatch/full.sh").unlink()
result, calls = self.run_gate()
self.assertNotEqual(result.returncode, 0)
self.assertEqual(len(calls), 1)
self.assertNotIn("CHECK PASSED", result.stdout)
if __name__ == "__main__":
unittest.main()