diff --git a/app/notes/verification-gate-and-workflow-docs.md b/app/notes/verification-gate-and-workflow-docs.md new file mode 100644 index 00000000..84bcd8a6 --- /dev/null +++ b/app/notes/verification-gate-and-workflow-docs.md @@ -0,0 +1,19 @@ +# Verification gate and app workflow documentation + +`tool/check_full.sh` now calls the bounded analyzer and memory-watchdog wrappers, +from the app directory regardless of the caller's working directory. It preserves +their actual nonzero statuses instead of accepting success-looking text from a +failed command. Wrapper summaries remain in `build/check/`; they name their own +full logs. Analysis failure stops before tests run, including a missing wrapper. + +`pipelines/README.md` now directs app tasks to the registered Mobile workflow and +read-only inspection of its current version. The checked-in flutter-fix, +flutter-design and flutter-preview definitions are labelled historical, and the +README distinguishes code, checks, server landing, preview publication and the +installed web root. No pipeline definition was edited or published. + +Checks: five Python orchestration fixtures pass; they cover successful wrapper +calls from the repository root, misleading success text on analysis/test failure, +a watchdog kill status, and a missing wrapper. `sh -n tool/check_full.sh` and +`git diff --check` pass. No real Flutter suite was run in this lane; the manager +runs integrated analysis and the full memory-watched suite after merging. diff --git a/app/pipelines/README.md b/app/pipelines/README.md index 09aca7a6..05c89d91 100644 --- a/app/pipelines/README.md +++ b/app/pipelines/README.md @@ -1,31 +1,58 @@ -# Flutter app pipelines +# Flutter app workflows -The published Pipelines workflows for `app/`, kept here so their history lives -with the code they work on. Publish a change with -`pipelines validate pipelines/.md` then -`pipelines submit pipelines/.md --name ""`. +Use the registered **Mobile** pipeline for one app behaviour, bug or visual +change. Its current flow prepares the worktree, implements the change, reviews +the diff, reviews rendered UI when affected, and commits a candidate for the +server's delivery checks and landing gate. -| Pipeline | File | One run does | -| --- | --- | --- | -| Flutter fix | `flutter-fix.md` | One behaviour or bug: Ground → Red → Code → Review → Verify → Commit | -| Flutter design | `flutter-design.md` | One visual defect on one screen: Capture → Compare → Code → Recapture → UIReview → Verify → Commit | -| Flutter preview | `flutter-preview.md` | Publish main to the preview: Gate → Publish → Smoke | +Inspect the live definition before a run; the server's registered version is +authoritative: -Running one item: +```sh +pipelines list +pipelines show Mobile +``` -1. From the repository root, `git worktree add -b flutter/ tmp/flutter- main` - and `flutter pub get` in its `app/`. -2. `pipelines run "Flutter fix" --workspace tmp/flutter-/app --prompt ""`. - Flutter design takes the screen and its routes; Flutter preview runs in the main - checkout's `app/` and takes a short name for the rollback directory. -3. A run ends with a local commit on its branch. The manager reads the diff, - merges the branch into main with `--no-ff`, runs `sh tool/check_full.sh` on main, - and publishes with Flutter preview. +Run one item in its own repository-root worktree (not inside `app/`), keeping +other worktrees available for independent work: -Every stage runs on `openai/gpt-6-luna` (app/AGENTS.md). Stages share the repository's -own tools: `tool/check_full.sh` (analyze plus the full suite, passing only on -`All tests passed!`), `tool/shoot_real_app.mjs` (phone and desktop captures of the -real app against the loopback fixture) and `tool/publish_preview.sh` (build, then -swap `build/web` by rename with a rollback directory). No stage restarts a server, -uses the shared Pipelines server for captures, uses `git stash`, or adds AI -attribution to commits. +```sh +git worktree add -b flutter/ tmp/flutter- main +cd tmp/flutter- +(cd app && /Users/alan/development/flutter/bin/flutter pub get) +pipelines run Mobile "" +``` + +Use Luna for pipeline stages unless the user requests another model. Verify +the exact provider/model in `pipelines show Mobile`; do not copy model IDs from +the old files here. Put task-specific requirements in the run prompt. Creating +or publishing a workflow version requires an explicit pipeline-change request. + +For evidence and recovery, use `pipelines inspect --json`. A failed run +keeps its branch and edits; inspect those before retrying or taking over. A +successful model stage is not proof of landing: verify the delivery-check +results and main's ancestry after the server's landing gate finishes. **App +change** is the separate Caplens workflow; it is not the Flutter client workflow. + +## Verification and delivery + +From `app/`, use `tool/memwatch/full.sh test/` and +`tool/analyze.sh ` while editing. For integrated acceptance, +`sh tool/check_full.sh` runs bounded analysis followed by the full suite through +the memory watchdog. It fails on the actual command exit status and records +wrapper summaries under `build/check/`. Never run a bare `flutter test`. + +Rendered changes also need interaction and inspected captures with +`tool/capture/` (see its README); test-device requirements are in `app/AGENTS.md`. +Code, checks, landing, publication and device installation are separate steps. +Publish only when requested, preserve a rollback, and do not restart shared +services over active work. `tool/publish_preview.sh` updates the checkout preview; +the installed web root uses the installation's delivery path. APK publication +uses `tool/publish_apk.sh` or the installation's `--apk` flow. + +## Historical definitions + +`flutter-fix.md`, `flutter-design.md` and `flutter-preview.md` preserve earlier +workflow definitions. They are not the current registered Mobile definition and +must not be submitted just to start or repair an ordinary run. In particular, +their names, model IDs, prompts and manual verification steps may be stale. diff --git a/app/tool/check_full.sh b/app/tool/check_full.sh index c2c55bad..a5f1901e 100644 --- a/app/tool/check_full.sh +++ b/app/tool/check_full.sh @@ -1,25 +1,28 @@ #!/bin/sh # Landing gate for app/ changes: analyze, then the full test suite. # -# Usage (from app/): sh tool/check_full.sh -# Exits 0 only when `flutter analyze` reports no issues and the full suite -# prints `All tests passed!`. A piped `tail` cannot hide a failure here: the -# verdict comes from the suite's own final line, which is echoed with the -# totals. Logs go to build/check/ for the caller's evidence. -set -u -FLUTTER="${FLUTTER:-flutter}" +# Usage: sh app/tool/check_full.sh (or sh tool/check_full.sh from app/). +# The wrappers bound output and test memory. Trust their exit statuses, never +# success-looking text in a failed command's log. Keep their summaries in +# build/check/; each summary also names the underlying full log. +set -eu +cd "$(dirname "$0")/.." mkdir -p build/check -"$FLUTTER" analyze > build/check/analyze.log 2>&1 -if ! grep -q "No issues found" build/check/analyze.log; then - grep -E " • " build/check/analyze.log - echo "CHECK FAILED: flutter analyze reported issues (build/check/analyze.log)" - exit 1 -fi -"$FLUTTER" test > build/check/test.log 2>&1 -verdict=$(grep -E "All tests passed!|Some tests failed" build/check/test.log | tail -1) -if ! echo "$verdict" | grep -q "All tests passed!"; then - grep -E "\[E\]" build/check/test.log | head -20 - echo "CHECK FAILED: $verdict (build/check/test.log)" - exit 1 -fi -echo "CHECK PASSED: analyze clean; $verdict" + +check() { + name=$1 + log=$2 + shift 2 + if "$@" > "$log" 2>&1; then + cat "$log" + else + status=$? + cat "$log" + echo "CHECK FAILED: $name exited $status ($log)" + return "$status" + fi +} + +check analyze build/check/analyze.log tool/analyze.sh +check test build/check/test.log tool/memwatch/full.sh test +echo "CHECK PASSED: analyze and memory-watched test suite passed" diff --git a/app/tool/test_check_full.py b/app/tool/test_check_full.py new file mode 100644 index 00000000..6601e4d4 --- /dev/null +++ b/app/tool/test_check_full.py @@ -0,0 +1,80 @@ +"""Exercise the landing gate's orchestration without launching Flutter tests.""" +import os +from pathlib import Path +import shutil +import subprocess +import tempfile +import unittest + + +class CheckFullTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory(prefix="check-full-") + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.app = self.root / "app" + self.tool = self.app / "tool" + (self.tool / "memwatch").mkdir(parents=True) + shutil.copyfile(Path(__file__).with_name("check_full.sh"), + self.tool / "check_full.sh") + self.wrapper("analyze.sh", "analyze", "ANALYZE") + self.wrapper("memwatch/full.sh", "memwatch", "TEST") + + def wrapper(self, path, name, prefix): + script = self.tool / path + script.write_text( + "#!/bin/sh\n" + f'printf "%s\\n" "{name}:$*:$PWD" >> "$CALL_LOG"\n' + f'printf "%s\\n" "${{{prefix}_OUTPUT:-bounded summary}}"\n' + f'exit "${{{prefix}_STATUS:-0}}"\n') + script.chmod(0o755) + + def run_gate(self, **overrides): + env = dict(os.environ, CALL_LOG=str(self.root / "calls"), **overrides) + result = subprocess.run(["sh", str(self.tool / "check_full.sh")], + cwd=self.root, env=env, capture_output=True, + text=True) + calls = self.root / "calls" + return result, calls.read_text().splitlines() if calls.exists() else [] + + def test_success_uses_both_wrappers_from_app_and_keeps_summaries(self): + result, calls = self.run_gate() + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(calls, [f"analyze::{self.app}", + f"memwatch:test:{self.app}"]) + self.assertIn("CHECK PASSED", result.stdout) + for name in ("analyze", "test"): + self.assertEqual((self.app / f"build/check/{name}.log").read_text(), + "bounded summary\n") + + def test_analysis_failure_cannot_be_hidden_by_success_text(self): + result, calls = self.run_gate(ANALYZE_STATUS="14", + ANALYZE_OUTPUT="No issues found") + self.assertEqual(result.returncode, 14) + self.assertEqual(len(calls), 1) + self.assertNotIn("CHECK PASSED", result.stdout) + self.assertFalse((self.app / "build/check/test.log").exists()) + + def test_test_failure_cannot_be_hidden_by_success_text(self): + result, calls = self.run_gate(TEST_STATUS="7", + TEST_OUTPUT="All tests passed!") + self.assertEqual(result.returncode, 7) + self.assertEqual(len(calls), 2) + self.assertIn("CHECK FAILED: test exited 7", result.stdout) + self.assertNotIn("CHECK PASSED", result.stdout) + + def test_watchdog_kill_status_is_preserved(self): + result, _ = self.run_gate(TEST_STATUS="137", TEST_OUTPUT="KILLED") + self.assertEqual(result.returncode, 137) + self.assertNotIn("CHECK PASSED", result.stdout) + + def test_missing_wrapper_fails_instead_of_claiming_success(self): + (self.tool / "memwatch/full.sh").unlink() + result, calls = self.run_gate() + self.assertNotEqual(result.returncode, 0) + self.assertEqual(len(calls), 1) + self.assertNotIn("CHECK PASSED", result.stdout) + + +if __name__ == "__main__": + unittest.main()