Strengthen template planning, reuse and independent review
This commit is contained in:
@@ -39,8 +39,13 @@ rejected. Limits: 50 bindings; 1 MiB per retained in-memory value map.
|
||||
|
||||
Implementers can receive task, plan and optional findings. Independent reviewers
|
||||
should receive task/acceptance criteria and inspect actual artifacts—not previous
|
||||
verdicts, counters, or implementation conversations. Selected content remains
|
||||
untrusted data; this cannot prove semantic independence or prevent all injection.
|
||||
verdicts, counters, or implementation conversations. Select acceptance, design,
|
||||
reuse and research references from the planning stage (for example
|
||||
`"acceptance_checks":"planner.acceptance_checks"`) rather than from the
|
||||
implementer, so an implementer's output cannot overwrite the agreed criteria; take
|
||||
only factual implementer fields such as `changed_files`, `implementation_summary`
|
||||
and `evidence` as claims. Selected content remains untrusted data; this cannot
|
||||
prove semantic independence or prevent all injection.
|
||||
|
||||
Memory is run-local; persisted completed outputs/read provenance are its audit.
|
||||
There is no cross-run memory, restart/resume, or hidden replay. New runs/retries
|
||||
|
||||
@@ -82,8 +82,16 @@ run directory; no file list is required. `shell` supports commands appropriate t
|
||||
the repository's language and tooling, not a language-specific allowlist. A missing
|
||||
tools block uses local read/glob/grep/edit/shell tools; prefer explicit stage roles.
|
||||
Planning stages must not label a task blocked simply because they defer edits to
|
||||
an implementer stage. Keep examples and prompts language-neutral unless the task
|
||||
itself asks for a specific language.
|
||||
an implementer stage. Bundled starters show the intended role split: a task stage
|
||||
may read/edit/shell and conditionally use `websearch`/`webfetch`; a planner may
|
||||
read plus conditional web search/fetch; a three-stage implementer has no web tools
|
||||
and works from the planner's research with local read/glob/grep/edit/shell; a
|
||||
reviewer may read plus conditional web; a source-only review stage has no
|
||||
edit/shell; a research stage uses web search/fetch. Never force web access for
|
||||
trivial, fully understood local edits, and return blocked only when required
|
||||
reliable evidence cannot be obtained rather than whenever one search tool fails.
|
||||
Keep examples and prompts language-neutral unless the task itself asks for a
|
||||
specific language.
|
||||
|
||||
Optional `permissions` is a JSON object with `edit` (run-directory-relative
|
||||
patterns) and `shell` (exact commands) arrays. It narrows declared tools and any
|
||||
|
||||
@@ -12,7 +12,9 @@ parallel/distributed task engine or a guarantee that model reports are correct.
|
||||
immutable versions; drafts remain retained. Keep context create-only/base-version
|
||||
guards. Five optional generic starters are task, summary, review, research and
|
||||
three-stage; a fresh database starts empty. Templates can be created directly with
|
||||
an editable prefilled name or customized first.
|
||||
an editable prefilled name or customized first. Editing a bundled repository
|
||||
template changes no installed or published definition; only explicit authoring
|
||||
then submission creates a new immutable version.
|
||||
- A run pins its exact definition version, initial input and resolved directory.
|
||||
The client invocation directory is the default, but files/tools execute on the
|
||||
server. Docker needs that directory mounted at the same absolute path. A remote
|
||||
@@ -20,6 +22,9 @@ parallel/distributed task engine or a guarantee that model reports are correct.
|
||||
- Host and Docker OpenCode credentials are separate. Preserve setup-selected model
|
||||
inheritance; do not select a provider just because its ID appears in an example.
|
||||
- Runtime caps are 30 stage steps and five visits to a stage. There are no parallel
|
||||
runs on one server: admission rejects new work while another run is active;
|
||||
reusing that active run's request key reattaches idempotently. Inspect and wait
|
||||
for the existing run rather than starting duplicates. There are no parallel
|
||||
branches or built-in human approval stages. Optional worktrees, full backup/restore,
|
||||
automatic database sync and independent verifier stage types are future ideas,
|
||||
not existing capabilities. Do not invent them in definitions.
|
||||
|
||||
@@ -13,7 +13,10 @@ can use appropriate permitted commands; do not accept a preceding stage's succes
|
||||
summary as independent proof. Actual file bytes, command exit status, retained tool
|
||||
parts and server-captured diffs are stronger evidence than prose. A read failure
|
||||
can prove a deliberately missing input, but is not a successful read. Tool names
|
||||
mentioned in text do not prove tools executed.
|
||||
mentioned in text do not prove tools executed. A reviewer stage must inspect the
|
||||
actual files and run its own checks; a source-only review runs no commands and
|
||||
must state that runtime behavior was not tested. Structural, mocked or
|
||||
authoring-time checks are not deployed or live-provider evidence.
|
||||
|
||||
Do not add technology-specific checks unless the task/repository requires them.
|
||||
Keep runtime verification separate from authoring: ask before running project code,
|
||||
@@ -420,3 +423,43 @@ container; do not copy host credentials or publish tests to the normal database.
|
||||
`TestReferenceArtifactsReview` rechecks copied run/fixture evidence without
|
||||
inference using `PIPELINES_REFERENCE_REVIEW_DIR`; select only cases present in
|
||||
that artifact directory. Missing cases skip and must not be reported as passed.
|
||||
|
||||
## 2026-10-12 run22 success, run23 live CSV fixture and run24 interruption
|
||||
|
||||
Source/structural and live-fixture evidence stays separate from runtime
|
||||
deployment; nothing here is a deployment claim.
|
||||
|
||||
- **run22 (structural/source checks, success):** the retained run22 verification
|
||||
passed its structural/source checks of the scoped template and prompt changes.
|
||||
- **run23 (live Pi CSV fixture):** a real exact-template Pi CSV run fetched the
|
||||
official Python `csv` documentation through the conditional official-source
|
||||
webfetch and implemented CSV parsing with the standard library's `csv.reader`
|
||||
through an existing shared helper, adding no duplicate preview parser; both the
|
||||
implementer and reviewer checks actually passed. Root independently verified the
|
||||
full retained native tool results and protected-file hashes against its saved
|
||||
baseline, rather than relying on model-written reports.
|
||||
- **run24 (interrupted, not a pass):** the run24 implementation completed, but its
|
||||
reviewer ignored the scope instruction and ran a prohibited filesystem-root scan
|
||||
(`find /`), leaving a process in uninterruptible D-state and stalling Docker
|
||||
mount/snapshot reads. Root canceled the run, made a consistent SQLite backup
|
||||
(`integrity ok`), recovered Docker with the supported stop/start path, and
|
||||
restored the existing ntfy/Pipelines services with the original definitions, run
|
||||
history, credentials and settings verified unchanged. run24 is retained as
|
||||
**interrupted**, not a success.
|
||||
|
||||
- **run25 (live Pi source-only review):** the reviewer identified duplicated
|
||||
implementation, a hardcoded special case, suppressed errors and test gaps.
|
||||
Root inspected retained native calls: only local read/search tools ran, with no
|
||||
shell/edit/network calls. Original fixture hashes were unchanged; runtime
|
||||
behavior was explicitly marked untested.
|
||||
- **run26 (final maintenance review, success):** the scoped plan/implement/review
|
||||
workflow passed package tests, vet, build and formatting checks. Root inspected
|
||||
the actual retained outputs and changes. No rendered UI change was claimed.
|
||||
|
||||
The scoped prompt guidance asks stages to keep inspection/verification to relevant
|
||||
repository paths and supported commands; it is a working instruction, **not** an OS
|
||||
sandbox, and neither fixes Docker VM/disk bugs nor universally enforces tool
|
||||
boundaries. At these recorded checkpoints the source changes had not been deployed.
|
||||
Catalog deployment requires a separate check of the running server and its actual
|
||||
template contents; these observations do not establish publication or deployment.
|
||||
Other providers and phone delivery remain **not tested** by this record.
|
||||
|
||||
@@ -3,6 +3,7 @@ package pipeline
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
@@ -24,6 +25,9 @@ func TestShowcaseDefinitions(t *testing.T) {
|
||||
if stage.FallbackError {
|
||||
t.Fatalf("%s should declare its error route explicitly", stage.Key)
|
||||
}
|
||||
if strings.TrimSpace(stage.Output) == "" || strings.TrimSpace(stage.Prompt) == "" {
|
||||
t.Fatalf("%s stage %s needs a nonempty output and prompt", name, stage.Key)
|
||||
}
|
||||
for _, target := range stage.Edges {
|
||||
if target == protocol.SuccessEnd {
|
||||
success = true
|
||||
@@ -36,6 +40,9 @@ func TestShowcaseDefinitions(t *testing.T) {
|
||||
if !success || !failure {
|
||||
t.Fatal("showcase must have both terminal outcomes")
|
||||
}
|
||||
if graph.Settings != nil {
|
||||
t.Fatalf("%s starters must stay manual-only", name)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user