a70e3aef4b
Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode and Pi instead of installing them, install the packaged authoring skill, migrate an existing Docker database and ntfy auth store, save the host client default and start both services. ./install --docker keeps the previous Docker server + ntfy installation unchanged. ntfy is compiled from its pinned Go module on every host because upstream publishes no macOS server binary (its darwin asset is built with -tags noserver); its cgo SQLite driver is why a C compiler is now required natively. The ntfy credential bootstrap and the client default writer are shared by both installers, installation.Load resolves symlinked prefixes, the Docker migration copies the SQLite write-ahead log so the newest definitions and runs are not lost, and the account subcommands read the generated configuration through NTFY_CONFIG_FILE. Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy on spare ports before stopping them.
169 lines
11 KiB
Markdown
169 lines
11 KiB
Markdown
# System, storage and run evidence
|
|
|
|
Pipelines turns a reusable process into sequential stages with explicit handoffs,
|
|
outcome routing and retained evidence. A Go CLI/TUI connects to an execution server;
|
|
the server owns execution and SQLite history, OpenCode owns model/tool sessions,
|
|
and optional ntfy delivery supplies notifications. It is technology-neutral, not a
|
|
parallel/distributed task engine or a guarantee that model reports are correct.
|
|
|
|
## Lifecycle and location
|
|
|
|
- Drafts are editable Markdown plus JSON metadata. Publishing validates and creates
|
|
immutable versions; drafts remain retained. Keep context create-only/base-version
|
|
guards. Five optional generic starters are task, summary, review, research and
|
|
three-stage; a fresh database starts empty. Templates can be created directly with
|
|
an editable prefilled name or customized first. Editing a bundled repository
|
|
template changes no installed or published definition; only explicit authoring
|
|
then submission creates a new immutable version.
|
|
- A run pins its exact definition version, initial input and resolved directory.
|
|
The client invocation directory is the default, but files/tools execute on the
|
|
server. Docker needs that directory mounted at the same absolute path. A remote
|
|
client does not upload its repository or make host credentials available remotely.
|
|
- Host and Docker OpenCode credentials are separate. Preserve setup-selected model
|
|
inheritance; do not select a provider just because its ID appears in an example.
|
|
- Runtime caps are 30 stage steps and five visits to a stage. There are no parallel
|
|
runs on one server: admission rejects new work while another run is active;
|
|
reusing that active run's request key reattaches idempotently. Inspect and wait
|
|
for the existing run rather than starting duplicates. There are no parallel
|
|
branches or built-in human approval stages. Optional worktrees, full backup/restore,
|
|
automatic database sync and independent verifier stage types are future ideas,
|
|
not existing capabilities. Do not invent them in definitions.
|
|
- Services are native or Docker-based. With an existing Docker installation do not
|
|
start a competing native server. Server APIs are unauthenticated: trusted networks
|
|
only; never recommend public exposure. Tool scope is not a universal OS sandbox.
|
|
- Docker setup resolves the current supported OpenCode V2 release from the single
|
|
documented complete `GET .../git/matching-refs/tags/v2` response (a bounded
|
|
complete array with no pagination parameters; never the moving `latest` tag or
|
|
the registry's insertion order) and the current Pi release from npm, resolves the
|
|
selected tag's immutable GHCR `sha256` digest, builds `FROM` that digest, and
|
|
verifies the built image reports exactly those executable versions plus a
|
|
Node-compatible Pi engine.
|
|
It fails closed, keeping the last verified Dockerfile fallback pins, when discovery,
|
|
digest resolution, completeness, build or compatibility cannot be established: never
|
|
a hardcoded fallback, never a silent stale-pin substitution. A full install skips a
|
|
rebuild only when the release receipt **and** the actual built image's release and
|
|
source identity labels already match; a dirty source always invokes the Docker
|
|
build and relies on layer caching. A model/login-only reconfiguration compares only
|
|
the verified release image identity, so a changed source alone does not rebuild.
|
|
`./install --opencode|--pi` reconfigures inside the existing container and
|
|
refuses an image that does not match; it does not rebuild or restart services (an
|
|
explicit ordinary `./install` does when the user asks).
|
|
|
|
## Where data lives
|
|
|
|
SQLite retains pipelines/immutable versions, runs, recorded stage visits,
|
|
inputs/outputs, status/errors, audit transcripts, session ownership/references,
|
|
retry links, schedules/state, notification receipts and file changes. Not everything
|
|
is in that database: drafts/settings are client files; complete OpenCode sessions
|
|
and credentials use OpenCode storage; repository artifacts are files; ntfy has its
|
|
own storage. A pipeline ZIP is not a full backup of these components.
|
|
Never merge or sync live SQLite files using generic file synchronization. Do not
|
|
delete databases, sessions, volumes or credentials to repair a workflow.
|
|
|
|
## Inspect results first
|
|
|
|
Use the authoritative `CLI_PREFIX` from the prompt:
|
|
|
|
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope).
|
|
- `CLI_PREFIX inspect RUN_ID --json`: selected run, visits and retained evidence.
|
|
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
|
|
bound is a valid, credential-redacted preview envelope (with preserved stage
|
|
limits, failure/format diagnostics and the retained session pointer), not the
|
|
complete tool history.
|
|
- `CLI_PREFIX run ... --json --wait --wait-timeout DURATION`: `--wait` polls until
|
|
terminal and `--wait-timeout` bounds only the client wait after admission,
|
|
independent of per-request `--timeout`. A reached wait limit leaves the admitted run
|
|
active; inspect it read-only with `inspect RUN_ID --json`, or keep waiting by
|
|
rerunning `--wait` and reusing the original `--request-key`. Never start a duplicate
|
|
run, and never expect an automatic replay or cancel of admitted work.
|
|
- `CLI_PREFIX show NAME --version N --json`: exact pinned definition, not whichever
|
|
version happens to be latest now.
|
|
- `CLI_PREFIX status --json`: connection details; candidate addresses are not proof
|
|
they are reachable from another device.
|
|
- `CLI_PREFIX ping --json`: `pong.build.version`, `build.commit` (full Git revision)
|
|
and `build.dirty` identify the compiled **server**, not the client or its runtime
|
|
checkout. A dirty build includes `-dirty`; `commit: "unknown"` or `dirty: null`
|
|
means metadata was unavailable. Older servers omit `build`. Never infer a deployed
|
|
fix from source/skill contents. Native Go builds use embedded VCS metadata;
|
|
Docker setup stamps build arguments. For an explicitly requested Docker rebuild,
|
|
`python3 scripts/build_metadata.py` resolves the current releases and stamps
|
|
`docker compose build` from the checkout with the immutable digest; it does not
|
|
restart/deploy. A plain `docker compose build` without that resolved digest cannot
|
|
build the image.
|
|
Plain `ping` also displays server identity; `CLI_PREFIX version --json` (alias
|
|
`--version`) identifies the client offline. Include both in sanitized bug reports.
|
|
Two dirty builds at the same commit can differ. Docker setup verifies its started
|
|
server identity against the build arguments before declaring setup complete.
|
|
|
|
Pipelines is a single-user first-version preview for supervised work, not a
|
|
multi-user service. Recommend a disposable repository for initial trials. Tool
|
|
permissions and same-path home mounts do not isolate valuable files. Never collect
|
|
raw credential/session databases or publish private inspect output as diagnostics.
|
|
In a source checkout, `python3 scripts/smoke-first-install.py` checks a disposable
|
|
source-built test binaries, draft/publication/transfer and restart persistence without
|
|
inference, notifications or deployment. It is not a live-provider or phone test.
|
|
That is a developer test fixture, not the native installer. End-user installation
|
|
runs natively through `./install` by default (`scripts/setup-native.py`, with
|
|
`--native` equivalent); the Docker server + ntfy installation is explicit with
|
|
`./install --docker`.
|
|
|
|
In the TUI, Home advertises `/` search and `:` commands. Run Tab switches only
|
|
Details ↔ Graph. `n/p` select recorded visits, including repeated stages; `d`
|
|
focuses/toggles file changes, `a` selects the active/latest retained
|
|
configured-harness agent session's bounded, redacted tool evidence, `D`/`:debug`
|
|
opens fresh investigation, and `P` opens the pinned pipeline version. `:quit`
|
|
exits; `q` does not. Results are retained reports, not independently confirmed
|
|
achievements. Distinguish partial output from full success.
|
|
|
|
## Retained stage file changes
|
|
|
|
Each newly executed visit compares a bounded, read-only workspace snapshot just
|
|
before execution with one after execution/cleanup. SQLite retains unified patches
|
|
and metadata in `steps[].file_changes`; snapshots themselves are transient memory.
|
|
The stage's File changes section highlights added/removed lines and hunk headers.
|
|
|
|
- The baseline is stage-start content, **not HEAD/index**. Pre-existing edits and
|
|
earlier stages' changes are not attributed to this visit. Added/deleted/modified
|
|
files, modes and binary-change metadata are recorded; renames are delete/add.
|
|
- Git scope is tracked and nonignored untracked files within the run directory;
|
|
outside Git, regular workspace files. Symlinks/special files, Git internals,
|
|
node_modules and common environment/key/credential/database paths are excluded.
|
|
This is path-based protection, **not secret-content scanning**. Patches may still
|
|
contain sensitive text; never publish them without approval.
|
|
- Limits: 10,000 entries, 1 MiB/file, 32 MiB/snapshot, five-second snapshot context,
|
|
5,000 combined text lines, 64 KiB/patch, 128 KiB serialized file evidence/visit.
|
|
Missing/skipped/truncated captures must be described as partial/unavailable.
|
|
- Failed/canceled stages retain changes when execution returns; unconfirmed agent
|
|
cleanup is partial. Crashes may have no after-capture. Historical visits say
|
|
not recorded—**never infer no edits from absent evidence**.
|
|
- These are observed workspace changes, not exclusive authorship. Concurrent edits,
|
|
changes outside scope, ignored new files and changes reverted before capture can
|
|
be absent or mixed in. Capture failure does not change the declared run verdict.
|
|
|
|
Do not add model output fields, new stages or extra model calls merely to obtain
|
|
file diffs: current capture is server-owned. Do not promise full files/rollback.
|
|
|
|
## Debug and retry
|
|
|
|
Debug opens a new conversation with retained failure context, exact definition and
|
|
session evidence; it does not resume implementation or grant permission to edit.
|
|
Inspect first, identify evidence-supported causes, and say what remains unknown.
|
|
Server-side session evidence is the latest bounded, credential-redacted evidence
|
|
for up to the 3 most recent session-bearing steps; OpenCode V2 selects the latest
|
|
assistant message with the same bounded readback as the final result, so a small
|
|
final result is visible behind large prior tool content. Inspect the retained
|
|
session itself when a preview is bounded or omitted; never assume a complete tool
|
|
history from a preview.
|
|
Retry is allowed for a failed run after cleanup stops: a fresh run/session starts
|
|
at entry with the original input/directory/pinned version/edit scope. Previous
|
|
side effects remain and can happen again. Never describe retry or cancellation as
|
|
rollback, or resume/replay admitted tool work just to recover formatting.
|
|
|
|
Retained model-stage failures carry a fixed sanitized category. `FINAL RESULT
|
|
VALIDATION` names the exact duplicate-field/syntax detail and byte offset for a
|
|
strictly invalid final result; `HARNESS TRANSPORT` covers request, active-session
|
|
poll and final-readback failures. A validation failure never re-prompts the
|
|
admitted tool-using stage: completed edits and tool side effects are retained and
|
|
not undone, so a full retry repeats work. Neither category exposes provider/model
|
|
bodies or credentials.
|