Commit Graph

55 Commits

Author SHA1 Message Date
alan 865993b200 Rework omni search onto a typed search contract and add a TUI artifacts view
Protocol: replace the loose Query/Kind/Matches/Limit search fields with a Search request object and a SearchResults response, and have the server answer search from Store.Search over the shared run projections (runFrom/runStamps).

CLI: add the read-only 'pipelines search QUERY [--type KIND] [--limit N] [--json]' command with usage/help, and document the typed targets, ranking and bounds in docs/cli.md.

TUI: the '/' popup keeps type-to-search, up/down, Enter and Esc, renders one ranked list with kind badges and a bracketed (non-colour) match snippet, and Enter opens the matched pipeline, run, step, artifact or field; footer text updated to match.

TUI: add a dedicated run artifacts view ('A') with open/copy of recorded files, and move artifacts out of the run-detail File changes section (artifactCaptureNotes).

Tests: add Store.OpenReadOnly for read-only inspection of a live database plus pipeline/server/TUI search tests and artifacts tests, including env-gated live checks.
2026-10-16 23:48:38 +01:00
alan fae1005962 Allow launching runs with whole-run and per-stage model overrides
Add `pipelines run --model PROVIDER/MODEL` and repeatable
`--stage-model STAGE=PROVIDER/MODEL`, resolved with precedence
per-stage > whole-run > the model pinned in the pipeline version.

- A shared pure resolver (`pipeline.ParseModelOverride` /
  `pipeline.EffectiveModels`) owns the precedence and validation: every value
  must be a `provider/model` pair (`stage.ValidModel`), every stage key must
  exist in the pinned graph, an unknown key fails naming the valid keys, and a
  per-stage key that names a non-model stage is refused.
- The server validates against the pinned version before admission (no run row
  is ever inserted for an invalid request), persists `model`/`stage_models` on
  the run and `provider`/`model` on each step, applies the effective pair to a
  clone of the stage definition so the stage and its repair/formatter path share
  it, overlays the returned run graph with the effective pair (presentation
  only), and reuses the recorded overrides on retry.
- The run record echoes the overrides it was launched with and each step records
  the provider/model it actually ran with; the TUI Evidence pane renders that
  effective pair. The pinned pipeline version and `show --version N` are never
  modified.
- CLI help, `docs/cli.md` and the packaged authoring skill references document
  the flags and the precedence. A TUI override picker remains a documented
  follow-up; the CLI is the deliverable.

Tests: pure precedence/validation table tests, store round-trip and
request-key-mismatch plus migration, server admission/application/overlay/retry
with a controlled executor, CLI flag/help/malformed-value tests, and a TUI
evidence test.
2026-10-16 23:33:53 +01:00
alan aac1be18cc docs: document line-oriented model/agent result format 2026-10-16 21:49:14 +01:00
alan 8c1eea53a9 Report a stale client and refuse an unreadable newer server
Exchange a build/protocol identity so an older CLI can no longer talk to a
newer native server silently:

- protocol carries ProtocolVersion and every server frame (including the
  unsolicited welcome) is stamped with the server build and protocol, so a
  client that never pings still learns the server identity.
- buildinfo.Compare classifies the relationship as InSync/Warn/Refuse and
  names both builds; the client Watch reports one warning per connection and
  returns a hard error when the server speaks a newer, uninterpretable
  protocol so the caller never acts on the frame.
- client.ReceiveNotice decodes the raw frame so an unmodeled top-level field
  produces a one-time "unrecognized field(s)" notice instead of being dropped
  silently (the case that hid the artifacts field).
- CLI routes warnings to stderr, keeps machine JSON stdout clean and fails
  non-zero on a refused mismatch; ping/connection print the client/server
  relationship. The TUI surfaces the same notices through the shared log.
- scripts/setup-native.py now warns when an older `pipelines` earlier on PATH
  shadows prefix/bin/pipelines, naming both paths and versions, without
  installing or modifying anything else.
- Docs, the packaged debugging skill reference and README no longer imply the
  installed CLI always matches the server.

Tests cover the comparison table, unknown-field detection and caching, the
server identity stamping on pong and welcome, the CLI/TUI reporting paths and
the installer PATH shadowing warning.
2026-10-16 18:37:01 +01:00
alan a6aff47117 Add a human decision stage that pauses a run for an operator's choice
A new registered stage type `human` (internal/stage/human.go) has no model,
agent, tools or configs: its required `prompt` block carries the operator
instructions and its non-error edges are the selectable options. It parses,
validates, renders, exports and imports like every other type (no parser
special-case, no config keys).

At runtime the runner parks the run instead of executing: the step and the run
persist `waiting` with self-contained {prompt, options} evidence, and the run
returns without starting later stages, capturing file diffs, scanning artifacts,
holding a goroutine/timer or applying a stage timeout. The wait is durable: a
restart leaves the waiting rows untouched (RecoverRuns excludes them) and a
rebuilt runtime over the same database still accepts the decision.

`pipelines decide <run-id> [stage] --option KEY [--feedback TEXT] [--json]` and
the TUI `c` picker resolve the decision from the pinned graph in one store
transaction, record the choice and feedback as the step's evidence and resume
along edges[KEY]. An unknown option is refused with the valid list, a stage that
is not waiting is refused, and a repeated decision for an already-decided visit
does not double-apply. Cancel and retry keep working while waiting and after a
decision; a human stage is never killed by a stage timeout or max_visits, and a
`notify` policy can fire an `on=waiting` event.

Docs: the authoring skill references (format.md, operations.md), docs/cli.md and
README describe the stage, the command and the TUI picker. Tests cover stage
validation, the parsing round-trip, the store transitions, a websocket wait/decide
integration, cancel/retry, waiting notifications and a persistence-layer rebuild
(second Open + newRunner over one database, no process restart).
2026-10-16 16:47:28 +01:00
alan f3e0593f4d Let runs carry an operator title shown in the CLI and TUI
Add a bounded, never-empty title to every run. `pipelines run --title "..."`
sets it explicitly and skips model generation; otherwise the server generates
it from the prompt through the shared summariser and falls back to the prompt's
first line, recording title_source (model/fallback/explicit) and title_model.

- protocol/store: RunSummary.Title plus Label(), a set_run_title message, title
  columns with an idempotent migration, CreateRunTitled, and SetRunTitle renames.
- summarise: Title/GenerateTitle/DeriveTitle/ExplicitTitle bounded to TitleCap.
- runner: generate the title outside the admission lock and only for a request
  that starts a run; a replay reuses the stored title and an explicit title
  skips the model entirely.
- CLI: `--title` on run and a `title <run-id> "<title>"` rename command; human
  `runs` and `inspect` show the title beside the pipeline name and version.
- TUI: the run list row, run detail header, Home run cards and last-run line,
  and the pipeline Results line show the title via Label().

Tests cover storage/migration/fallback, explicit-title and rename, the
end-to-end CLI flow, and title rendering; CLI and operations docs updated.
2026-10-16 13:36:48 +01:00
alan 9f88e4ff71 Recover malformed model-stage results with a bounded, work-preserving repair
A model stage whose final result was malformed failed the whole run even when
the work and evidence had survived, because the only recovery was an isolated
formatter call on a fixed 30-second deadline that could time out on its own
(run 55, ui_reviewer of pipeline 9 v6).

Add an ordered, deterministic local repair pass (internal/stage/result_repair.go)
that runs before any model call: (a) append missing closing delimiters via the
existing closeJSONContainers; (b) remove one unambiguous trailing comma before a
closing } or ]; (c) insert exactly the single comma/colon the JSON parser reported
at its byte offset. Every candidate must pass losslessSyntaxRepair(original,
candidate) and decodeModelResult before it is accepted, so no field value is
invented, removed, reordered or rewritten. Ambiguous payloads (an unescaped quote
or colon inside a string value, or a stray delimiter with no single completion)
return not-ok and fall through to the bounded model re-emit rather than guessing.
stageResultText still strips surrounding prose and a single code fence first;
model_prompt.go now also strips prose around a single balanced-but-malformed
object span so the deterministic pass sees just the object.

Replace the fixed 30-second formatter deadline with repairDeadline(payloadBytes,
remaining), which grows monotonically with payload size, has a floor, and is
clamped to the attempt's remaining budget minus a safety margin, so the stage
context remains the outer bound and a large result is not guaranteed to time out.
The one-request-per-attempt, 32 KiB bound and the deny-all, retained, new-session
property of RepairFormat are unchanged for OpenCodeClient and PiClient.

Record an explicit recovery status on every terminal path: repaired, partial or
unrecoverable, plus a byte offset. A candidate that is valid JSON but still fails
envelope validation is marked partial and never returned as a completed result;
the unrecoverable path retains the bounded original payload, the precise parse
error (including its byte offset) and surfaces the status in the FINAL RESULT
VALIDATION diagnostic.

safeToReplay is intentionally unchanged. The deterministic local repair and the
fresh deny-all, no-tools, retained formatter session are independent of the
stage's tool set and never re-prompt or replay the admitted agent session, so a
shell-enabled review stage gains malformed-result recovery without any replay of
admitted work. Risk: no change to mutation safety; the only added cost is one
bounded formatter call, already budgeted. Replayability is not extended for shell
stages.

Docs and packaged skill references (docs/model-results.md,
internal/author/skill/references/{format,debugging,verification}.md) describe the
three-tier recovery, the status words and the derived deadline; the safeToReplay
decision and its risk are recorded there alongside the commit.

Tests add mocked, offline coverage for the deterministic and partial paths, a
repairDeadline monotonicity/floor/clamp table, and a shell-stage regression
proving a recoverable malformed result does not fail the run or replay tool work.
2026-10-16 13:02:53 +01:00
alan 914345cb31 Move harness profiles between host and Docker and stop silent draft retargeting
Native installs left Docker-era harness profiles in harnesses.json, so every
authoring launch failed trying to reach a Docker service, and a retained draft
bound to the old Docker server was silently projected onto the configured one.

- harness: add Profile.Mode and VerifyDockerLink as the single mode/verification
  helpers. `harness setup --host` configures or converts a host profile and
  clears docker_directory; reconfiguring a Docker-linked profile re-verifies the
  recorded directory still owns compose.yaml and fails naming it instead of
  saving a link that cannot launch. Setup's summary and `harness list` (via the
  new List/ListView shape) report the resulting mode and, for Docker, the
  directory, without changing the stored Config schema.
- setup-native.py: after build/install-skill it writes host (no docker_directory)
  profiles for the detected harnesses, defaulting to the detected OpenCode profile
  rather than a stale pi default, taking models from --model or the existing valid
  profile and never inventing one; it clears a stale Docker link and preserves
  unrelated keys/profiles. Detection-only: no harness install, upgrade or download.
- draft: add Draft.Resolve, the shared mismatch helper, which projects a draft
  onto the configured server and returns a dual-URL notice. CLI validate/submit
  and the TUI draft browser use it so a foreign draft is never silently
  retargeted; the TUI pre-wraps the notice to the panel width.
- Tests cover the Docker->host and host->host transitions, the missing-directory
  failure, installer write/migration (Python), and the CLI/TUI notice, using the
  real config file path in a temp dir.
- Update help text, README, docs and the packaged authoring skill reference to
  describe host as the default and the Docker-linked case.
2026-10-15 19:53:10 +01:00
alan f57457aec9 Make agent-stage retries safe and retain timeout failure evidence
Fix four stage failure-handling and observability defects (A2, A3, A6, A9).

- A2: add safeToReplay(def) as the single definition of retryable post-admission
  work - only an explicit tools block that selects neither edit nor shell. The
  agent/pi WorkStarted guards now break only when the stage is not safe to
  replay, so read-only stages genuinely retry a post-admission timeout while
  mutating and legacy-bundle stages still never replay admitted work.
- A6: re-emit a malformed final result on an explicit read-only stage up to the
  retry budget, preserving format_repairs evidence; mutating stages keep the
  immediate return and their session.
- A3: timed-out stages retain usable evidence. waitForSessionIdle records one
  bounded sanitized OperationEvidence on deadline; the agent deferred cleanup
  does a bounded best-effort latest-assistant readback and settles knowable V2
  usage; RetainSession adds a retained_session pointer to failed step
  transcripts so the session is visible beneath the BoundTranscript threshold.
- A9: a stage outcome routed to #failure now yields a non-empty run diagnostic
  ("stage <name> returned outcome <outcome>") while preferring an existing step
  diagnostic.

Adds focused stage and server tests for retry safety, malformed-result
re-emission, timeout evidence retention and run-level diagnostics; updates the
packaged skill references and docs/cli.md, docs/model-results.md.
2026-10-15 18:27:22 +01:00
alan 0f5a188c6a Record and serve per-stage artifacts
Every agent stage now writes into a per-run/step artifacts directory under
the data dir (<data>/artifacts/<run>/<step>), passed to Pi as
PIPELINES_ARTIFACTS_DIR, exposed on the OpenCode client env seam, and stated
in the agent prompt boilerplate so agents save images, screenshots, reports
and logs there.

After each step the server scans that directory with a bounded, symlink- and
traversal-safe inventory (os.OpenRoot + Lstat + O_NOFOLLOW) recording name,
size_bytes, content type (extension then 512-byte sniff fallback) and URL,
enforcing max files, per-file bytes and total bytes while counting skipped
and truncated entries. A missing, unreadable or oversized directory yields
only a best-effort diagnostic and never changes the step outcome.

The list is carried on protocol.RunStep as Artifacts []StageArtifact plus
skipped/truncated counters (omitempty, backwards compatible), persisted in a
new step_artifacts table keyed (run_id, sequence) alongside the
notifications/file-changes tables and loaded by GetRun.

GET /artifacts/{run}/{step}/{name} on the existing handler serves the bytes
with http.ServeContent, so Content-Type, Content-Length, HEAD and Range
(206/416) come from the stdlib; unknown runs/steps, absolute paths,
traversal, symlink escapes and non-regular files are refused.

The TUI lists each selected step's artifacts (name + human size) in the
retained-evidence section and adds an artifacts summary in the style of the
file-changes summary. Adds docs/stage-artifacts.md linked from README and an
artifacts subsection to the packaged operations skill reference.

Tests cover the scan records and caps, missing/oversized directories,
HTTP HEAD/Range/traversal/symlink/unknown refusals and protocol round-trips.
2026-10-15 17:51:18 +01:00
alan 2618f0d244 Allow concurrent runs in different workspaces
Track each active run's resolved workspace alongside its cancel function
in a single runner.active struct map. A new request key is now refused
only when an already-active run is changing the same resolved workspace;
the refusal names the conflicting run id and advises waiting or using a
different workspace. Workspace resolution is unchanged (default cwd,
absolute request workspace, EvalSymlinks), so two spellings of one
directory still compare equal. The conflict check and registration stay
inside the existing r.mu critical section, so concurrent starts for one
workspace cannot both pass. Idempotent repeats of an active request key
and the restart-recovery guard are unchanged.

Add concurrency_test.go covering concurrent different-workspace runs,
same-workspace refusal, symlink/default-cwd equivalence, idempotent
repeats, restart recovery and retry workspace scoping; strengthen the
runner_test.go refusal assertion. Update docs/cli.md,
docs/first-version-testing.md and the packaged operations skill reference
to describe per-workspace concurrency.

Assisted-by: DeepSeek V4.1 Flash
2026-10-15 15:08:44 +01:00
alan 8adc6b8080 Bind the native server to the LAN so the phone client can reach it
The native install hardcoded -addr 127.0.0.1, so the server was unreachable from
the phone: the emulator worked because it proxies to the host loopback, but a
real device could not connect even though ntfy (bound to 0.0.0.0:8090) worked.

The bind address is now a manifest setting (server_address() in manage.py, with
older manifests still defaulting to loopback) and setup-native.py records it,
defaulting to 0.0.0.0 like the Docker install did and offering
--bind-address 127.0.0.1 for a loopback-only server. An invalid address is
rejected, and setup prints the phone client URL.

Verified: scripts test suite 116 tests OK, and after reinstalling the server
listens on *:8080 and answers {"healthy":true} on 127.0.0.1, 10.0.0.75,
10.0.0.249 and the tailnet address.
2026-10-15 01:57:27 +01:00
alan a70e3aef4b Install Pipelines natively by default on macOS and Linux
Add scripts/setup-native.py and make ./install (and --native) build the server, the CLI
and ntfy from source into ~/.local/share/pipelines, detect an existing host OpenCode
and Pi instead of installing them, install the packaged authoring skill, migrate an
existing Docker database and ntfy auth store, save the host client default and start
both services. ./install --docker keeps the previous Docker server + ntfy
installation unchanged.

ntfy is compiled from its pinned Go module on every host because upstream publishes
no macOS server binary (its darwin asset is built with -tags noserver); its cgo
SQLite driver is why a C compiler is now required natively. The ntfy credential
bootstrap and the client default writer are shared by both installers,
installation.Load resolves symlinked prefixes, the Docker migration copies the
SQLite write-ahead log so the newest definitions and runs are not lost, and the
account subcommands read the generated configuration through NTFY_CONFIG_FILE.

Verified on macOS: 115 Python tests, go build/vet/test, and scratch installs that
migrate 9 pipelines and 31 runs, build ntfy v2.30.0 and start both services healthy
on spare ports before stopping them.
2026-10-13 17:20:14 +01:00
alan 9b305df725 Record final live template and catalog deployment verification 2026-10-12 23:17:29 +01:00
alan 143f237268 Strengthen template planning, reuse and independent review 2026-10-12 23:10:37 +01:00
alan 03ed4e02e4 Add real workflow and execution screenshots to README 2026-10-12 22:25:10 +01:00
alan 4d583faae1 Use the normal main clone path for installation 2026-10-12 00:23:37 +01:00
alan 72dc1f1078 Document the cloneable preview and clean up test formatting 2026-10-11 23:07:11 +01:00
alan 05ae530442 Make Pipelines Docker-only with OpenCode and Pi
Install current verified harness releases inside Docker, preserve persistent accounts and settings, and provide separate interactive and pipeline model defaults with terminal onboarding and fuzzy pickers.

Add Docker-aware authoring transport, Pi stage execution, bounded workflow controls, reliable final-result readback and usage settlement, and finite short-lived CLI pipe draining. Keep dated installation evidence and packaged authoring guidance consistent.

Verified with a real Docker installation, both harness stages and interactive authors, the built-in plan/implement/review starter, independent host race/vet checks, installer tests, and packaged skill tests. Generated release receipts stay ignored in clean checkouts.
2026-10-11 23:00:25 +01:00
alan c42a19f7d3 Improve workflow UI, retain stage diffs, and add portable pipeline transfers
Make OpenCode authoring responsive and expand the managed skill references. Add conflict-aware import/export, real-provider reference workflows, and tracked V2 generation for reliable budget accounting.
2026-10-10 20:55:42 +01:00
alan a66d540125 Add stage execution limits and result-first workflow views 2026-10-10 03:03:46 +01:00
alan 7b10e3889e Replace niche templates with simple starters and direct template submission 2026-10-10 02:37:31 +01:00
alan 4cae37f950 Allow portable drafts to publish to the connected server 2026-10-10 02:18:48 +01:00
alan c74f7f3b41 Center pipeline graphs and share port rows to reduce graph height 2026-10-10 02:08:52 +01:00
alan d73a021cd4 Compact graph columns and branch spacing and simplify Home hints 2026-10-10 02:00:39 +01:00
alan 6bc344dd29 Prompt for unnamed drafts directly before publication 2026-10-10 01:46:08 +01:00
alan 68042aad5d Mount host repositories for Docker execution with host user ownership 2026-10-10 01:22:02 +01:00
alan 7565693957 Expand pipeline panes and select the verified Docker client endpoint 2026-10-10 00:56:55 +01:00
alan 2cc703b8a5 Expand focused run panes and prioritize graph height 2026-10-10 00:47:51 +01:00
alan 977af96e23 Restore box-and-arrow run diagrams without viewport list fallback 2026-10-10 00:43:52 +01:00
alan f8ffde92f7 Add live stage progress and centered auto-fitting run graphs 2026-10-10 00:37:35 +01:00
alan dac34b0097 Clarify run inspector focus and add execution stage navigation 2026-10-10 00:29:36 +01:00
alan e9a8eaf8ae Show full audited run graphs and scope searchable run history 2026-10-10 00:26:32 +01:00
alan a1c0eb4a72 Handle native structured-format incompatibility and preserve failure evidence 2026-10-10 00:01:02 +01:00
alan 16389b8270 Prime OpenCode catalog before container login and model validation 2026-10-09 23:34:01 +01:00
alan 08b8647d5b Validate providers and retain complete run debugging context 2026-10-09 23:21:01 +01:00
alan f3d829bea5 Bind both Docker services to 0.0.0.0 by default 2026-10-09 22:07:46 +01:00
alan de3a3c2363 Automatically configure LAN networking without installer prompts 2026-10-09 22:04:17 +01:00
alan a3e4d7817a Default installer to full Docker setup 2026-10-09 21:50:22 +01:00
alan f700d1beb2 Configure full Docker setup and confirm consequential actions 2026-10-09 21:34:47 +01:00
alan f60599230c Retain OpenCode sessions and verify fresh retry isolation 2026-10-09 21:16:28 +01:00
alan dc9b7b1c02 Soften TUI surfaces, typography and loading indicators 2026-10-09 21:03:39 +01:00
alan dd50344170 Offer nvim or OpenCode when creating a pipeline 2026-10-09 20:53:00 +01:00
alan a50e8648f6 Add service status and usage dashboard, refresh externally published edits 2026-10-09 20:20:44 +01:00
alan 67ac79765f Harden model results with lossless isolated repair and scoped permissions 2026-10-09 20:02:41 +01:00
alan 0f80ff4e95 Add shared live run graphs, debugging, editor choice and draft cleanup 2026-10-09 19:17:43 +01:00
alan 6bd0e28ce6 Record run input directory and retry failed runs with original inputs 2026-10-09 14:23:21 +01:00
alan 42600f6961 Open a run's pinned pipeline from the runs page 2026-10-09 14:13:09 +01:00
alan ed93d055bf Run tasks directly with language-independent project tools 2026-10-09 01:48:46 +01:00
alan 998b0941d3 Install shared authoring skill and add retained drafts browser 2026-10-09 01:33:59 +01:00