Add cross-system read-only search

One query now finds anything in the system over the wire and from the CLI.

- protocol: add Search/SearchResults message types and a SearchRequest
  (query, optional kind filter, optional limit) / SearchResponse (typed,
  ranked matches plus truncation and scan-bound counters) pair, with typed
  payloads for pipeline, version, run, step/visit, field text and artifact.
- store: add Store.Search, a read-only query over pipelines,
  pipeline_versions, pipeline_runs, run_steps (+step_control) and
  step_artifacts. Ranks exact name (3) > name substring (2) > body text (1),
  breaks ties by recency then a stable kind/target order, and reports the
  matched field. Bounds work with a 50-default/200-max result cap, a 2000-row
  per-kind scan limit and a 2000-step / 1 MiB field-text scan budget.
- terms: a bare status word or lone date expression (today, yesterday, this
  week, YYYY-MM-DD) selects runs; anything else stays free text.
- server: dispatch protocol.Search to stores[0].Search; read-only and
  unconfirmed.
- cli: add `pipelines search QUERY [--type KIND] [--limit N] [--json]`, help
  text and a BROWSE line.
- store: add OpenReadOnly so the live-database check never runs DDL.
- docs: document the grammar, kinds, ranking and bounds in docs/cli.md,
  scope the TUI omnisearch to pipelines in docs/ui-ux-review.md, and add a
  search bullet to the packaged operations reference.
- tests: protocol round trip, store/ranking/term-parsing/limit/scan-bound
  unit tests, a server end-to-end test, and an env-guarded live check.

No new dependencies; go.mod/go.sum unchanged.
This commit is contained in:
2026-10-16 15:43:25 +01:00
parent dce624b428
commit c700b28e6a
16 changed files with 1914 additions and 8 deletions
+42 -4
View File
@@ -61,6 +61,42 @@ ntfy hosts are not misrepresented as local. Publishing URLs and credentials are
never returned. Missing public phone configuration is labeled explicitly.
`pipelines service status` remains the separate OS autostart status command.
## Search
`pipelines search QUERY [--type KIND] [--limit N] [--json]` runs one read-only
query across the whole system: pipeline names and their versions, run identity,
status and dates, stage and visit identifiers, artifact names and types, and the
text of stage inputs, outputs, evidence and transcripts. It never reads
artifact contents, never makes a per-result follow-up request, and asks for no
confirmation.
- Free text matches case-insensitively. A bare status word (`queued`, `running`,
`succeeded`, `failed`, `canceled`, `interrupted`) selects those runs, and a lone
date expression (`today`, `yesterday`, `this week` — Monday 00:00 local — or a
literal `YYYY-MM-DD`) selects the runs created in that window; any other text,
including a status word inside a sentence, stays free text.
- `--type` is the only narrowing mechanism and takes one kind: `pipeline`,
`version`, `run`, `step`, `field` or `artifact`. An unknown kind is an error,
never a silently ignored filter.
- Ranking: an exact name match (score 3) beats a name substring (2) beats a
body-text match (1); equal tiers break by recency, then a stable kind/target
order. Every result carries `score`, `matched_field` and `recency` so a client
can re-rank.
- Bounds: the result cap defaults to 50 and clamps to 200 (`0` = default);
every kind scans at most 2000 rows, and field-text scanning stops at 2000 steps
or 1 MiB decoded. `truncated` plus `scanned_steps`, `scanned_bytes`, `cap` and
`scan_bound_bytes` report when a bound applied.
- Targets are stable and navigable: `pipeline:<id>`, `pipeline:<id>:v<version>`,
`run:<id>`, `run:<id>:<sequence>`, `run:<id>:<sequence>:<field>`, or an
artifact URL served by the artifact endpoint.
```sh
pipelines search "release notes" --json
pipelines search failed --json
pipelines search 2026-10-16 --type run --json
pipelines search "acceptance check" --type field --limit 10
```
## Draft-first harness workflow
```sh
@@ -194,6 +230,7 @@ Submitting activates included schedules, so review them before publishing.
| TUI action | Agent-capable CLI |
| --- | --- |
| Browse/search pipelines | `list --json`, `show NAME --json` |
| Search everything | `search QUERY --json` (see [Search](#search)) |
| Original source and history | `export NAME --version N`, `versions NAME --json` |
| Browse templates/source | `templates --json`, `template KEY --source` |
| Create or edit a definition | `new`, `edit`, `drafts` |
@@ -217,10 +254,11 @@ printf '%s' '{"type":"get_pipeline","pipeline_id":1,"version":1}' | pipelines rp
Existing protocol commands: `ping`, `list_pipelines`, `get_pipeline`,
`list_templates`, `validate`, `submit`, `run_pipeline`, `list_runs`, `get_run`,
`cancel_run`, `retry_run`, `get_status`. `list_runs` also returns all-history `usage`
aggregates. See `internal/protocol/message.go` for JSON envelopes. Mutating RPCs
have real effects; they are not previews. TUI-only animation, scrolling, and
transient client logs are presentation, not separate server capabilities.
`cancel_run`, `retry_run`, `get_status`, `search`. `list_runs` also returns
all-history `usage` aggregates. See `internal/protocol/message.go` for JSON
envelopes. Mutating RPCs have real effects; they are not previews. TUI-only
animation, scrolling, and transient client logs are presentation, not separate
server capabilities.
## Scripted runs
+1 -1
View File
@@ -38,7 +38,7 @@ and external editor/OpenCode behavior were not assessed by that review.
| Status | Narrow cards hide addresses; scrolling does not reveal them. | Wrap card contents and scroll the actual service-card canvas. |
| Task form | Tool lists consume prompt space; wrong footer. | Larger task area, compact scope summary, correct submit/cancel controls. |
| CLI pipeline picker | Background Home controls remain visible. | Picker-specific filter/select/cancel controls. |
| Search | “Omnisearch” promises broader scope than pipeline search. | “Search pipelines” and specific open/cancel instructions. |
| Search | “Omnisearch” promises broader scope than pipeline search. | Keep the TUI omnisearch pipeline-scoped and labelled “Search pipelines” with open/cancel instructions; the broad cross-kind search is the server-backed `pipelines search QUERY [--type KIND]` command, whose grammar, kinds, ranking and bounds are documented in [CLI and agent authoring](cli.md#search). |
| Commands | No descriptions or indication of hidden entries. | Selected-command explanation and visible range/count. |
| Versions | Selected/latest markers truncate on narrow screens. | Correct view-version footer; further metadata prioritization remains a follow-up. |
| Naming/publication | Naming inherits publication shortcuts. | Input-specific name-saving/retention footer. |
@@ -71,6 +71,21 @@ delete databases, sessions, volumes or credentials to repair a workflow.
Use the authoritative `CLI_PREFIX` from the prompt:
- `CLI_PREFIX runs --json`: run history (optional `--pipeline NAME` scope).
- `CLI_PREFIX search QUERY [--type KIND] [--limit N] --json`: one read-only query
across pipelines, versions, runs, steps, field text and artifacts. QUERY is free
text, a bare status word (`queued`/`running`/`succeeded`/`failed`/`canceled`/
`interrupted`) selecting those runs, or a lone date expression (`today`,
`yesterday`, `this week` — Monday 00:00 local — or `YYYY-MM-DD`) selecting the
runs created in that window; anything else stays free text. `--type` is the only
narrowing mechanism and takes one of `pipeline`, `version`, `run`, `step`,
`field`, `artifact`; an unknown kind is an error. Ranking is exact name (3) >
name substring (2) > body text (1), equal tiers by recency, and every hit
reports `score`, `matched_field` and `recency`. The cap defaults to 50 and clamps
to 200; every kind scans at most 2000 rows and field scanning stops at 2000
steps / 1 MiB and reports
`truncated`/`scanned_steps`/`scanned_bytes`/`cap`/`scan_bound_bytes`. Targets are
stable: `pipeline:<id>`, `pipeline:<id>:v<version>`, `run:<id>`,
`run:<id>:<sequence>`, `run:<id>:<sequence>:<field>`, or an artifact URL.
- `CLI_PREFIX inspect RUN_ID --json`: selected run, visits and retained evidence.
Exact stage outputs are authoritative; a transcript over the 32 KiB retained
bound is a valid, credential-redacted preview envelope (with preserved stage
+50 -1
View File
@@ -37,9 +37,10 @@ type options struct {
url, name, prompt, template, workspace, paths, key, opencode, model string
output string
conflict string
kind string
all bool
json, source, noOpen, yes, allow, createOnly, force, wait bool
version, expected int
version, expected, limit int
timeout, waitTimeout time.Duration
}
@@ -99,6 +100,9 @@ func (a App) flags(command string) (*flag.FlagSet, *options) {
f.DurationVar(&o.waitTimeout, "wait-timeout", 0, "Client limit for --wait after the run is admitted (0 waits until terminal); separate from per-request --timeout")
case "runs":
f.StringVar(&o.name, "pipeline", "", "Filter by exact pipeline name or ID")
case "search":
f.StringVar(&o.kind, "type", "", "Narrow to one kind: "+strings.Join(protocol.SearchKinds, ", "))
f.IntVar(&o.limit, "limit", 0, "Maximum results (default 50, max 200)")
case "retry":
f.BoolVar(&o.yes, "yes", false, "Confirm retry without an interactive prompt")
f.StringVar(&o.key, "request-key", "", "Idempotency key (default: generated unique key)")
@@ -697,6 +701,31 @@ func (a App) Run(ctx context.Context, args []string) error {
return err
case "import":
return a.importBundle(ctx, o, f.Args(), call)
case "search":
query := strings.Join(pos, " ")
if strings.TrimSpace(query) == "" {
return fmt.Errorf("search requires a query; use pipelines help search")
}
r, err := call(protocol.Message{Type: protocol.Search, Search: &protocol.SearchRequest{Query: query, Kind: o.kind, Limit: o.limit}})
if err != nil {
return err
}
if r.SearchResults == nil {
return fmt.Errorf("server did not return search results; update pipelines-server")
}
if o.json {
return a.output(r.SearchResults)
}
if len(r.SearchResults.Results) == 0 {
fmt.Fprintln(a.Out, "No matches")
}
for _, hit := range r.SearchResults.Results {
fmt.Fprintf(a.Out, "%s\t%s\t%d\t%s\t%s\n", hit.Kind, hit.Target, hit.Score, hit.MatchedField, searchLabel(hit))
}
if r.SearchResults.Truncated {
fmt.Fprintf(a.Err, "Results truncated (cap %d, scan bound %d bytes)\n", r.SearchResults.Cap, r.SearchResults.ScanBoundBytes)
}
return nil
case "show", "versions":
value, err := one()
if err != nil {
@@ -819,6 +848,26 @@ func (a App) output(value any) error {
return encoder.Encode(value)
}
// searchLabel renders one search hit's compact human line. Each kind exposes a
// different typed payload; the target stays the navigable identity.
func searchLabel(hit protocol.SearchResult) string {
switch {
case hit.Pipeline != nil:
return fmt.Sprintf("%s · v%d · %d stages", hit.Pipeline.Name, hit.Pipeline.Version, hit.Pipeline.Stages)
case hit.Version != nil:
return fmt.Sprintf("pipeline %d · v%d · %s", hit.Version.PipelineID, hit.Version.Version, hit.Version.ContentHash)
case hit.Run != nil:
return fmt.Sprintf("run %d · %s · %s", hit.Run.ID, hit.Run.PipelineName, hit.Run.Status)
case hit.Step != nil:
return fmt.Sprintf("run %d · step %d · %s · %s", hit.Step.RunID, hit.Step.Sequence, hit.Step.Stage, hit.Step.Outcome)
case hit.Field != nil:
return fmt.Sprintf("run %d · step %d · %s: %s", hit.Field.RunID, hit.Field.Sequence, hit.Field.Field, hit.Field.Snippet)
case hit.Artifact != nil:
return fmt.Sprintf("run %d · step %d · %s · %s", hit.Artifact.RunID, hit.Artifact.Sequence, hit.Artifact.Name, hit.Artifact.ContentType)
}
return ""
}
type callFunc func(protocol.Message) (protocol.Message, error)
type resolveFunc func(string) (protocol.PipelineSummary, error)
+2
View File
@@ -10,6 +10,7 @@ var usage = map[string]string{
"validate": "validate <draft.md> [--json]",
"submit": "submit <draft.md> [--name NAME] [--create-only] [--yes] [--json]",
"list": "list [--json]",
"search": "search QUERY [--type KIND] [--limit N] [--json]",
"show": "show <name|id> [--version N] [--json]",
"versions": "versions <name|id> [--json]",
"export": "export [name|id ...] [--output FILE.zip] [--all] [--version N]",
@@ -44,6 +45,7 @@ var notes = map[string]string{
"retry": "Start a new attempt of a failed run with the same initial input, directory, pipeline version, and edit scope. Original history is retained. Already-completed side effects are not undone. --wait/--wait-timeout behave as for run: a reached wait limit leaves the new run active; inspect read-only with `inspect <run-id> --json`, or reattach with --wait and the original --request-key instead of starting a duplicate retry.",
"debug": "Open a fresh configured harness conversation to investigate a run using inspect/show/runs CLI queries. --harness opencode|pi|nvim chooses another configured harness or read-only nvim packet. Does not resume implementation or retry. --json returns the prompt without launching anything.",
"rpc": "Accepts a JSON protocol envelope; '-' reads stdin. Every TUI server operation is available here too. Mutating commands are not dry runs.",
"search": "One read-only query over pipelines, versions, runs, steps, field text and artifacts. QUERY is free text, a bare status word (queued, running, succeeded, failed, canceled, interrupted) selecting those runs, or a lone date expression (today, yesterday, this week [Monday 00:00 local], or YYYY-MM-DD) selecting the runs created in that window. --type is the only narrowing mechanism and takes one kind: pipeline, version, run, step, field or artifact; an unknown kind is an error. Ranking: an exact name match (3) beats a name substring (2) beats a body-text match (1); equal tiers break by recency, then a stable kind/target tie-break, and every hit reports score, matched_field and recency so a client can re-rank. Bounds: the result cap defaults to 50 and clamps to 200 (0 = default); every kind scans at most 2000 rows and field-text scanning is additionally capped at 2000 steps / 1 MiB; Truncated plus scanned_steps, scanned_bytes and scan_bound_bytes report when a bound applied. Targets are stable: pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>, run:<id>:<sequence>:<field>, or an artifact URL.",
}
var serviceHelp = map[string]string{
+1
View File
@@ -13,6 +13,7 @@ AUTHOR
BROWSE
list List registered pipelines
search QUERY Search pipelines, versions, runs, steps, field text, artifacts
show <name|id> Inspect graph, settings, source and history
versions <name|id> List immutable versions
export [name|id ...] Select/export definitions as ZIP; one name prints Markdown
+10 -2
View File
@@ -107,11 +107,19 @@ func (s *Store) CreateRun(ctx context.Context, d *protocol.PipelineDetail, key,
return id, true, nil
}
// runFrom is the shared pipeline_runs/pipelines join every run-row read builds
// on, so the run projections cannot drift.
const runFrom = `FROM pipeline_runs r JOIN pipelines p ON p.id=r.pipeline_id`
// runStamps formats a run row's created/finished timestamps; reused by runSelect
// and searchRuns instead of copied per query.
const runStamps = `strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',r.finished_at),'')`
const runSelect = `SELECT r.id,r.pipeline_id,p.name,r.version,v.content_hash,r.status,r.outcome,r.current_stage,
strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',r.finished_at),''),r.diagnostic,
` + runStamps + `,r.diagnostic,
CASE WHEN r.status='succeeded' THEN r.output_json ELSE COALESCE((SELECT output_json FROM run_steps WHERE run_id=r.id AND status='completed' ORDER BY sequence DESC LIMIT 1),'{}') END,
(SELECT COUNT(*) FROM run_steps WHERE run_id=r.id AND status='completed')
FROM pipeline_runs r JOIN pipelines p ON p.id=r.pipeline_id JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
` + runFrom + ` JOIN pipeline_versions v ON v.pipeline_id=r.pipeline_id AND v.version=r.version`
func scanRun(row interface{ Scan(...any) error }, r *protocol.RunSummary) error {
var output string
+686
View File
@@ -0,0 +1,686 @@
package pipeline
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"sort"
"strconv"
"strings"
"time"
"unicode"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// Search answers one ranked, bounded, read-only query over the existing tables.
// It scans no artifact contents and makes no follow-up calls: each kind runs at
// most one query, field text is decoded with decodeValues over a fixed scan
// budget, and ranking/tie-breaking happens in memory over the candidate set.
func (s *Store) Search(ctx context.Context, req protocol.SearchRequest, now time.Time) (*protocol.SearchResponse, error) {
kind := strings.TrimSpace(req.Kind)
if kind != "" && searchKindIndex(kind) < 0 {
return nil, fmt.Errorf("unknown search kind %q; use one of %s", req.Kind, strings.Join(protocol.SearchKinds, ", "))
}
response := &protocol.SearchResponse{
Query: req.Query,
Kind: kind,
Results: []protocol.SearchResult{},
Cap: searchLimit(req.Limit),
ScanBoundBytes: protocol.SearchMaxScanBytes,
}
// An empty query is a well-defined no-op, never a match-everything scan.
if strings.TrimSpace(req.Query) == "" {
return response, nil
}
terms := parseSearchTerms(req.Query, now)
// A bare status word or date selects runs; it is never treated as body text.
selectorOnly := terms.Text == ""
tx, err := s.db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return nil, err
}
defer tx.Rollback()
var results []protocol.SearchResult
var scan searchScan
for i, candidate := range protocol.SearchKinds {
if kind != "" && searchKindIndex(kind) != i {
continue
}
if selectorOnly && candidate != protocol.SearchKindRun {
continue
}
var hits []protocol.SearchResult
var err error
switch candidate {
case protocol.SearchKindPipeline:
hits, err = searchPipelines(ctx, tx, terms, &scan)
case protocol.SearchKindVersion:
hits, err = searchVersions(ctx, tx, terms, &scan)
case protocol.SearchKindRun:
hits, err = searchRuns(ctx, tx, terms, &scan)
case protocol.SearchKindStep:
hits, err = searchSteps(ctx, tx, terms, &scan)
case protocol.SearchKindField:
hits, err = searchFields(ctx, tx, terms, &scan)
case protocol.SearchKindArtifact:
hits, err = searchArtifacts(ctx, tx, terms, &scan)
}
if err != nil {
return nil, err
}
results = append(results, hits...)
}
if err := tx.Commit(); err != nil {
return nil, err
}
response.ScannedSteps = scan.steps
response.ScannedBytes = scan.bytes
if scan.truncated {
response.Truncated = true
}
sortSearchResults(results)
if len(results) > response.Cap {
results = results[:response.Cap]
response.Truncated = true
}
if results == nil {
results = []protocol.SearchResult{}
}
response.Results = results
return response, nil
}
func searchKindIndex(kind string) int {
for i, k := range protocol.SearchKinds {
if k == kind {
return i
}
}
return -1
}
// searchLimit clamps a requested cap. Zero, negative or over-max requests fall
// back to the default or the maximum, so the cap is always enforced server-side.
func searchLimit(limit int) int {
if limit <= 0 {
return protocol.SearchDefaultLimit
}
if limit > protocol.SearchMaxLimit {
return protocol.SearchMaxLimit
}
return limit
}
// searchTerms is the parsed query. Text carries free text; Status and the
// From/To window carry a bare status word or a date expression that selects runs.
type searchTerms struct {
Text string
Status string
HasDate bool
From time.Time
To time.Time
}
// parseSearchTerms recognises a lone status word and a lone date expression and
// leaves everything else as free text, so ordinary words are never misread as a
// selector. Relative dates resolve against the injected now in its own location;
// the comparison window is converted to UTC to match the stored timestamps.
func parseSearchTerms(query string, now time.Time) searchTerms {
trimmed := strings.TrimSpace(query)
lower := strings.ToLower(trimmed)
for _, status := range protocol.SearchStatuses {
if lower == status {
return searchTerms{Status: status}
}
}
location := now.Location()
day := time.Date(now.Year(), now.Month(), now.Day(), 0, 0, 0, 0, location)
switch lower {
case "today":
return dateTerms(day, day.AddDate(0, 0, 1))
case "yesterday":
return dateTerms(day.AddDate(0, 0, -1), day)
case "this week":
// Monday 00:00 local is the start of the week.
start := day.AddDate(0, 0, -((int(day.Weekday()) + 6) % 7))
return dateTerms(start, start.AddDate(0, 0, 7))
}
if parsed, err := time.ParseInLocation("2006-01-02", trimmed, location); err == nil {
return dateTerms(parsed, parsed.AddDate(0, 0, 1))
}
return searchTerms{Text: trimmed}
}
func dateTerms(from, to time.Time) searchTerms {
return searchTerms{HasDate: true, From: from.UTC(), To: to.UTC()}
}
const searchSQLTime = "2006-01-02 15:04:05"
// searchStepJoin is the shared run_steps/step_control source and projections the
// step and field scans build on, so those queries cannot drift apart.
const (
searchStepJoin = `FROM run_steps s LEFT JOIN step_control c ON c.run_id=s.run_id AND c.sequence=s.sequence`
searchStepEvidence = `COALESCE(c.evidence_json,'')`
searchStepRecency = `COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',s.started_at),'')`
)
// searchRowBudget bounds how many raw rows one non-body kind scans before it
// stops and reports Truncated, so a LIKE scan over a very large table cannot
// scale without limit. Field text uses the separate SearchMaxScanSteps budget.
const searchRowBudget = 2000
// searchLike builds a case-insensitive substring pattern, escaping LIKE
// metacharacters so a query can never broaden the pattern.
func searchLike(text string) string {
escaped := strings.NewReplacer(`\`, `\\`, `%`, `\%`, `_`, `\_`).Replace(strings.ToLower(text))
return "%" + escaped + "%"
}
// searchTier ranks a "name" match: exact beats a substring. The third tier (a
// body-text match) is assigned by the caller.
func searchTier(value, text string) (int, bool) {
if text == "" {
return 0, false
}
if strings.EqualFold(value, text) {
return 3, true
}
if strings.Contains(strings.ToLower(value), strings.ToLower(text)) {
return 2, true
}
return 0, false
}
func searchContains(value, text string) bool {
return text != "" && strings.Contains(strings.ToLower(value), strings.ToLower(text))
}
func sortSearchResults(results []protocol.SearchResult) {
sort.SliceStable(results, func(i, j int) bool {
a, b := results[i], results[j]
if a.Score != b.Score {
return a.Score > b.Score
}
if a.Recency != b.Recency {
return a.Recency > b.Recency
}
if a.Kind != b.Kind {
return a.Kind < b.Kind
}
return a.Target < b.Target
})
}
func searchPipelines(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
if terms.Text == "" {
return nil, nil
}
rows, err := tx.QueryContext(ctx, `SELECT p.id,p.name,v.version,json_array_length(v.graph_json,'$.stages'),COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',v.created_at),'')
FROM pipelines p JOIN pipeline_versions v ON v.pipeline_id=p.id
WHERE v.version=(SELECT MAX(version) FROM pipeline_versions WHERE pipeline_id=p.id)
AND lower(p.name) LIKE ? ESCAPE '\'
ORDER BY p.name,p.id LIMIT ?`, searchLike(terms.Text), searchRowBudget+1)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
scanned := 0
for rows.Next() {
if scanned == searchRowBudget {
scan.truncated = true
break
}
scanned++
var id int64
var name, recency string
var version, stages int
if err := rows.Scan(&id, &name, &version, &stages, &recency); err != nil {
return nil, err
}
score, ok := searchTier(name, terms.Text)
if !ok {
continue
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindPipeline, Target: searchPipelineTarget(id),
Score: score, MatchedField: "name", Recency: recency,
Pipeline: &protocol.SearchPipeline{ID: id, Name: name, Version: version, Stages: stages},
})
}
return results, rows.Err()
}
func searchVersions(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
if terms.Text == "" {
return nil, nil
}
query := `SELECT v.pipeline_id,v.version,v.content_hash,COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',v.created_at),''),v.source
FROM pipeline_versions v WHERE lower(v.source) LIKE ? ESCAPE '\'`
args := []any{searchLike(terms.Text)}
var wanted int64
numeric := false
if n, err := parsePositiveInt(terms.Text); err == nil {
query += " OR v.version=?"
args = append(args, n)
wanted, numeric = n, true
}
query += " ORDER BY v.pipeline_id,v.version DESC LIMIT ?"
args = append(args, searchRowBudget+1)
rows, err := tx.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
scanned := 0
for rows.Next() {
if scanned == searchRowBudget {
scan.truncated = true
break
}
scanned++
var pipelineID int64
var version int
var hash, recency, source string
if err := rows.Scan(&pipelineID, &version, &hash, &recency, &source); err != nil {
return nil, err
}
score, matched := 1, "source"
if numeric && int64(version) == wanted {
score, matched = 3, "version"
} else if !searchContains(source, terms.Text) {
continue
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindVersion, Target: searchVersionTarget(pipelineID, version),
Score: score, MatchedField: matched, Recency: recency,
Version: &protocol.SearchVersion{PipelineID: pipelineID, Version: version, ContentHash: hash, CreatedAt: recency},
})
}
return results, rows.Err()
}
func searchRuns(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
// Reuse runSelect's join and timestamp projection rather than restating the
// run-row SELECT; only the fields search needs are added.
base := `SELECT r.id,p.name,r.status,r.workspace,` + runStamps + `,r.diagnostic ` + runFrom
var query string
var args []any
switch {
case terms.Status != "":
query = base + " WHERE r.status=?"
args = append(args, terms.Status)
case terms.HasDate:
query = base + " WHERE r.created_at>=? AND r.created_at<?"
args = append(args, terms.From.Format(searchSQLTime), terms.To.Format(searchSQLTime))
default:
query = base + ` WHERE lower(p.name) LIKE ? ESCAPE '\' OR lower(r.workspace) LIKE ? ESCAPE '\' OR lower(r.status) LIKE ? ESCAPE '\' OR lower(r.diagnostic) LIKE ? ESCAPE '\' OR CAST(r.id AS TEXT)=?`
pattern := searchLike(terms.Text)
args = append(args, pattern, pattern, pattern, pattern, terms.Text)
}
query += " ORDER BY r.id DESC LIMIT ?"
args = append(args, searchRowBudget+1)
rows, err := tx.QueryContext(ctx, query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
scanned := 0
for rows.Next() {
if scanned == searchRowBudget {
scan.truncated = true
break
}
scanned++
var id int64
var name, status, workspace, created, finished, diagnostic string
if err := rows.Scan(&id, &name, &status, &workspace, &created, &finished, &diagnostic); err != nil {
return nil, err
}
score, matched := 1, "status"
if terms.Status != "" {
matched = "status"
} else if terms.HasDate {
matched = "created_at"
} else if s, ok := searchTier(name, terms.Text); ok {
score, matched = s, "pipeline_name"
} else if numericEqual(terms.Text, id) {
score, matched = 3, "id"
} else if searchContains(workspace, terms.Text) {
matched = "workspace"
} else if searchContains(status, terms.Text) {
matched = "status"
} else if searchContains(diagnostic, terms.Text) {
matched = "diagnostic"
} else {
continue
}
run := &protocol.SearchRun{ID: id, PipelineName: name, Status: status, Workspace: workspace, CreatedAt: created, FinishedAt: finished}
if created != "" && finished != "" {
if secs, ok := durationSeconds(created, finished); ok {
run.DurationSeconds = secs
}
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindRun, Target: searchRunTarget(id),
Score: score, MatchedField: matched, Recency: created, Run: run,
})
}
return results, rows.Err()
}
func searchSteps(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
if terms.Text == "" {
return nil, nil
}
pattern := searchLike(terms.Text)
rows, err := tx.QueryContext(ctx, `SELECT s.run_id,s.sequence,s.stage_key,s.outcome,s.target,s.diagnostic,`+searchStepRecency+`,`+searchStepEvidence+`
`+searchStepJoin+`
WHERE lower(s.stage_key) LIKE ? ESCAPE '\' OR lower(s.outcome) LIKE ? ESCAPE '\' OR lower(s.target) LIKE ? ESCAPE '\' OR lower(s.diagnostic) LIKE ? ESCAPE '\'
ORDER BY s.run_id DESC,s.sequence DESC LIMIT ?`, pattern, pattern, pattern, pattern, searchRowBudget+1)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
scanned := 0
for rows.Next() {
if scanned == searchRowBudget {
scan.truncated = true
break
}
scanned++
var runID int64
var sequence int
var stage, outcome, target, diagnostic, recency, control string
if err := rows.Scan(&runID, &sequence, &stage, &outcome, &target, &diagnostic, &recency, &control); err != nil {
return nil, err
}
score, matched := 1, "diagnostic"
if s, ok := searchTier(stage, terms.Text); ok {
score, matched = s, "stage"
} else if searchContains(outcome, terms.Text) {
matched = "outcome"
} else if searchContains(target, terms.Text) {
matched = "target"
} else if searchContains(diagnostic, terms.Text) {
matched = "diagnostic"
} else {
continue
}
visit := 0
if control != "" && control != "null" {
var evidence protocol.ControlEvidence
if err := json.Unmarshal([]byte(control), &evidence); err == nil {
visit = evidence.Visit
}
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindStep, Target: searchStepTarget(runID, sequence),
Score: score, MatchedField: matched, Recency: recency,
Step: &protocol.SearchStep{RunID: runID, Sequence: sequence, Stage: stage, Visit: visit, Outcome: outcome, Target: target, Diagnostic: diagnostic},
})
}
return results, rows.Err()
}
// searchScan accumulates the shared bounded-scan budget so the response can
// report how much body text was inspected and whether any bound was reached.
type searchScan struct {
steps int
bytes int
truncated bool
}
func searchFields(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
if terms.Text == "" {
return nil, nil
}
// One extra step is requested so a scan that hits the step bound can see that
// more rows remain and report Truncated rather than ending silently.
rows, err := tx.QueryContext(ctx, `SELECT s.run_id,s.sequence,s.input_json,s.output_json,s.transcript,`+searchStepEvidence+`,`+searchStepRecency+`
`+searchStepJoin+`
ORDER BY s.run_id DESC,s.sequence DESC LIMIT ?`, protocol.SearchMaxScanSteps+1)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
for rows.Next() {
if scan.steps >= protocol.SearchMaxScanSteps {
scan.truncated = true
break
}
var runID int64
var sequence int
var input, output, transcript, evidence, recency string
if err := rows.Scan(&runID, &sequence, &input, &output, &transcript, &evidence, &recency); err != nil {
return nil, err
}
size := len(input) + len(output) + len(transcript) + len(evidence)
if scan.bytes+size > protocol.SearchMaxScanBytes {
scan.truncated = true
break
}
scan.bytes += size
scan.steps++
for _, field := range []struct {
name string
raw string
}{
{name: "input", raw: input},
{name: "output", raw: output},
{name: "evidence", raw: evidence},
} {
if field.raw == "" {
continue
}
var values map[string]any
if err := decodeValues(field.raw, &values); err != nil {
continue
}
text, ok := findMatch(values, terms.Text)
if !ok {
continue
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindField, Target: searchFieldTarget(runID, sequence, field.name),
Score: 1, MatchedField: field.name, Recency: recency,
Field: &protocol.SearchField{RunID: runID, Sequence: sequence, Field: field.name, Snippet: searchSnippet(text, terms.Text)},
})
}
if searchContains(transcript, terms.Text) {
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindField, Target: searchFieldTarget(runID, sequence, "transcript"),
Score: 1, MatchedField: "transcript", Recency: recency,
Field: &protocol.SearchField{RunID: runID, Sequence: sequence, Field: "transcript", Snippet: searchSnippet(transcript, terms.Text)},
})
}
}
return results, rows.Err()
}
func searchArtifacts(ctx context.Context, tx *sql.Tx, terms searchTerms, scan *searchScan) ([]protocol.SearchResult, error) {
if terms.Text == "" {
return nil, nil
}
pattern := searchLike(terms.Text)
rows, err := tx.QueryContext(ctx, `SELECT a.run_id,a.sequence,a.artifacts_json,COALESCE(strftime('%Y-%m-%dT%H:%M:%SZ',r.created_at),'')
FROM step_artifacts a JOIN pipeline_runs r ON r.id=a.run_id
WHERE lower(a.artifacts_json) LIKE ? ESCAPE '\'
ORDER BY a.run_id DESC,a.sequence DESC LIMIT ?`, pattern, searchRowBudget+1)
if err != nil {
return nil, err
}
defer rows.Close()
var results []protocol.SearchResult
scanned := 0
for rows.Next() {
if scanned == searchRowBudget {
scan.truncated = true
break
}
scanned++
var runID int64
var sequence int
var artifacts, recency string
if err := rows.Scan(&runID, &sequence, &artifacts, &recency); err != nil {
return nil, err
}
var evidence stageArtifactsEvidence
if err := json.Unmarshal([]byte(artifacts), &evidence); err != nil {
continue
}
for _, file := range evidence.Files {
score, matched := 1, "content_type"
if s, ok := searchTier(file.Name, terms.Text); ok {
score, matched = s, "name"
} else if !searchContains(file.ContentType, terms.Text) {
continue
}
results = append(results, protocol.SearchResult{
Kind: protocol.SearchKindArtifact, Target: file.URL,
Score: score, MatchedField: matched, Recency: recency,
Artifact: &protocol.SearchArtifact{RunID: runID, Sequence: sequence, Name: file.Name, ContentType: file.ContentType, SizeBytes: file.SizeBytes, URL: file.URL},
})
}
}
return results, rows.Err()
}
// findMatch walks decoded JSON values in a stable key order and returns the
// first string leaf containing the query, so a hit and its snippet are
// deterministic rather than map-iteration dependent.
func findMatch(value any, text string) (string, bool) {
switch typed := value.(type) {
case string:
if searchContains(typed, text) {
return typed, true
}
case map[string]any:
keys := make([]string, 0, len(typed))
for key := range typed {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
if match, ok := findMatch(typed[key], text); ok {
return match, true
}
}
case []any:
for _, item := range typed {
if match, ok := findMatch(item, text); ok {
return match, true
}
}
}
return "", false
}
// searchSnippet reduces text to a single clean line and centers a short window
// on the first case-insensitive match, folding runes so UTF-8 boundaries are
// never split and the result is always well-formed.
func searchSnippet(text, query string) string {
normalized := protocol.NormalizeText(text)
runes := []rune(normalized)
index := matchRunes(runes, []rune(query))
if index < 0 {
return truncateRunes(normalized, 80)
}
const margin = 40
start := index - margin
if start < 0 {
start = 0
}
end := index + len([]rune(query)) + margin
if end > len(runes) {
end = len(runes)
}
snippet := string(runes[start:end])
if start > 0 {
snippet = "…" + snippet
}
if end < len(runes) {
snippet = snippet + "…"
}
return snippet
}
// matchRunes returns the rune index of query in text, case-insensitively.
func matchRunes(text, query []rune) int {
if len(query) == 0 || len(query) > len(text) {
return -1
}
for i := 0; i+len(query) <= len(text); i++ {
match := true
for j := range query {
if unicode.ToLower(text[i+j]) != unicode.ToLower(query[j]) {
match = false
break
}
}
if match {
return i
}
}
return -1
}
func truncateRunes(text string, limit int) string {
runes := []rune(text)
if len(runes) <= limit {
return text
}
return string(runes[:limit]) + "…"
}
func searchPipelineTarget(id int64) string { return fmt.Sprintf("pipeline:%d", id) }
func searchVersionTarget(id int64, version int) string {
return fmt.Sprintf("pipeline:%d:v%d", id, version)
}
func searchRunTarget(id int64) string { return fmt.Sprintf("run:%d", id) }
func searchStepTarget(id int64, sequence int) string { return fmt.Sprintf("run:%d:%d", id, sequence) }
func searchFieldTarget(id int64, sequence int, field string) string {
return fmt.Sprintf("run:%d:%d:%s", id, sequence, field)
}
func parsePositiveInt(text string) (int64, error) {
n, err := strconv.ParseInt(strings.TrimSpace(text), 10, 64)
if err != nil || n <= 0 {
return 0, fmt.Errorf("not a positive integer")
}
return n, nil
}
func numericEqual(text string, id int64) bool {
n, err := parsePositiveInt(text)
return err == nil && n == id
}
func durationSeconds(created, finished string) (float64, bool) {
start, err := time.Parse(time.RFC3339, created)
if err != nil {
return 0, false
}
end, err := time.Parse(time.RFC3339, finished)
if err != nil {
return 0, false
}
if end.Before(start) {
return 0, false
}
return end.Sub(start).Seconds(), true
}
+119
View File
@@ -0,0 +1,119 @@
package pipeline
import (
"context"
"database/sql"
"os"
"strings"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// TestLiveSearchRealDatabase is an env-guarded read-only check against a real
// database. It opens the file read-only (no DDL, no journal change, no writes),
// derives queries from the data itself, and reports one result of each kind it
// can observe. Run it with:
//
// PIPELINES_LIVE_SEARCH_DB=/path/to/pipelines.db go test ./internal/pipeline -run LiveSearch -v
func TestLiveSearchRealDatabase(t *testing.T) {
path := os.Getenv("PIPELINES_LIVE_SEARCH_DB")
if path == "" {
t.Skip("set PIPELINES_LIVE_SEARCH_DB to a real pipelines.db")
}
store, err := OpenReadOnly(path)
if err != nil {
t.Fatalf("open read-only: %v", err)
}
defer store.Close()
requireSearchSchema(t, store)
ctx := context.Background()
pipelines, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
if len(pipelines) == 0 {
t.Skip("database has no pipelines")
}
now := time.Now()
seen := map[string]protocol.SearchResult{}
record := func(query string) {
response, err := store.Search(ctx, protocol.SearchRequest{Query: query}, now)
if err != nil {
t.Fatalf("search %q: %v", query, err)
}
for _, hit := range response.Results {
if _, ok := seen[hit.Kind]; !ok {
seen[hit.Kind] = hit
}
}
}
queries := []string{}
for _, p := range pipelines {
queries = append(queries, p.Name)
}
// Derive stage/field/artifact queries from a bounded number of recent runs.
runs, err := store.ListRuns(ctx)
if err != nil {
t.Fatal(err)
}
for i := 0; i < len(runs) && i < 5; i++ {
detail, err := store.GetRun(ctx, runs[i].ID)
if err != nil {
continue
}
for j := 0; j < len(detail.Steps) && j < 3; j++ {
step := detail.Steps[j]
if step.Stage != "" {
queries = append(queries, step.Stage)
}
for _, artifact := range step.Artifacts {
queries = append(queries, artifact.Name)
}
for _, key := range []string{"summary", "result", "report", "message", "evidence", "reason"} {
if text, ok := step.Output[key].(string); ok {
for _, word := range strings.Fields(text) {
if len([]rune(word)) >= 6 {
queries = append(queries, word)
break
}
}
}
}
}
}
for _, query := range queries {
record(query)
}
if len(seen) == 0 {
t.Fatal("live database returned no matches for any derived query")
}
for _, kind := range protocol.SearchKinds {
if hit, ok := seen[kind]; ok {
t.Logf("kind=%s target=%s score=%d matched=%s", kind, hit.Target, hit.Score, hit.MatchedField)
} else {
t.Logf("kind=%s not observed in this database", kind)
}
}
if _, ok := seen[protocol.SearchKindRun]; !ok {
t.Fatal("live database returned no run matches")
}
}
// requireSearchSchema skips (rather than fails) when the supplied database
// predates the tables search reads, so pointing the check at a stale local file
// reports the mismatch instead of a confusing "no such table" error.
func requireSearchSchema(t *testing.T, store *Store) {
t.Helper()
for _, table := range []string{"pipelines", "pipeline_versions", "pipeline_runs", "run_steps", "step_control", "step_artifacts"} {
var name string
err := store.db.QueryRow("SELECT name FROM sqlite_master WHERE type='table' AND name=?", table).Scan(&name)
if err == sql.ErrNoRows {
t.Skipf("database lacks the current search schema (missing %q); point PIPELINES_LIVE_SEARCH_DB at a current database", table)
}
if err != nil {
t.Fatal(err)
}
}
}
+601
View File
@@ -0,0 +1,601 @@
package pipeline
import (
"context"
"path/filepath"
"strconv"
"strings"
"testing"
"time"
"unicode/utf8"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// searchFixture builds one pipeline whose name, source, run, stage, field text
// and artifact all contain "zephyr", so a single free-text query exercises
// every kind.
func searchFixture(t *testing.T) (*Store, int64) {
t.Helper()
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "search.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { store.Close() })
if _, err := store.Submit(ctx, "Zephyr Build", Template([]string{"zephyr"}, true)); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
definition, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
run, _, err := store.CreateRun(ctx, definition, "zephyr-key", "/work/zephyr", false, nil, map[string]any{"prompt": "zephyr task"})
if err != nil {
t.Fatal(err)
}
if err := store.BeginStep(ctx, run, 1, "zephyr", map[string]any{"prompt": "zephyr task"}); err != nil {
t.Fatal(err)
}
artifactURL := "/artifacts/" + strconv.FormatInt(run, 10) + "/1/zephyr.txt"
if err := store.FinishStep(ctx, run, protocol.RunStep{
Sequence: 1, Status: "completed", Outcome: "success", Target: protocol.SuccessEnd,
Output: map[string]any{"summary": "zephyr completed"},
Control: &protocol.ControlEvidence{Visit: 1, MaxVisits: 2, ModelTarget: "zephyr-model"},
Artifacts: []protocol.StageArtifact{{Name: "zephyr.txt", SizeBytes: 12, ContentType: "text/plain", URL: artifactURL}},
}); err != nil {
t.Fatal(err)
}
if err := store.FinishRun(ctx, run, "succeeded", "success", "", map[string]any{"summary": "zephyr completed"}); err != nil {
t.Fatal(err)
}
return store, run
}
func searchKinds(response *protocol.SearchResponse) map[string]bool {
kinds := map[string]bool{}
for _, hit := range response.Results {
kinds[hit.Kind] = true
}
return kinds
}
func TestSearchReturnsEveryKind(t *testing.T) {
store, run := searchFixture(t)
response, err := store.Search(context.Background(), protocol.SearchRequest{Query: "zephyr"}, time.Now())
if err != nil {
t.Fatal(err)
}
kinds := searchKinds(response)
for _, kind := range protocol.SearchKinds {
if !kinds[kind] {
t.Fatalf("kind %q missing from %+v", kind, response.Results)
}
}
if response.Cap != protocol.SearchDefaultLimit || response.ScanBoundBytes != protocol.SearchMaxScanBytes {
t.Fatalf("bounds not reported: %+v", response)
}
if response.Truncated {
t.Fatalf("a tiny store must not report truncation: %+v", response)
}
// Every target must point at an addressable entity an existing command resolves.
ctx := context.Background()
for _, hit := range response.Results {
switch hit.Kind {
case protocol.SearchKindPipeline:
id := targetInt(t, hit.Target, "pipeline:")
if _, err := store.Get(ctx, id, 0); err != nil {
t.Fatalf("pipeline target %q does not resolve: %v", hit.Target, err)
}
case protocol.SearchKindVersion:
parts := strings.Split(strings.TrimPrefix(hit.Target, "pipeline:"), ":v")
if len(parts) != 2 {
t.Fatalf("bad version target %q", hit.Target)
}
if _, err := store.Get(ctx, parseInt(t, parts[0]), int(parseInt(t, parts[1]))); err != nil {
t.Fatalf("version target %q does not resolve: %v", hit.Target, err)
}
case protocol.SearchKindRun:
if _, err := store.GetRun(ctx, targetInt(t, hit.Target, "run:")); err != nil {
t.Fatalf("run target %q does not resolve: %v", hit.Target, err)
}
case protocol.SearchKindStep, protocol.SearchKindField:
fields := strings.Split(strings.TrimPrefix(hit.Target, "run:"), ":")
if len(fields) < 2 {
t.Fatalf("bad step/field target %q", hit.Target)
}
detail, err := store.GetRun(ctx, parseInt(t, fields[0]))
if err != nil {
t.Fatalf("step/field target %q does not resolve: %v", hit.Target, err)
}
sequence := parseInt(t, fields[1])
found := false
for _, step := range detail.Steps {
if int64(step.Sequence) == sequence {
found = true
}
}
if !found {
t.Fatalf("step/field target %q names no recorded step", hit.Target)
}
case protocol.SearchKindArtifact:
if !strings.HasPrefix(hit.Target, "/artifacts/") {
t.Fatalf("artifact target %q is not a served URL", hit.Target)
}
}
}
if run <= 0 {
t.Fatal("fixture run missing")
}
}
func targetInt(t *testing.T, target, prefix string) int64 {
t.Helper()
if !strings.HasPrefix(target, prefix) {
t.Fatalf("target %q lacks prefix %q", target, prefix)
}
return parseInt(t, strings.TrimPrefix(target, prefix))
}
func parseInt(t *testing.T, value string) int64 {
t.Helper()
n, err := parsePositiveInt(value)
if err != nil {
t.Fatalf("bad number %q: %v", value, err)
}
return n
}
func TestSearchFieldTextCoversInputOutputEvidence(t *testing.T) {
store, _ := searchFixture(t)
response, err := store.Search(context.Background(), protocol.SearchRequest{Query: "zephyr", Kind: protocol.SearchKindField}, time.Now())
if err != nil {
t.Fatal(err)
}
fields := map[string]bool{}
for _, hit := range response.Results {
if hit.Field == nil {
t.Fatalf("field kind without a field payload: %+v", hit)
}
fields[hit.Field.Field] = true
if !strings.Contains(hit.Field.Snippet, "zephyr") {
t.Fatalf("snippet lacks the match: %+v", hit.Field)
}
}
for _, field := range []string{"input", "output", "evidence"} {
if !fields[field] {
t.Fatalf("field text search missed %q: %+v", field, response.Results)
}
}
}
func TestSearchRankingExactSubstringBodyAndRecency(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "ranking.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
submit := func(name string) {
if _, err := store.Submit(ctx, name, Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
}
submit("alpha")
submit("alphabet")
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
run, _, err := store.CreateRun(ctx, definition, "alpha-key", "/work", false, nil, map[string]any{"prompt": "body alpha match"})
if err != nil {
t.Fatal(err)
}
if err := store.BeginStep(ctx, run, 1, "one", map[string]any{"prompt": "body alpha match"}); err != nil {
t.Fatal(err)
}
if err := store.FinishStep(ctx, run, protocol.RunStep{Sequence: 1, Status: "completed", Outcome: "success", Target: protocol.SuccessEnd}); err != nil {
t.Fatal(err)
}
response, err := store.Search(ctx, protocol.SearchRequest{Query: "alpha"}, time.Now())
if err != nil {
t.Fatal(err)
}
seen := map[int]bool{}
for i, hit := range response.Results {
seen[hit.Score] = true
if i > 0 && response.Results[i-1].Score < hit.Score {
t.Fatalf("scores not descending: %+v", response.Results)
}
}
if !seen[3] || !seen[2] || !seen[1] {
t.Fatalf("expected exact/substring/body tiers: %+v", response.Results)
}
// The exact pipeline match outranks the substring pipeline match.
exactIndex, substringIndex := -1, -1
for i, hit := range response.Results {
if hit.Pipeline == nil {
continue
}
if hit.Pipeline.Name == "alpha" {
exactIndex = i
}
if hit.Pipeline.Name == "alphabet" {
substringIndex = i
}
if hit.Pipeline.Name == "alpha" && hit.Score != 3 {
t.Fatalf("exact name scored %d", hit.Score)
}
if hit.Pipeline.Name == "alphabet" && hit.Score != 2 {
t.Fatalf("substring name scored %d", hit.Score)
}
}
if exactIndex < 0 || substringIndex < 0 || exactIndex > substringIndex {
t.Fatalf("exact name must outrank a name substring: %+v", response.Results)
}
// Equal-tier ties break by recency.
older, _, err := store.CreateRun(ctx, definition, "alpha-older", "/work", false, nil, map[string]any{"prompt": "alpha"})
if err != nil {
t.Fatal(err)
}
newer, _, err := store.CreateRun(ctx, definition, "alpha-newer", "/work", false, nil, map[string]any{"prompt": "alpha"})
if err != nil {
t.Fatal(err)
}
setCreated(t, store, older, "2026-01-01 00:00:00")
setCreated(t, store, newer, "2026-06-01 00:00:00")
response, err = store.Search(ctx, protocol.SearchRequest{Query: "alpha"}, time.Now())
if err != nil {
t.Fatal(err)
}
var olderIndex, newerIndex = -1, -1
for i, hit := range response.Results {
if hit.Run == nil || hit.Run.ID == 0 {
continue
}
if hit.Run.ID == older {
olderIndex = i
}
if hit.Run.ID == newer {
newerIndex = i
}
}
if olderIndex < 0 || newerIndex < 0 || newerIndex > olderIndex {
t.Fatalf("recency tie-break lost: older=%d newer=%d", olderIndex, newerIndex)
}
}
func TestSearchStatusWordsSelectRuns(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "statuses.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "statuses", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
// One run per accepted status word, each with a distinct created_at so a
// date selector can never be mistaken for the status selector.
byStatus := map[string]int64{}
for i, status := range protocol.SearchStatuses {
run, _, err := store.CreateRun(ctx, definition, "status-"+status, "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := store.db.Exec("UPDATE pipeline_runs SET status=?,created_at=? WHERE id=?", status, "2026-01-0"+strconv.Itoa(i+1)+" 09:00:00", run); err != nil {
t.Fatal(err)
}
byStatus[status] = run
}
now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
for _, status := range protocol.SearchStatuses {
response, err := store.Search(ctx, protocol.SearchRequest{Query: status}, now)
if err != nil {
t.Fatal(err)
}
var runs []int64
for _, hit := range response.Results {
if hit.Run == nil {
t.Fatalf("status %q returned a non-run kind %q", status, hit.Kind)
}
runs = append(runs, hit.Run.ID)
}
if len(runs) != 1 || runs[0] != byStatus[status] {
t.Fatalf("status %q selected %v, want [%d]", status, runs, byStatus[status])
}
}
}
func TestSearchStatusAndDateSelectors(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "selector.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "selectors", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
yesterdayRun, _, err := store.CreateRun(ctx, definition, "yesterday", "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
todayRun, _, err := store.CreateRun(ctx, definition, "today", "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
if err := store.FinishRun(ctx, yesterdayRun, "failed", "error", "boom", nil); err != nil {
t.Fatal(err)
}
setCreated(t, store, yesterdayRun, "2026-10-15 09:00:00")
setCreated(t, store, todayRun, "2026-10-16 09:00:00")
now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
selects := func(query string) map[int64]bool {
t.Helper()
response, err := store.Search(ctx, protocol.SearchRequest{Query: query}, now)
if err != nil {
t.Fatal(err)
}
ids := map[int64]bool{}
for _, hit := range response.Results {
if hit.Run != nil {
ids[hit.Run.ID] = true
}
}
return ids
}
if ids := selects("failed"); !ids[yesterdayRun] || ids[todayRun] {
t.Fatalf("status selector: %+v", ids)
}
if ids := selects("today"); !ids[todayRun] || ids[yesterdayRun] {
t.Fatalf("today selector: %+v", ids)
}
if ids := selects("yesterday"); !ids[yesterdayRun] || ids[todayRun] {
t.Fatalf("yesterday selector: %+v", ids)
}
if ids := selects("2026-10-15"); !ids[yesterdayRun] || ids[todayRun] {
t.Fatalf("literal date selector: %+v", ids)
}
if ids := selects("this week"); !ids[yesterdayRun] || !ids[todayRun] {
t.Fatalf("this week selector: %+v", ids)
}
// Non-selector text stays free text.
if ids := selects("selectors"); !ids[yesterdayRun] || !ids[todayRun] {
t.Fatalf("free text was misread as a selector: %+v", ids)
}
}
func TestParseSearchTerms(t *testing.T) {
now := time.Date(2026, 10, 16, 12, 0, 0, 0, time.UTC)
cases := []struct {
query string
status string
hasDate bool
from, to string
text string
}{
{query: "failed", status: "failed"},
{query: " RUNNING ", status: "running"},
{query: "today", hasDate: true, from: "2026-10-16 00:00:00", to: "2026-10-17 00:00:00"},
{query: "yesterday", hasDate: true, from: "2026-10-15 00:00:00", to: "2026-10-16 00:00:00"},
{query: "this week", hasDate: true, from: "2026-10-12 00:00:00", to: "2026-10-19 00:00:00"},
{query: "2026-10-05", hasDate: true, from: "2026-10-05 00:00:00", to: "2026-10-06 00:00:00"},
{query: "failed to build", text: "failed to build"},
{query: "build", text: "build"},
}
for _, test := range cases {
terms := parseSearchTerms(test.query, now)
if terms.Status != test.status || terms.Text != test.text || terms.HasDate != test.hasDate {
t.Fatalf("parse %q = %+v", test.query, terms)
}
if test.hasDate {
if got := terms.From.Format(searchSQLTime); got != test.from {
t.Fatalf("parse %q from = %s, want %s", test.query, got, test.from)
}
if got := terms.To.Format(searchSQLTime); got != test.to {
t.Fatalf("parse %q to = %s, want %s", test.query, got, test.to)
}
}
}
}
func TestSearchKindFilterAndLimits(t *testing.T) {
store, _ := searchFixture(t)
ctx := context.Background()
response, err := store.Search(ctx, protocol.SearchRequest{Query: "zephyr", Kind: protocol.SearchKindRun}, time.Now())
if err != nil {
t.Fatal(err)
}
for _, hit := range response.Results {
if hit.Kind != protocol.SearchKindRun {
t.Fatalf("kind filter leaked %q", hit.Kind)
}
}
if _, err := store.Search(ctx, protocol.SearchRequest{Query: "zephyr", Kind: "bogus"}, time.Now()); err == nil {
t.Fatal("invalid kind must return a clear error")
}
for _, test := range []struct {
limit int
want int
}{{limit: 0, want: protocol.SearchDefaultLimit}, {limit: -5, want: protocol.SearchDefaultLimit}, {limit: 1, want: 1}, {limit: 500, want: protocol.SearchMaxLimit}} {
response, err := store.Search(ctx, protocol.SearchRequest{Query: "zephyr", Limit: test.limit}, time.Now())
if err != nil {
t.Fatal(err)
}
if response.Cap != test.want {
t.Fatalf("limit %d => cap %d, want %d", test.limit, response.Cap, test.want)
}
if len(response.Results) > response.Cap {
t.Fatalf("limit %d returned %d results", test.limit, len(response.Results))
}
}
// A cap below the candidate count drops matches and must report Truncated.
truncated, err := store.Search(ctx, protocol.SearchRequest{Query: "zephyr", Limit: 1}, time.Now())
if err != nil {
t.Fatal(err)
}
if len(truncated.Results) != 1 || !truncated.Truncated {
t.Fatalf("result cap must truncate: %+v", truncated)
}
}
func TestSearchEdgeCases(t *testing.T) {
store, run := searchFixture(t)
ctx := context.Background()
empty, err := store.Search(ctx, protocol.SearchRequest{Query: " "}, time.Now())
if err != nil || len(empty.Results) != 0 {
t.Fatalf("empty query: %+v %v", empty, err)
}
none, err := store.Search(ctx, protocol.SearchRequest{Query: "no-such-token-xyzzy"}, time.Now())
if err != nil || len(none.Results) != 0 {
t.Fatalf("no-match query: %+v %v", none, err)
}
if none.Results == nil {
t.Fatal("results must be an empty slice, not nil")
}
// A run without a finish reports no duration.
setFinishedNull(t, store, run)
response, err := store.Search(ctx, protocol.SearchRequest{Query: "zephyr", Kind: protocol.SearchKindRun}, time.Now())
if err != nil {
t.Fatal(err)
}
for _, hit := range response.Results {
if hit.Run != nil && hit.Run.FinishedAt != "" {
t.Fatalf("null finished_at surfaced a value: %+v", hit.Run)
}
}
}
func TestSearchSnippetBoundaries(t *testing.T) {
if got := searchSnippet("prefix café résumé naïve suffix", "naïve"); !strings.Contains(got, "naïve") || !utf8.ValidString(got) {
t.Fatalf("unicode snippet malformed: %q", got)
}
if got := searchSnippet("line one\n\tline two\r\n zephyr here", "zephyr"); strings.ContainsAny(got, "\n\r\t") || !strings.Contains(got, "zephyr") {
t.Fatalf("whitespace snippet not normalized: %q", got)
}
if got := searchSnippet(strings.Repeat("é", 200)+"needle"+strings.Repeat("é", 200), "needle"); !utf8.ValidString(got) || !strings.Contains(got, "needle") {
t.Fatalf("boundary snippet malformed: %q", got)
}
if got := searchSnippet("no match here", "absent"); got == "" || !utf8.ValidString(got) {
t.Fatalf("no-match snippet must stay a clean line: %q", got)
}
}
func TestSearchScanBoundTruncates(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "scan.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "scan", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
// An older step that does match, and a newer step whose body alone exceeds
// the scan budget: the budget must stop the scan and be reported.
older, _, err := store.CreateRun(ctx, definition, "scan-older", "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
if err := store.BeginStep(ctx, older, 1, "one", nil); err != nil {
t.Fatal(err)
}
if err := store.FinishStep(ctx, older, protocol.RunStep{Sequence: 1, Status: "completed", Outcome: "success", Target: protocol.SuccessEnd, Output: map[string]any{"summary": "needle in older step"}}); err != nil {
t.Fatal(err)
}
newer, _, err := store.CreateRun(ctx, definition, "scan-newer", "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
if err := store.BeginStep(ctx, newer, 1, "one", nil); err != nil {
t.Fatal(err)
}
blob := strings.Repeat("x", protocol.SearchMaxScanBytes+1024)
if err := store.FinishStep(ctx, newer, protocol.RunStep{Sequence: 1, Status: "completed", Outcome: "success", Target: protocol.SuccessEnd, Output: map[string]any{"blob": blob}}); err != nil {
t.Fatal(err)
}
response, err := store.Search(ctx, protocol.SearchRequest{Query: "needle"}, time.Now())
if err != nil {
t.Fatal(err)
}
if !response.Truncated {
t.Fatalf("scan bound not reported: %+v", response)
}
for _, hit := range response.Results {
if hit.Field != nil {
t.Fatalf("scan passed the bound and matched: %+v", hit.Field)
}
}
}
// TestSearchScanBoundTruncatesByRowCount proves the step-count bound is
// observable: more recorded steps than the budget sets Truncated even though
// every body is tiny and fits the byte budget.
func TestSearchScanBoundTruncatesByRowCount(t *testing.T) {
ctx := context.Background()
store, err := Open(filepath.Join(t.TempDir(), "scanrows.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "scanrows", Template([]string{"one"}, true)); err != nil {
t.Fatal(err)
}
items, _ := store.List(ctx)
definition, _ := store.Get(ctx, items[0].ID, 1)
run, _, err := store.CreateRun(ctx, definition, "scanrows", "/work", false, nil, nil)
if err != nil {
t.Fatal(err)
}
tx, err := store.db.Begin()
if err != nil {
t.Fatal(err)
}
for sequence := 1; sequence <= protocol.SearchMaxScanSteps+1; sequence++ {
if _, err := tx.Exec(`INSERT INTO run_steps(run_id,sequence,stage_key,input_json) VALUES(?,?,?,?)`, run, sequence, "one", "{}"); err != nil {
t.Fatal(err)
}
}
if err := tx.Commit(); err != nil {
t.Fatal(err)
}
response, err := store.Search(ctx, protocol.SearchRequest{Query: "needle"}, time.Now())
if err != nil {
t.Fatal(err)
}
if !response.Truncated {
t.Fatalf("step-count bound not reported: scanned_steps=%d total=%d", response.ScannedSteps, protocol.SearchMaxScanSteps+1)
}
if response.ScannedSteps != protocol.SearchMaxScanSteps {
t.Fatalf("scanned_steps=%d, want %d", response.ScannedSteps, protocol.SearchMaxScanSteps)
}
if response.ScannedBytes > protocol.SearchMaxScanBytes {
t.Fatalf("scanned_bytes=%d exceeds bound %d", response.ScannedBytes, protocol.SearchMaxScanBytes)
}
}
func setCreated(t *testing.T, store *Store, runID int64, timestamp string) {
t.Helper()
if _, err := store.db.Exec("UPDATE pipeline_runs SET created_at=? WHERE id=?", timestamp, runID); err != nil {
t.Fatal(err)
}
}
func setFinishedNull(t *testing.T, store *Store, runID int64) {
t.Helper()
if _, err := store.db.Exec("UPDATE pipeline_runs SET finished_at=NULL,status='running' WHERE id=?", runID); err != nil {
t.Fatal(err)
}
}
+21
View File
@@ -8,6 +8,7 @@ import (
"fmt"
"git.a-wan.net/alan/pipelines/internal/protocol"
"git.a-wan.net/alan/pipelines/internal/stage"
"os"
"path/filepath"
"strings"
"sync"
@@ -50,6 +51,26 @@ CREATE TABLE IF NOT EXISTS submissions (id INTEGER PRIMARY KEY, name TEXT NOT NU
func (s *Store) Close() error { return s.db.Close() }
// OpenReadOnly opens an existing database without running schema DDL or
// changing journal mode, so a live database can be queried read-only while its
// server keeps writing. It fails when the file does not exist. It is used by
// tests that inspect a real database and is never part of server startup.
func OpenReadOnly(path string) (*Store, error) {
if _, err := os.Stat(path); err != nil {
return nil, err
}
db, err := sql.Open("sqlite", "file:"+path+"?mode=ro")
if err != nil {
return nil, err
}
db.SetMaxOpenConns(1)
if _, err := db.Exec("PRAGMA busy_timeout=5000"); err != nil {
db.Close()
return nil, err
}
return &Store{db: db, dir: filepath.Dir(path)}, nil
}
// Directory is the resolved directory holding the database, used to place
// adjacent server state such as stage artifacts.
func (s *Store) Directory() string { return s.dir }
+5
View File
@@ -133,6 +133,11 @@ type Message struct {
WritePaths []string `json:"write_paths,omitempty"`
RequestKey string `json:"request_key,omitempty"`
Workspace string `json:"workspace,omitempty"`
// Search is the broad, read-only search traffic over pipelines, versions,
// runs, steps, field text and artifacts. See search.go for the per-type
// payloads and the ranking/kind/bound vocabulary.
Search *SearchRequest `json:"search,omitempty"`
SearchResults *SearchResponse `json:"search_results,omitempty"`
// Chat is the brokered OpenCode chat traffic (route B). See chat.go for the
// per-type payload structs and the wire shapes they mirror.
ChatSessions []ChatSession `json:"chat_sessions,omitempty"`
+158
View File
@@ -0,0 +1,158 @@
package protocol
// Search brokered by the Pipelines server: one query can find a pipeline, a
// pipeline version, a run, a recorded step/visit, the text of a step field
// (input/output/evidence/transcript) or a recorded artifact. A client sends
// Message{Type: Search, ID: <opaque>, Search: <SearchRequest>} and the server
// echoes the ID with a SearchResults response, matching every other exchange.
const (
Search = "search"
SearchResults = "search_results"
)
// Search.Kind narrowing values. They are the only narrowing mechanism and are
// enforced server-side; an empty Kind searches every kind.
const (
SearchKindPipeline = "pipeline"
SearchKindVersion = "version"
SearchKindRun = "run"
SearchKindStep = "step"
SearchKindField = "field"
SearchKindArtifact = "artifact"
)
// SearchKinds is the accepted Kind vocabulary in display order; the CLI help
// and notes are kept in step with it.
var SearchKinds = []string{
SearchKindPipeline,
SearchKindVersion,
SearchKindRun,
SearchKindStep,
SearchKindField,
SearchKindArtifact,
}
// SearchStatuses is the accepted set of bare status words a query may name to
// select runs. It mirrors the run status vocabulary; a bare word selects runs
// rather than matching as free text.
var SearchStatuses = []string{"queued", "running", "succeeded", "failed", "canceled", "interrupted"}
// Search budget and cap constants. They are fixed, enforced server-side and
// reported on SearchResponse so a client can see the bound that applied.
const (
// SearchDefaultLimit is the result cap when a request omits Limit (or sends 0).
SearchDefaultLimit = 50
// SearchMaxLimit is the largest accepted result cap; larger requests clamp.
SearchMaxLimit = 200
// SearchMaxScanSteps bounds how many recorded steps one query inspects while
// scanning input/output/transcript field text.
SearchMaxScanSteps = 2000
// SearchMaxScanBytes bounds the total field bytes (input_json + output_json +
// evidence_json + transcript) one query decodes and searches before it stops
// and reports Truncated.
SearchMaxScanBytes = 1 << 20
)
// SearchRequest is the search request body. Query is the free text, a bare
// status word or a date expression; Kind optionally narrows to one SearchKind;
// Limit caps the returned results (0 = server default).
type SearchRequest struct {
Query string `json:"query"`
Kind string `json:"kind,omitempty"`
Limit int `json:"limit,omitempty"`
}
// SearchResponse is the ranked, bounded set of matches. Results are ordered by
// score, then recency, then a stable kind/target tie-break. Truncated reports
// that either the result cap or the field-scan budget dropped matches;
// ScannedSteps/ScannedBytes and ScanBoundBytes make the body-scan bound
// observable.
type SearchResponse struct {
Query string `json:"query"`
Kind string `json:"kind,omitempty"`
Results []SearchResult `json:"results"`
Truncated bool `json:"truncated,omitempty"`
ScannedSteps int `json:"scanned_steps"`
ScannedBytes int `json:"scanned_bytes"`
Cap int `json:"cap"`
ScanBoundBytes int `json:"scan_bound_bytes"`
}
// SearchResult is one typed, navigable match. Kind selects which payload is
// set; Target is a stable token an existing CLI command can resolve
// (pipeline:<id>, pipeline:<id>:v<version>, run:<id>, run:<id>:<sequence>,
// run:<id>:<sequence>:<field>, or an artifact URL). Score and MatchedField
// expose the ranking tier and why the hit matched so a client can re-rank;
// Recency is the hit's sortable UTC timestamp.
type SearchResult struct {
Kind string `json:"kind"`
Target string `json:"target"`
Score int `json:"score"`
MatchedField string `json:"matched_field"`
Recency string `json:"recency,omitempty"`
Pipeline *SearchPipeline `json:"pipeline,omitempty"`
Version *SearchVersion `json:"version,omitempty"`
Run *SearchRun `json:"run,omitempty"`
Step *SearchStep `json:"step,omitempty"`
Field *SearchField `json:"field,omitempty"`
Artifact *SearchArtifact `json:"artifact,omitempty"`
}
// SearchPipeline is a pipeline name/identity hit: its latest version and stage count.
type SearchPipeline struct {
ID int64 `json:"id"`
Name string `json:"name"`
Version int `json:"version"`
Stages int `json:"stages"`
}
// SearchVersion is a pipeline version/source hit.
type SearchVersion struct {
PipelineID int64 `json:"pipeline_id"`
Version int `json:"version"`
ContentHash string `json:"content_hash"`
CreatedAt string `json:"created_at"`
}
// SearchRun is a run identity/metadata hit. FinishedAt and DurationSeconds are
// omitted for a run without a recorded finish.
type SearchRun struct {
ID int64 `json:"id"`
PipelineName string `json:"pipeline_name"`
Status string `json:"status"`
Workspace string `json:"workspace,omitempty"`
CreatedAt string `json:"created_at"`
FinishedAt string `json:"finished_at,omitempty"`
DurationSeconds float64 `json:"duration_seconds,omitempty"`
}
// SearchStep is a recorded step/visit hit.
type SearchStep struct {
RunID int64 `json:"run_id"`
Sequence int `json:"sequence"`
Stage string `json:"stage"`
Visit int `json:"visit,omitempty"`
Outcome string `json:"outcome,omitempty"`
Target string `json:"target,omitempty"`
Diagnostic string `json:"diagnostic,omitempty"`
}
// SearchField is a stage input/output/evidence/transcript text hit with a
// snippet around the first match.
type SearchField struct {
RunID int64 `json:"run_id"`
Sequence int `json:"sequence"`
Field string `json:"field"`
Snippet string `json:"snippet"`
}
// SearchArtifact is a recorded artifact hit; contents are never read, only the
// recorded metadata.
type SearchArtifact struct {
RunID int64 `json:"run_id"`
Sequence int `json:"sequence"`
Name string `json:"name"`
ContentType string `json:"content_type"`
SizeBytes int64 `json:"size_bytes"`
URL string `json:"url"`
}
+82
View File
@@ -0,0 +1,82 @@
package protocol
import (
"encoding/json"
"strings"
"testing"
)
func TestSearchRequestResponseRoundTrip(t *testing.T) {
request := Message{Type: Search, ID: "req-1", Search: &SearchRequest{Query: "release", Kind: SearchKindRun, Limit: 5}}
encoded, err := json.Marshal(request)
if err != nil {
t.Fatal(err)
}
var decoded Message
if err := json.Unmarshal(encoded, &decoded); err != nil {
t.Fatal(err)
}
if decoded.Type != Search || decoded.ID != "req-1" || decoded.Search == nil {
t.Fatalf("request envelope lost: %+v", decoded)
}
if decoded.Search.Query != "release" || decoded.Search.Kind != SearchKindRun || decoded.Search.Limit != 5 {
t.Fatalf("search request lost: %+v", decoded.Search)
}
response := Message{Type: SearchResults, ID: "req-1", SearchResults: &SearchResponse{
Query: "release",
Results: []SearchResult{{Kind: SearchKindPipeline, Target: "pipeline:1", Score: 3, MatchedField: "name", Pipeline: &SearchPipeline{ID: 1, Name: "release", Version: 2, Stages: 3}}},
Cap: 50,
ScanBoundBytes: SearchMaxScanBytes,
}}
encoded, err = json.Marshal(response)
if err != nil {
t.Fatal(err)
}
var echoed Message
if err := json.Unmarshal(encoded, &echoed); err != nil {
t.Fatal(err)
}
if echoed.ID != "req-1" || echoed.Type != SearchResults || echoed.SearchResults == nil {
t.Fatalf("response envelope or ID echo lost: %+v", echoed)
}
got := echoed.SearchResults
if got.Query != "release" || got.Cap != 50 || got.ScanBoundBytes != SearchMaxScanBytes || len(got.Results) != 1 {
t.Fatalf("search response lost: %+v", got)
}
hit := got.Results[0]
if hit.Pipeline == nil || hit.Pipeline.Name != "release" || hit.Pipeline.Version != 2 || hit.Pipeline.Stages != 3 || hit.Score != 3 || hit.MatchedField != "name" || hit.Target != "pipeline:1" {
t.Fatalf("pipeline payload lost: %+v", hit)
}
}
func TestSearchResultOmitsAbsentPayloadsAndEmptyKind(t *testing.T) {
encoded, err := json.Marshal(SearchResult{Kind: SearchKindRun, Target: "run:1", Score: 1, MatchedField: "status"})
if err != nil {
t.Fatal(err)
}
for _, key := range []string{"pipeline", "version", "run", "step", "field", "artifact", "recency"} {
if strings.Contains(string(encoded), `"`+key+`":`) {
t.Fatalf("absent %s payload must be omitted: %s", key, encoded)
}
}
var decoded SearchResult
if err := json.Unmarshal([]byte(`{"kind":"run","target":"run:1","score":1,"matched_field":"status"}`), &decoded); err != nil {
t.Fatal(err)
}
if decoded.Pipeline != nil || decoded.Version != nil || decoded.Run != nil || decoded.Step != nil || decoded.Field != nil || decoded.Artifact != nil {
t.Fatalf("absent payloads must decode nil: %+v", decoded)
}
response := Message{Type: SearchResults, ID: "x", SearchResults: &SearchResponse{Query: "q", Results: []SearchResult{}}}
encoded, err = json.Marshal(response)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(encoded), `"kind"`) {
t.Fatalf("empty response kind must be omitted: %s", encoded)
}
if !strings.Contains(string(encoded), `"results":[]`) {
t.Fatalf("results must always be present: %s", encoded)
}
}
+105
View File
@@ -0,0 +1,105 @@
package server
import (
"bytes"
"context"
"encoding/json"
"net/http/httptest"
"path/filepath"
"strconv"
"strings"
"testing"
"git.a-wan.net/alan/pipelines/internal/cli"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// TestCLISearchEndToEndEveryKind seeds a store whose pipeline, source, run,
// stage, field text and artifact all contain one token, then runs the CLI
// search command over a real Handler and asserts the server returned one match
// of every kind and rejects an unknown --type rather than ignoring it.
func TestCLISearchEndToEndEveryKind(t *testing.T) {
t.Setenv("PIPELINES_HARNESS_CONFIG", filepath.Join(t.TempDir(), "harnesses.json"))
ctx, cancel := context.WithCancel(context.Background())
store, err := pipeline.Open(filepath.Join(t.TempDir(), "search-e2e.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
if _, err := store.Submit(ctx, "Zephyr Build", pipeline.Template([]string{"zephyr"}, true)); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
definition, err := store.Get(ctx, items[0].ID, 1)
if err != nil {
t.Fatal(err)
}
run, _, err := store.CreateRun(ctx, definition, "zephyr-key", "/work/zephyr", false, nil, map[string]any{"prompt": "zephyr task"})
if err != nil {
t.Fatal(err)
}
if err := store.BeginStep(ctx, run, 1, "zephyr", map[string]any{"prompt": "zephyr task"}); err != nil {
t.Fatal(err)
}
if err := store.FinishStep(ctx, run, protocol.RunStep{
Sequence: 1, Status: "completed", Outcome: "success", Target: protocol.SuccessEnd,
Output: map[string]any{"summary": "zephyr completed"},
Artifacts: []protocol.StageArtifact{{Name: "zephyr.txt", SizeBytes: 12, ContentType: "text/plain", URL: "/artifacts/" + strconv.FormatInt(run, 10) + "/1/zephyr.txt"}},
}); err != nil {
t.Fatal(err)
}
handler := Handler(ctx, store)
defer func() { cancel(); WaitRuns(handler) }()
http := httptest.NewServer(handler)
defer http.Close()
url := "ws" + strings.TrimPrefix(http.URL, "http") + "/connect"
runCLI := func(args ...string) (string, error) {
var out bytes.Buffer
app := cli.App{Out: &out, Err: &out, In: strings.NewReader("")}
err := app.Run(ctx, append([]string{"--url", url}, args...))
return out.String(), err
}
output, err := runCLI("search", "zephyr", "--json")
if err != nil {
t.Fatal(err)
}
var response protocol.SearchResponse
if err := json.Unmarshal([]byte(output), &response); err != nil {
t.Fatal(err, output)
}
kinds := map[string]bool{}
for _, hit := range response.Results {
kinds[hit.Kind] = true
if hit.Target == "" || hit.MatchedField == "" || hit.Score == 0 {
t.Fatalf("hit lacks ranking/navigation context: %+v", hit)
}
}
for _, kind := range protocol.SearchKinds {
if !kinds[kind] {
t.Fatalf("kind %q missing from end-to-end search: %s", kind, output)
}
}
if output, err := runCLI("search", "zephyr", "--type", "run", "--json"); err != nil {
t.Fatal(err)
} else {
var filtered protocol.SearchResponse
if err := json.Unmarshal([]byte(output), &filtered); err != nil {
t.Fatal(err)
}
for _, hit := range filtered.Results {
if hit.Kind != protocol.SearchKindRun {
t.Fatalf("kind filter leaked %q", hit.Kind)
}
}
}
if _, err := runCLI("search", "zephyr", "--type", "bogus"); err == nil || !strings.Contains(err.Error(), "unknown search kind") {
t.Fatalf("invalid kind must error, got %v", err)
}
}
+16
View File
@@ -192,6 +192,22 @@ func HandlerWithOptions(ctx context.Context, options Options, stores ...*pipelin
} else {
response.Pipelines = items
}
} else if request.Type == protocol.Search {
if len(stores) == 0 {
response.Type = protocol.Error
response.Error = "search unavailable: no store"
} else if request.Search == nil {
response.Type = protocol.Error
response.Error = "search requires a query"
} else {
response.Type = protocol.SearchResults
var err error
response.SearchResults, err = stores[0].Search(ctx, *request.Search, time.Now())
if err != nil {
response.Type = protocol.Error
response.Error = err.Error()
}
}
} else if runs != nil && (request.Type == protocol.RunPipeline || request.Type == protocol.RetryRun || request.Type == protocol.ListRuns || request.Type == protocol.GetRun || request.Type == protocol.CancelRun || request.Type == protocol.GetRunEvidence) {
var err error
switch request.Type {