Handle native structured-format incompatibility and preserve failure evidence

This commit is contained in:
2026-10-10 00:01:02 +01:00
parent 16389b8270
commit a1c0eb4a72
21 changed files with 800 additions and 42 deletions
+13
View File
@@ -79,6 +79,19 @@ the step transcript. Credentials are redacted, but project/provider messages may
still contain private context: review before sharing. Older failures cannot gain
discarded subprocess output retroactively; Debug attempts retained session reads.
Native OpenCode schema support is checked with a retained, deny-all, no-inference
POST/readback probe. OpenCode 1.18.35 accepts explicit formats but rejects their
stored messages on readback; affected backends omit `format` and require the same
strict local `{outcome, output}` validation. Compatible backends retain schema
mode with no automatic tool-replaying schema retries. The compatibility decision
and probe session are recorded, without rewriting published definitions/history.
For old format-bearing sessions whose HTTP readback fails, Debug uses read-only
`opencode export` with an exact session-ID check to recover original messages.
`StructuredOutputError` is surfaced directly; DSML/tool-call markup and flattened
results are rejected and retained, never silently interpreted as successful tool
execution or wrapped into a guessed result. Native HTTP evidence distinguishes
adapter/HTTP failures from actual subprocess exit codes.
To validate a model on the execution computer, run:
```sh
+24 -4
View File
@@ -5,13 +5,24 @@ Every model result must be one object containing exactly `outcome` and `output`.
(including nested output keys), missing fields, unknown fields and invalid
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
The native stable OpenCode adapter requests `format.type=json_schema` with this
envelope schema and `retryCount=0`, reads the returned structured result when
available, and preserves the original tool transcript. Local validation remains
mandatory. V2's published prompt/generate API does not accept a schema-output
The native stable OpenCode adapter first checks the complete format POST/readback
contract in a separate retained, deny-all, no-inference session. Compatible native
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
OpenCode 1.18.35 accepts a format but rejects its stored messages with
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
incompatibility, requests omit `format`; strict local validation remains mandatory.
Other probe failures, such as authentication or network errors, are not silently
downgraded. Returned structured results remain preferred when available, and tool
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
field; that adapter retains local validation instead of sending unsupported
fields or claiming provider-side schema enforcement.
Output-contract prose describes the map **inside** `output`; it never overrides
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
is not evidence of a real tool invocation and cannot be accepted as a result.
Flattened output fields are rejected, not guessed into a new envelope. Invalid
responses are retained even when they are not eligible for syntax repair.
## Format recovery is not execution retry
Parser errors retain their diagnostic and byte offset: for example, `missing
@@ -46,6 +57,15 @@ and a recognized safe error category (such as session not found or service
unavailable). Arbitrary stderr, provider messages and response bodies are not
copied into the diagnostic. A `/wait` command failure is not evidence that the
agent prompt failed to start, and does not trigger replay of stage work.
Native `StructuredOutputError` is named directly, rather than reduced to a generic
command failure. Native HTTP evidence records the actual HTTP operation/status;
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
Cleanup failures preserve the original generation error and active ownership.
For existing native sessions with format-bearing messages, the HTTP readback
workaround is read-only CLI `opencode export`. The exported session ID must match
the requested ID. Nothing is rewritten or removed from the database/history.
Debug receives the original messages plus a note identifying that recovery path.
## Optional machine-readable restrictions
+58
View File
@@ -0,0 +1,58 @@
# Native structured-output regression (2026-10-09)
The supplied `~/findings.md` describes Math Demo run #12 on another computer,
using OpenCode 1.18.35 and `opencode-go/deepseek-v4.1-flash`. Its recorded response
contains `StructuredOutputError` and DSML as text, with flattened output fields.
The supplied evidence does not identify the component responsible for converting
that provider tool call into text. That upstream cause remains unconfirmed.
## Confirmed independently
On a fresh `ghcr.io/anomalyco/opencode:1.18.35` container, without credentials or
inference:
- `POST /session/ID/message` with `noReply:true` and an explicit JSON-schema
format succeeds, but subsequent HTTP message readback fails with HTTP 400.
- Omitting `retryCount` does not fix it. Explicit `format:{type:"text"}` also fails.
- Omitting `format` makes both POST and message readback succeed.
- `opencode export ID` recovers the original format-bearing messages which the
HTTP readback cannot encode. No stored message needs to be modified.
The adapter now probes this contract without inference. Affected native backends
omit the incompatible field and enforce the outcome/output envelope locally.
Compatible native schema mode is retained and covered by the mock HTTP contract
tests; a real schema-tool invocation on a compatible native release is **not tested**.
No automatic stage/tool replay or DSML-to-result interpretation was added.
## Real provider regression
The opt-in server test uses an equivalent math graph, not a copy of the remote
immutable definition (its complete source was not supplied). It deliberately
uses small-map output-contract prose to exercise the envelope clarification.
Using the existing local native OpenCode 1.18.35/OpenCode Go credentials, it passed:
- `7 + 2`: sum → multiply → parity → add_one → report; result **28**;
`7 + 2 = 9; 9 * 3 = 27; 27 + 1 = 28 (odd branch)`.
- `7 + 3`: sum → multiply → parity → halve → report; result **15**;
`7 + 3 = 10; 10 * 3 = 30; 30 / 2 = 15 (even branch)`.
- Native retained-session readback through server evidence retrieval passed.
The first inference check reached result 28 but caught an ambiguous *test prompt*
that yielded the literal word `previous` in an equation. The fixture now explicitly
specifies numeric substitutions; the full odd/even check then passed. Published
pipeline definitions were not rewritten.
Commands (supply a native CLI path on your computer):
```sh
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode-1.18.35 \
go test -count=1 -run TestStableNativeSessionContractLive -v ./internal/opencode
# Explicit opt-in: existing credentials, ten small paid-provider calls,
# retained stage/probe sessions, and no project tools.
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode-1.18.35 \
PIPELINES_TEST_NATIVE_INFERENCE=1 \
go test -count=1 -run TestNativeMathDemoLive -v ./internal/server
```
The other computer's original run/session was not accessed or modified here.
+9
View File
@@ -32,6 +32,12 @@ type Connection struct {
done chan error
models map[string]map[string]string
responses map[string][]byte
formatChecked bool
formatSupported bool
FormatProbe *FormatProbe
LastHTTPStatus int
LastHTTPOperation string
MessageReadbackDiagnostic string
}
// Reference records only our ephemeral loopback service credentials, never provider auth.
@@ -210,6 +216,8 @@ func (c *Connection) Close() {
}
func (c *Connection) Native(ctx context.Context, method, path string, body any) ([]byte, error) {
c.LastHTTPStatus = 0
c.LastHTTPOperation = strings.ToUpper(method) + " " + path
u, err := url.Parse(c.URL + path)
if err != nil {
return nil, err
@@ -241,6 +249,7 @@ func (c *Connection) Native(ctx context.Context, method, path string, body any)
return nil, fmt.Errorf("OpenCode native API request failed")
}
defer response.Body.Close()
c.LastHTTPStatus = response.StatusCode
if response.StatusCode < 200 || response.StatusCode >= 300 {
// The stage adapter stores bounded, redacted failure evidence separately;
// the public error string remains free of provider bodies and credentials.
+90
View File
@@ -0,0 +1,90 @@
package opencode
import (
"context"
"encoding/json"
"fmt"
"strings"
)
// FormatProbe is retained, deny-all and no-inference. Some native releases
// accept a format on POST but reject their own stored user message on GET.
type FormatProbe struct {
SessionID string `json:"session_id"`
Supported bool `json:"supported"`
Diagnostic string `json:"diagnostic,omitempty"`
Response string `json:"response,omitempty"`
}
func (c *Connection) supportsFormat(ctx context.Context, format json.RawMessage) (bool, error) {
if c.formatChecked {
return c.formatSupported, nil
}
created, err := c.Native(ctx, "post", "/session", map[string]any{"title": "Pipelines structured-format compatibility probe (no inference)", "permission": []map[string]string{{"permission": "*", "pattern": "*", "action": "deny"}}})
if err != nil {
return false, err
}
var session struct {
ID string `json:"id"`
}
if json.Unmarshal(created, &session) != nil || !sessionID.MatchString(session.ID) {
return false, fmt.Errorf("invalid structured-format probe session")
}
c.FormatProbe = &FormatProbe{SessionID: session.ID}
posted, err := c.Native(ctx, "post", "/session/"+session.ID+"/message", map[string]any{"noReply": true, "format": format, "parts": []map[string]string{{"type": "text", "text": "Compatibility probe only. No reply, inference or tool execution requested."}}})
if err != nil {
c.FormatProbe.Response = string(posted)
return false, err
}
data, err := c.Native(ctx, "get", "/session/"+session.ID+"/message", nil)
if err != nil {
// Downgrade only this specific readback incompatibility, never an auth,
// network or unrelated HTTP failure. Do not alter the stored probe.
c.FormatProbe.Response = string(data)
if !strings.Contains(err.Error(), "HTTP 400") || !strings.Contains(string(data), "Expected OutputFormat") {
return false, err
}
c.FormatProbe.Diagnostic = "native format readback incompatibility; explicit format omitted, strict local JSON validation remains enabled"
c.FormatProbe.Response = string(data)
c.formatChecked = true
return false, nil
}
var messages []json.RawMessage
if json.Unmarshal(data, &messages) != nil || len(messages) == 0 {
return false, fmt.Errorf("invalid structured-format probe readback")
}
c.formatChecked = true
c.formatSupported = true
c.FormatProbe.Supported = true
return true, nil
}
// ExecutionError exposes a safe classification, not arbitrary provider messages,
// prompts or credentials. Full redacted responses belong in failure evidence.
func ExecutionError(data json.RawMessage) error {
var failure struct {
Name string `json:"name"`
Tag string `json:"_tag"`
Data struct {
StatusCode int `json:"statusCode"`
} `json:"data"`
}
if json.Unmarshal(data, &failure) != nil {
return fmt.Errorf("OpenCode agent reported an execution error")
}
name := failure.Name
if name == "" {
name = failure.Tag
}
if name == "StructuredOutputError" {
return fmt.Errorf("StructuredOutputError: Model did not produce structured output; inspect retained response for tool-call markup or result-envelope mismatch")
}
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
if !safeNames[name] {
return fmt.Errorf("OpenCode agent reported an execution error")
}
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
return fmt.Errorf("OpenCode agent execution error: %s (HTTP %d)", name, failure.Data.StatusCode)
}
return fmt.Errorf("OpenCode agent execution error: %s", name)
}
+92
View File
@@ -0,0 +1,92 @@
package opencode
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestNativeFormatProbeDowngradesOnlyReadbackIncompatibility(t *testing.T) {
for _, status := range []int{200, 400, 401, 500} {
t.Run(http.StatusText(status), func(t *testing.T) {
probes, generations := 0, 0
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
switch r.Method + " " + r.URL.Path {
case "POST /session":
probes++
var body struct{ Permission []map[string]string }
json.NewDecoder(r.Body).Decode(&body)
if len(body.Permission) != 1 || body.Permission[0]["action"] != "deny" {
t.Fatal("probe has tools", body)
}
w.Write([]byte(`{"id":"ses_probe"}`))
case "POST /session/ses_probe/message":
var body struct {
NoReply bool `json:"noReply"`
}
json.NewDecoder(r.Body).Decode(&body)
if !body.NoReply {
t.Fatal("probe requested inference")
}
w.Write([]byte(`{}`))
case "GET /session/ses_probe/message":
w.WriteHeader(status)
if status == 200 {
w.Write([]byte(`[{"info":{"role":"user","format":{"type":"json_schema"}}}]`))
} else {
w.Write([]byte(`{"name":"BadRequest","data":{"message":"Expected OutputFormatJsonSchema at info.format"}}`))
}
case "POST /session/ses_primary/message":
generations++
var body map[string]json.RawMessage
json.NewDecoder(r.Body).Decode(&body)
if (len(body["format"]) > 0) != (status == 200) {
t.Error("incorrect format compatibility mode", string(body["format"]))
}
w.Write([]byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1},"structured":{"outcome":"success","output":{"value":9}}},"parts":[]}`))
default:
t.Error("unexpected operation", r.Method, r.URL.Path)
}
}))
defer srv.Close()
c := &Connection{Backend: "v1", URL: srv.URL, models: map[string]map[string]string{"ses_primary": {"providerID": "test", "modelID": "test"}}, responses: map[string][]byte{}}
body := map[string]any{"prompt": "task", "format": map[string]any{"type": "json_schema", "schema": map[string]any{"type": "object"}, "retryCount": 0}}
data, err := c.Call(context.Background(), "post", "/api/session/ses_primary/generate", body)
if status == 401 || status == 500 {
if err == nil || generations != 0 {
t.Fatal("unrelated failure silently downgraded", err, generations)
}
return
}
if err != nil || !strings.Contains(string(data), "value") || generations != 1 {
t.Fatal(err, string(data), generations)
}
if c.FormatProbe == nil || c.FormatProbe.SessionID != "ses_probe" || c.FormatProbe.Supported != (status == 200) {
t.Fatal("probe evidence lost", c.FormatProbe)
}
if _, err = c.Call(context.Background(), "post", "/api/session/ses_primary/generate", body); err != nil || probes != 1 {
t.Fatal("capability probe was not cached", err, probes)
}
})
}
}
func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T) {
response := []byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1},"error":{"name":"StructuredOutputError","data":{"message":"Model did not produce structured output; secret-token"}}},"parts":[{"type":"text","text":"<|DSML|invoke name=\"StructuredOutput\">not JSON</|DSML|invoke>"}]}`)
_, err := messageText(response)
if err == nil || !strings.Contains(err.Error(), "StructuredOutputError: Model did not produce structured output") || strings.Contains(err.Error(), "secret-token") {
t.Fatal(err)
}
if _, partialErr := messageText([]byte(strings.Replace(string(response), `"completed":1`, `"completed":0`, 1))); partialErr == nil || !strings.Contains(partialErr.Error(), "StructuredOutputError") {
t.Fatal("incomplete timestamp hid provider failure", partialErr)
}
c := &Connection{Backend: "v1", responses: map[string][]byte{"ses_failure": response}}
retained, err := c.Call(context.Background(), "post", "/api/experimental/session/ses_failure/wait", nil)
if err == nil || string(retained) != string(response) {
t.Fatal("wait failure discarded provider response", err, string(retained))
}
}
+22 -2
View File
@@ -52,11 +52,31 @@ func TestStableNativeSessionContractLive(t *testing.T) {
if len(session.Permission) != 3 || session.Permission[2]["permission"] != "bash" {
t.Fatal("scoped native permission contract not retained", string(data))
}
// Validate the real native JSON-schema request without provider inference.
_, err = c.Native(ctx, "post", "/session/"+result.Data.ID+"/message", map[string]any{"noReply": true, "format": map[string]any{"type": "json_schema", "retryCount": 0, "schema": map[string]any{"type": "object", "required": []string{"outcome", "output"}, "additionalProperties": false, "properties": map[string]any{"outcome": map[string]any{"type": "string", "enum": []string{"success", "error"}}, "output": map[string]any{"type": "object"}}}}, "parts": []map[string]string{{"type": "text", "text": "No inference; validate structured format."}}})
// Validate the complete POST -> retained GET contract, not merely POST
// acceptance. The probe itself never runs a model or tools.
format, _ := json.Marshal(map[string]any{"type": "json_schema", "retryCount": 0, "schema": map[string]any{"type": "object", "required": []string{"outcome", "output"}, "additionalProperties": false, "properties": map[string]any{"outcome": map[string]any{"type": "string", "enum": []string{"success", "error"}}, "output": map[string]any{"type": "object", "additionalProperties": true}}}})
supported, err := c.supportsFormat(ctx, format)
if err != nil {
t.Fatal("native format capability probe failed", err)
}
payload := map[string]any{"noReply": true, "parts": []map[string]string{{"type": "text", "text": "No inference; validate retained message readback."}}}
if supported {
payload["format"] = json.RawMessage(format)
}
_, err = c.Native(ctx, "post", "/session/"+result.Data.ID+"/message", payload)
if err != nil {
t.Fatal("native schema request rejected", err)
}
if _, err = c.Native(ctx, "get", "/session/"+result.Data.ID+"/message", nil); err != nil {
t.Fatal("effective format breaks retained-message readback", err)
}
t.Logf("schema mode supported: %v; retained compatibility probe: %s", supported, c.FormatProbe.SessionID)
if !supported {
recovered, exportErr := c.Messages(ctx, c.FormatProbe.SessionID)
if exportErr != nil || !json.Valid(recovered) || c.MessageReadbackDiagnostic == "" {
t.Fatal("cannot recover old format-bearing messages via read-only export", exportErr, string(recovered))
}
}
ref, err := c.Reference()
if err != nil {
t.Fatal(err)
+76
View File
@@ -0,0 +1,76 @@
package opencode
import (
"bytes"
"context"
"encoding/json"
"fmt"
"os/exec"
"strings"
)
type boundedOutput struct {
bytes.Buffer
overflow bool
limit int
}
func (b *boundedOutput) Write(p []byte) (int, error) {
n := len(p)
remaining := b.limit - b.Len()
if n > remaining {
b.overflow = true
p = p[:remaining]
}
_, _ = b.Buffer.Write(p)
return n, nil
}
// Messages preserves old sessions with native OutputFormat readback failures:
// the CLI export reads the local stored data without that HTTP encoder. It does
// not regenerate, repair the database, delete history or change conversations.
func (c *Connection) Messages(ctx context.Context, id string) ([]byte, error) {
if !sessionID.MatchString(id) {
return nil, fmt.Errorf("invalid session ID")
}
c.MessageReadbackDiagnostic = ""
data, err := c.Native(ctx, "get", "/session/"+id+"/message", nil)
if err == nil {
return data, nil
}
if !strings.Contains(err.Error(), "HTTP 400") || !strings.Contains(string(data), "Expected OutputFormat") {
return data, err
}
if ctx.Err() != nil {
return data, ctx.Err()
}
executable := c.Executable
if executable == "" {
executable = Executable()
}
cmd := exec.CommandContext(ctx, executable, "export", id)
cmd.Dir = c.Directory
stdout, stderr := &boundedOutput{limit: MaxResponse}, &boundedOutput{limit: 8 << 10}
cmd.Stdout = stdout
cmd.Stderr = stderr
if exportErr := cmd.Run(); exportErr != nil {
if ctx.Err() != nil {
return data, ctx.Err()
}
return data, fmt.Errorf("%w; read-only native export fallback failed", err)
}
if stdout.overflow {
return data, fmt.Errorf("%w; native export exceeds 1 MiB evidence limit", err)
}
var exported struct {
Info struct {
ID string `json:"id"`
} `json:"info"`
Messages json.RawMessage `json:"messages"`
}
if json.Unmarshal(stdout.Bytes(), &exported) != nil || exported.Info.ID != id || len(exported.Messages) == 0 || exported.Messages[0] != '[' {
return data, fmt.Errorf("%w; native export returned invalid or mismatched session evidence", err)
}
c.MessageReadbackDiagnostic = "Native HTTP message readback rejected stored OutputFormat (HTTP 400); read-only CLI export recovered the original messages without modifying the session."
return exported.Messages, nil
}
+46
View File
@@ -0,0 +1,46 @@
package opencode
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
func TestOldFormatReadbackRecoveredWithoutRewritingSession(t *testing.T) {
for _, wrongID := range []bool{false, true} {
bin := filepath.Join(t.TempDir(), "opencode")
id := "ses_old"
if wrongID {
id = "ses_unrelated"
}
export := `{"info":{"id":"` + id + `"},"messages":[{"info":{"role":"assistant","error":{"name":"StructuredOutputError"}},"parts":[{"type":"text","text":"original DSML evidence"}]}]}`
if err := os.WriteFile(bin, []byte("#!/bin/sh\n[ \"$1\" = export ] || exit 8\n[ \"$2\" = ses_old ] || exit 9\necho '"+export+"'\n"), 0700); err != nil {
t.Fatal(err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
t.Error("evidence reader changed stored session", r.Method)
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(400)
w.Write([]byte(`{"name":"BadRequest","data":{"message":"Expected OutputFormatJsonSchema at info.format"}}`))
}))
c := &Connection{Backend: "v1", URL: srv.URL, Executable: bin}
data, err := c.Messages(context.Background(), "ses_old")
srv.Close()
if wrongID {
if err == nil {
t.Fatal("unrelated session export accepted")
}
continue
}
if err != nil || !json.Valid(data) || !strings.Contains(string(data), "original DSML evidence") || c.MessageReadbackDiagnostic == "" {
t.Fatal(err, string(data))
}
}
}
+23 -7
View File
@@ -36,11 +36,14 @@ func messageText(data []byte) (string, error) {
} `json:"info"`
Parts []struct{ Type, Text string } `json:"parts"`
}
if json.Unmarshal(data, &message) != nil || message.Info.Role != "assistant" || message.Info.Time.Completed == 0 {
if json.Unmarshal(data, &message) != nil || message.Info.Role != "assistant" {
return "", fmt.Errorf("OpenCode returned no completed assistant result")
}
if len(message.Info.Error) > 0 && string(message.Info.Error) != "null" {
return "", fmt.Errorf("OpenCode agent reported an execution error")
return "", ExecutionError(message.Info.Error)
}
if message.Info.Time.Completed == 0 {
return "", fmt.Errorf("OpenCode returned no completed assistant result")
}
if len(message.Info.Structured) > 0 && string(message.Info.Structured) != "null" && (message.Info.Finish == "stop" || message.Info.Finish == "tool-calls") {
return string(message.Info.Structured), nil
@@ -159,7 +162,17 @@ func (c *Connection) Call(ctx context.Context, method, path string, body any) ([
}
payload := map[string]any{"model": c.models[id], "parts": []map[string]string{{"type": "text", "text": text}}}
if len(request.Format) > 0 {
payload["format"] = request.Format
supported, err := c.supportsFormat(ctx, request.Format)
if err != nil {
var evidence []byte
if c.FormatProbe != nil {
evidence = []byte(c.FormatProbe.Response)
}
return evidence, fmt.Errorf("native structured-format compatibility check: %w", err)
}
if supported {
payload["format"] = request.Format
}
}
response, err := c.Native(ctx, "post", "/session/"+id+"/message", payload)
if err != nil {
@@ -176,11 +189,14 @@ func (c *Connection) Call(ctx context.Context, method, path string, body any) ([
return json.Marshal(map[string]any{"data": map[string]string{"text": result}})
case "wait":
_, err := messageText(c.responses[id])
return nil, err
case "message":
data, err := c.Native(ctx, "get", "/session/"+id+"/message", nil)
if err != nil {
return nil, err
return c.responses[id], err
}
return nil, nil
case "message":
data, err := c.Messages(ctx, id)
if err != nil {
return data, err
}
var messages []struct {
Info json.RawMessage `json:"info"`
+20 -1
View File
@@ -18,11 +18,18 @@ func TestNativeSessionModelPermissionsGenerationAndAbort(t *testing.T) {
}
switch r.Method + " " + r.URL.Path {
case "POST /session":
var raw map[string]json.RawMessage
json.NewDecoder(r.Body).Decode(&raw)
if strings.Contains(string(raw["title"]), "compatibility probe") {
w.Write([]byte(`{"id":"ses_probe"}`))
return
}
var body struct {
Model map[string]string
Permission []map[string]string
}
json.NewDecoder(r.Body).Decode(&body)
data, _ := json.Marshal(raw)
json.Unmarshal(data, &body)
if body.Model["id"] != "model" || body.Model["providerID"] != "provider" || body.Permission[0]["action"] != "deny" || body.Permission[1]["permission"] != "bash" {
t.Error("model or scoped permissions lost", body)
}
@@ -43,6 +50,18 @@ func TestNativeSessionModelPermissionsGenerationAndAbort(t *testing.T) {
}
prompted = true
w.Write([]byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1}},"parts":[{"type":"text","text":"result"}]}`))
case "POST /session/ses_probe/message":
var body struct {
NoReply bool `json:"noReply"`
Format map[string]any
}
json.NewDecoder(r.Body).Decode(&body)
if !body.NoReply || body.Format["retryCount"] != float64(0) {
t.Error("probe requested inference or retries", body)
}
w.Write([]byte(`{}`))
case "GET /session/ses_probe/message":
w.Write([]byte(`[{"info":{"role":"user"}}]`))
case "GET /session/ses_native/message":
w.Write([]byte(`[{"info":{"role":"assistant","finish":"stop","time":{"completed":1}},"parts":[{"type":"text","text":"result"}]}]`))
case "POST /session/ses_native/abort":
+96
View File
@@ -0,0 +1,96 @@
package server
import (
"context"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/pipeline"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// Explicitly opt in: this uses existing native-provider credentials, makes ten
// small model calls, and retains all stage/probe sessions. No tools are enabled.
func TestNativeMathDemoLive(t *testing.T) {
executable := os.Getenv("PIPELINES_TEST_OPENCODE_V1")
if executable == "" || os.Getenv("PIPELINES_TEST_NATIVE_INFERENCE") != "1" {
t.Skip("set PIPELINES_TEST_OPENCODE_V1 and PIPELINES_TEST_NATIVE_INFERENCE=1 for real native math inference")
}
t.Setenv("PIPELINES_OPENCODE", executable)
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
defer cancel()
directory := t.TempDir()
store, err := pipeline.Open(filepath.Join(directory, "math.db"))
if err != nil {
t.Fatal(err)
}
defer store.Close()
var source strings.Builder
add := func(key, output, prompt, edges string) {
fmt.Fprintf(&source, "# stage %s\n\n```output\n%s\n```\n\n```prompt\n%s\n```\n\n```tools\n```\n\n```configs\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=0\n```\n\n```edges\n%s\nerror -> #failure\n```\n\n", key, output, prompt, edges)
}
contract := "Return only the small output map {a,b,value,steps}; value is an integer, steps an array of equation strings."
add("sum", contract, "Parse input.prompt as a + b. Compute value=a+b and steps=[\"a + b = value\"] with actual numbers. Preserve a and b.", "success -> multiply")
add("multiply", contract, "Preserve a,b and steps. Set value=input.value*3. Append the decimal input.value, followed by ' * 3 = ', followed by the decimal new value, as one equation string. For example, input.value=9 yields '9 * 3 = 27'. Never use placeholder words in equations.", "success -> parity")
add("parity", contract, "Do not change any output fields. Select outcome even when input.value is even, odd when it is odd.", "even -> halve\nodd -> add_one")
add("halve", contract, "Preserve a,b and steps. Set value=input.value/2. Append the decimal input.value, ' / 2 = ', the decimal new value, and ' (even branch)' as one equation string. For example, input.value=30 yields '30 / 2 = 15 (even branch)'. Never use placeholder words in equations.", "success -> report")
add("add_one", contract, "Preserve a,b and steps. Set value=input.value+1. Append the decimal input.value, ' + 1 = ', the decimal new value, and ' (odd branch)' as one equation string. For example, input.value=27 yields '27 + 1 = 28 (odd branch)'. Never use placeholder words in equations.", "success -> report")
add("report", "Return only {a,b,value,steps,summary}.", "Preserve a,b,value,steps. Set summary to the exact equation strings in input.steps joined by \"; \".", "success -> #success")
if _, err = store.Submit(ctx, "native math regression", source.String()); err != nil {
t.Fatal(err)
}
items, err := store.List(ctx)
if err != nil {
t.Fatal(err)
}
r, err := newRunner(ctx, store)
if err != nil {
t.Fatal(err)
}
defer func() { cancel(); r.wg.Wait() }()
for _, test := range []struct {
input, value, summary string
path []string
}{
{"7 + 2", "28", "7 + 2 = 9; 9 * 3 = 27; 27 + 1 = 28 (odd branch)", []string{"sum", "multiply", "parity", "add_one", "report"}},
{"7 + 3", "15", "7 + 3 = 10; 10 * 3 = 30; 30 / 2 = 15 (even branch)", []string{"sum", "multiply", "parity", "halve", "report"}},
} {
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "native-math-" + test.value, Workspace: directory, Input: map[string]any{"prompt": test.input}})
if err != nil {
t.Fatal(err)
}
r.wg.Wait()
run, err = store.GetRun(ctx, run.ID)
if err != nil {
t.Fatal(err)
}
if run.Status != "succeeded" || fmt.Sprint(run.Output["value"]) != test.value || run.Output["summary"] != test.summary {
t.Fatalf("native math failed: input=%s status=%s diagnostic=%s output=%v", test.input, run.Status, run.Diagnostic, run.Output)
}
var path []string
for _, step := range run.Steps {
path = append(path, step.Stage)
if step.SessionID == "" || step.AgentActive {
t.Fatal("session not retained/stopped", step)
}
}
if !reflect.DeepEqual(path, test.path) {
t.Fatal("unexpected stage replay/path", path, test.path)
}
evidence, err := r.evidence(ctx, run.ID)
if err != nil || len(evidence) == 0 {
t.Fatal("live session evidence unavailable", err)
}
for _, item := range evidence {
if item.Diagnostic != "" {
t.Fatal("native session readback failed", item.Diagnostic)
}
}
t.Logf("PASS input=%s result=%s path=%v summary=%s; native session readback passed", test.input, test.value, path, test.summary)
}
}
+4 -3
View File
@@ -3,6 +3,7 @@ package stage
import (
"context"
"encoding/json"
"errors"
"fmt"
"git.a-wan.net/alan/pipelines/internal/opencode"
"strings"
@@ -87,13 +88,13 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
defer cancel()
if err != nil {
if stopErr := c.Interrupt(cleanup, c.SessionID); stopErr != nil {
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection", c.SessionID)
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection: %w", c.SessionID, errors.Join(err, stopErr))
return
}
}
if c.OnSession != nil {
if persistErr := c.OnSession(c.SessionID, false); persistErr != nil {
err = persistErr
err = errors.Join(err, persistErr)
text = ""
return
}
@@ -148,7 +149,7 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
}
for _, m := range messages.Data {
if len(m.Error) > 0 && string(m.Error) != "null" {
return "", fmt.Errorf("agent reported an execution error")
return "", opencode.ExecutionError(m.Error)
}
if m.Finish == "stop" || (m.Finish == "tool-calls" && m.Completed && len(m.Structured) > 0 && string(m.Structured) != "null") {
if len(m.Structured) > 0 && string(m.Structured) != "null" {
+40
View File
@@ -0,0 +1,40 @@
package stage
import (
"context"
"errors"
"strings"
"testing"
)
func TestAgentCleanupFailurePreservesOriginalFailureAndActiveOwnership(t *testing.T) {
original := errors.New("StructuredOutputError: Model did not produce structured output")
cleanup := errors.New("interrupt failed")
prompts, releases := 0, 0
c := &AgentClient{OpenCodeClient: OpenCodeClient{Directory: t.TempDir()}, OnSession: func(_ string, active bool) error {
if !active {
releases++
}
return nil
}}
c.command = func(_ context.Context, _ string, args ...string) ([]byte, error) {
switch args[2] {
case "/api/session":
return []byte(`{"data":{"id":"ses_cleanup"}}`), nil
case "/api/session/ses_cleanup/prompt":
prompts++
return []byte(`{}`), nil
case "/api/experimental/session/ses_cleanup/wait":
return nil, original
case "/api/session/ses_cleanup/interrupt":
return nil, cleanup
default:
t.Errorf("unexpected operation %v", args)
return nil, errors.New("unexpected operation")
}
}
_, err := c.Generate(context.Background(), ModelConfig{Provider: "fixture", Model: "fixture"}, "task")
if !errors.Is(err, original) || !errors.Is(err, cleanup) || !strings.Contains(err.Error(), "ses_cleanup") || prompts != 1 || releases != 0 {
t.Fatal("cleanup lost original diagnostic, replayed work or released active ownership", err, prompts, releases)
}
}
+44 -16
View File
@@ -2,6 +2,7 @@ package stage
import (
"encoding/json"
"git.a-wan.net/alan/pipelines/internal/opencode"
"regexp"
"strings"
"time"
@@ -10,24 +11,28 @@ import (
// Failure evidence is bounded and credential-redacted; request bodies (which
// contain prompts) and the subprocess environment are never recorded here.
type OperationEvidence struct {
Operation string `json:"operation"`
Backend string `json:"backend"`
Executable string `json:"executable"`
Directory string `json:"directory"`
Provider string `json:"provider"`
Model string `json:"model"`
StartedAt string `json:"started_at"`
DurationMS int64 `json:"duration_ms"`
ExitCode int `json:"exit_code"`
HTTPStatus int `json:"http_status,omitempty"`
Diagnostic string `json:"diagnostic"`
Stdout string `json:"stdout,omitempty"`
Stderr string `json:"stderr,omitempty"`
Operation string `json:"operation"`
Backend string `json:"backend"`
Executable string `json:"executable"`
Directory string `json:"directory"`
Provider string `json:"provider"`
Model string `json:"model"`
StartedAt string `json:"started_at"`
DurationMS int64 `json:"duration_ms"`
ExitCode int `json:"exit_code"`
HTTPStatus int `json:"http_status,omitempty"`
Diagnostic string `json:"diagnostic"`
Stdout string `json:"stdout,omitempty"`
Stderr string `json:"stderr,omitempty"`
Transport string `json:"transport,omitempty"`
NativeOperation string `json:"native_operation,omitempty"`
ExitKind string `json:"exit_kind,omitempty"`
}
type OperationLog struct {
Provider, Model string
Failures []OperationEvidence
FormatChecks []opencode.FormatProbe
}
var sensitiveKey = regexp.MustCompile(`(?i)^(authorization|password|secret|api[-_]?key|access[-_]?token|refresh[-_]?token|token|cookie|set-cookie|client[-_]?secret)$`)
@@ -37,7 +42,9 @@ var credentialURL = regexp.MustCompile(`(https?://)[^/\s:@]+:[^/\s@]+@`)
func RedactEvidence(text string) string {
var data any
if json.Unmarshal([]byte(text), &data) == nil {
decoder := json.NewDecoder(strings.NewReader(text))
decoder.UseNumber()
if json.Valid([]byte(text)) && decoder.Decode(&data) == nil {
var clean func(any)
clean = func(v any) {
switch x := v.(type) {
@@ -91,6 +98,13 @@ func (l *OperationLog) record(e OperationEvidence, started time.Time) {
return
}
e.Provider, e.Model = l.Provider, l.Model
if e.Backend == "v1" {
e.Transport = "native_http"
e.ExitKind = "not a subprocess exit"
} else {
e.Transport = "cli_subprocess"
e.ExitKind = "subprocess exit code"
}
e.StartedAt = started.UTC().Format(time.RFC3339Nano)
e.DurationMS = time.Since(started).Milliseconds()
e.Diagnostic = RedactEvidence(e.Diagnostic)
@@ -102,7 +116,7 @@ func (l *OperationLog) record(e OperationEvidence, started time.Time) {
}
func (l *OperationLog) Transcript(original string) string {
if l == nil || len(l.Failures) == 0 {
if l == nil || (len(l.Failures) == 0 && len(l.FormatChecks) == 0) {
return original
}
var data map[string]any
@@ -112,7 +126,21 @@ func (l *OperationLog) Transcript(original string) string {
data["original_transcript"] = original
}
}
data["opencode_failures"] = l.Failures
if len(l.Failures) > 0 {
data["opencode_failures"] = l.Failures
}
if len(l.FormatChecks) > 0 {
data["native_format_checks"] = l.FormatChecks
}
encoded, _ := json.Marshal(data)
return string(encoded)
}
func (l *OperationLog) recordFormatProbe(probe opencode.FormatProbe) {
if l == nil || len(l.FormatChecks) >= 4 {
return
}
probe.Response = RedactEvidence(probe.Response)
probe.Diagnostic = RedactEvidence(probe.Diagnostic)
l.FormatChecks = append(l.FormatChecks, probe)
}
+13
View File
@@ -3,6 +3,7 @@ package stage
import (
"context"
"encoding/json"
"git.a-wan.net/alan/pipelines/internal/opencode"
"os"
"path/filepath"
"strings"
@@ -40,6 +41,9 @@ exit 7
}
func TestEvidenceRedactionAndLimit(t *testing.T) {
if output := RedactEvidence(`{"value":9007199254740993,"token":"private-value"}`); !strings.Contains(output, "9007199254740993") || strings.Contains(output, "private-value") {
t.Fatal("redaction changed evidence numbers or exposed token", output)
}
for _, input := range []string{`{"nested":{"access_token":"private-value","password":"private-value"}}`, "Authorization: Basic private-value\n", "https://user:private-value@example.test/path?token=private-value", "api_key=private-value"} {
if output := RedactEvidence(input); strings.Contains(output, "private-value") {
t.Fatal(output)
@@ -49,3 +53,12 @@ func TestEvidenceRedactionAndLimit(t *testing.T) {
t.Fatal(len(output))
}
}
func TestSuccessfulFormatProbeIsRetainedWithoutBeingCalledAFailure(t *testing.T) {
log := &OperationLog{}
log.recordFormatProbe(opencode.FormatProbe{SessionID: "ses_probe", Supported: true})
transcript := log.Transcript("")
if !strings.Contains(transcript, "native_format_checks") || !strings.Contains(transcript, "ses_probe") || strings.Contains(transcript, "opencode_failures") {
t.Fatal(transcript)
}
}
+4 -2
View File
@@ -22,10 +22,12 @@ func (c OpenCodeClient) SessionEvidence(ctx context.Context, id string) (string,
c.Backend = connection.Backend
var session, messages, log []byte
logDiagnostic := ""
readbackDiagnostic := ""
if c.Backend == "v1" {
session, err = connection.Native(ctx, "get", "/session/"+id, nil)
if err == nil {
messages, err = connection.Native(ctx, "get", "/session/"+id+"/message", nil)
messages, err = connection.Messages(ctx, id)
readbackDiagnostic = connection.MessageReadbackDiagnostic
}
} else {
session, err = c.call(ctx, "get", "/api/session/"+id, nil)
@@ -40,6 +42,6 @@ func (c OpenCodeClient) SessionEvidence(ctx context.Context, id string) (string,
}
}
}
data, _ := json.Marshal(map[string]string{"session": RedactEvidence(string(session)), "messages": RedactEvidence(string(messages)), "log": RedactEvidence(string(log)), "log_diagnostic": logDiagnostic})
data, _ := json.Marshal(map[string]string{"session": RedactEvidence(string(session)), "messages": RedactEvidence(string(messages)), "log": RedactEvidence(string(log)), "log_diagnostic": logDiagnostic, "readback_diagnostic": readbackDiagnostic})
return string(data), err
}
+55
View File
@@ -0,0 +1,55 @@
package stage
import (
"context"
"encoding/json"
"strings"
"testing"
)
func TestToolMarkupAndFlattenedResultsAreRejectedAndRetained(t *testing.T) {
for _, text := range []string{
`<|DSML|invoke name="StructuredOutput">{"outcome":"success","output":{"value":9}}</|DSML|invoke>`,
`<tool_call>{"outcome":"success","output":{"value":9}}</tool_call>`,
`{"outcome":"success","a":7,"b":2,"value":9,"steps":["7 + 2 = 9"]}`,
} {
f := &repairFixture{response: text}
_, err := (ModelStage{Client: f}).Run(context.Background(), modelDefinition(), nil)
if err == nil || f.generateCalls != 1 || f.repairCalls != 0 {
t.Fatal("markup/envelope error accepted or replayed", text, err, f)
}
if len(f.audits) != 1 || f.audits[0].Original != text || f.audits[0].Accepted {
t.Fatal("invalid response evidence discarded", f.audits)
}
}
valid := `{"outcome":"success","output":{"evidence":"<tool_call> is a quoted artifact"}}`
if _, err := decodeModelResult(valid, modelDefinition().Edges); err != nil {
t.Fatal("valid quoted evidence rejected", err)
}
f := &repairFixture{response: strings.TrimSuffix(valid, "}")}
if result, err := (ModelStage{Client: f}).Run(context.Background(), modelDefinition(), nil); err != nil || result.Output["evidence"] != "<tool_call> is a quoted artifact" {
t.Fatal("quoted markup blocked safe syntax repair", result, err)
}
}
func TestRuntimeEnvelopeOverridesInnerOutputContract(t *testing.T) {
def := modelDefinition()
def.Blocks["output"] = "Return only {a,b,value,steps}."
spec, err := (ModelStage{}).Spec(def)
if err != nil {
t.Fatal(err)
}
prompt, err := modelPrompt(def, spec, Values{"prompt": "7 + 2"})
if err != nil {
t.Fatal(err)
}
for _, required := range []string{"output_contract describes ONLY", "Never flatten", "StructuredOutput", "Do not print tool-call markup"} {
if !strings.Contains(prompt, required) {
t.Fatal("missing envelope rule", required)
}
}
result, err := decodeModelResult(`{"outcome":"success","output":{"a":7,"b":2,"value":9,"steps":["7 + 2 = 9"]}}`, def.Edges)
if err != nil || result.Output["value"] != json.Number("9") {
t.Fatal(result, err)
}
}
+46
View File
@@ -49,6 +49,12 @@ later stage, input is exactly the output map from the preceding executed stage.
Use that runtime data with the stage instructions; no input contract is declared.
Select exactly one of allowed_outcomes. Return exactly one JSON object with these
two keys: {"outcome":"declared outcome", "output":{...runtime output fields...}}.
output_contract describes ONLY the fields inside output, not the outer result.
If stage instructions ask for a small output map, put that map inside output;
they do not replace this runtime envelope. Never flatten stage fields alongside
outcome. If StructuredOutput is offered, its arguments must be the complete
{"outcome":"declared outcome","output":{...}} object, not individual output fields.
Do not print tool-call markup (including DSML) as text or simulate a tool call.
The output must be an object, not null. Do not include Markdown, commentary, or
destination stage names. Preserve handoff context when the stage instructions require it.
This call has no tools. Do not invent file edits, command results, screenshots, or
@@ -63,6 +69,9 @@ Stage definition and runtime input:
// Never pick a winner among multiple objects or salvage nested JSON from a
// malformed outer object. The extracted object still gets strict validation.
func stageResultText(text string) string {
if !json.Valid([]byte(strings.TrimSpace(text))) && toolMarkup(text) {
return text
}
remaining := strings.TrimSpace(text)
// A single surrounding fence is presentation, even if its JSON is incomplete.
for _, prefix := range []string{"```json\n", "```\n"} {
@@ -103,7 +112,44 @@ func stageResultText(text string) string {
return text
}
func toolMarkup(text string) bool {
depth, inString, escape := 0, false, false
for i := 0; i < len(text); i++ {
if inString {
if escape {
escape = false
} else if text[i] == '\\' {
escape = true
} else if text[i] == '"' {
inString = false
}
continue
}
if depth > 0 && text[i] == '"' {
inString = true
continue
}
for _, marker := range []string{"<|DSML|", "<|DSML|", "<tool_call>", "<invoke name=", "<function_calls>"} {
if strings.HasPrefix(text[i:], marker) {
return true
}
}
switch text[i] {
case '{', '[':
depth++
case '}', ']':
if depth > 0 {
depth--
}
}
}
return false
}
func decodeModelResult(text string, edges map[string]string) (Result, error) {
if !json.Valid([]byte(text)) && toolMarkup(text) {
return Result{}, fmt.Errorf("%w: provider returned tool-call markup as text; expected the outcome/output JSON envelope", ErrInvalidModelResult)
}
// Preflight syntax to retain the real parser diagnostic and byte position,
// rather than replacing truncation with a generic "invalid result fields".
var syntax json.RawMessage
+13 -2
View File
@@ -63,16 +63,27 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
if c.connection != nil && c.connection.Backend == "v1" {
started := time.Now()
data, err := c.connection.Call(ctx, method, path, body)
if probe := c.connection.FormatProbe; probe != nil {
c.Operations.recordFormatProbe(*probe)
if !probe.Supported && probe.Diagnostic != "" {
c.Operations.record(OperationEvidence{Operation: "get /session/" + probe.SessionID + "/message (format compatibility probe)", Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: 400, Diagnostic: probe.Diagnostic, Stdout: probe.Response}, started)
}
c.connection.FormatProbe = nil
}
if err != nil {
status := 0
if match := httpStatusPattern.FindStringSubmatch(err.Error()); len(match) > 0 {
status, _ = strconv.Atoi(match[1])
}
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: status, Diagnostic: err.Error(), Stdout: string(data)}, started)
evidenceStatus := status
if evidenceStatus == 0 {
evidenceStatus = c.connection.LastHTTPStatus
}
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
if ctx.Err() != nil {
return nil, ctx.Err()
}
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err}
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
}
return data, nil
}
+12 -5
View File
@@ -80,20 +80,27 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC
return Result{}, err
}
original := stageResultText(text)
record := func(candidate, method string, accepted bool) {
if recorder, ok := client.(repairRecorder); ok {
bounded := original
if len(bounded) > 32<<10 {
bounded = bounded[:32<<10] + "\n[truncated]"
}
recorder.RecordFormatRepair(FormatRepair{Original: bounded, Candidate: candidate, Diagnostic: parseError.Error(), Accepted: accepted, Method: method})
}
}
if len(original) > 32<<10 {
record("", "validation", false)
return Result{}, fmt.Errorf("%w; format repair skipped: response exceeds 32 KiB repair bound", parseError)
}
if json.Valid([]byte(original)) {
record("", "validation", false)
return Result{}, parseError
}
if tokens, ok := jsonLexemes(original); !ok || len(tokens) == 0 || tokens[0] != "{" {
record("", "validation", false)
return Result{}, fmt.Errorf("%w; unsafe to repair incomplete values or a non-object response", parseError)
}
record := func(candidate, method string, accepted bool) {
if recorder, ok := client.(repairRecorder); ok {
recorder.RecordFormatRepair(FormatRepair{Original: original, Candidate: candidate, Diagnostic: parseError.Error(), Accepted: accepted, Method: method})
}
}
// The most common truncation has one unique repair: close the existing
// containers. This cannot change a verdict or any completed evidence value.
if candidate, ok := closeJSONContainers(original); ok {