Handle native structured-format incompatibility and preserve failure evidence
This commit is contained in:
@@ -79,6 +79,19 @@ the step transcript. Credentials are redacted, but project/provider messages may
|
||||
still contain private context: review before sharing. Older failures cannot gain
|
||||
discarded subprocess output retroactively; Debug attempts retained session reads.
|
||||
|
||||
Native OpenCode schema support is checked with a retained, deny-all, no-inference
|
||||
POST/readback probe. OpenCode 1.18.35 accepts explicit formats but rejects their
|
||||
stored messages on readback; affected backends omit `format` and require the same
|
||||
strict local `{outcome, output}` validation. Compatible backends retain schema
|
||||
mode with no automatic tool-replaying schema retries. The compatibility decision
|
||||
and probe session are recorded, without rewriting published definitions/history.
|
||||
For old format-bearing sessions whose HTTP readback fails, Debug uses read-only
|
||||
`opencode export` with an exact session-ID check to recover original messages.
|
||||
`StructuredOutputError` is surfaced directly; DSML/tool-call markup and flattened
|
||||
results are rejected and retained, never silently interpreted as successful tool
|
||||
execution or wrapped into a guessed result. Native HTTP evidence distinguishes
|
||||
adapter/HTTP failures from actual subprocess exit codes.
|
||||
|
||||
To validate a model on the execution computer, run:
|
||||
|
||||
```sh
|
||||
|
||||
+24
-4
@@ -5,13 +5,24 @@ Every model result must be one object containing exactly `outcome` and `output`.
|
||||
(including nested output keys), missing fields, unknown fields and invalid
|
||||
outcomes are rejected. Free-form `output` contract prose is not a JSON Schema.
|
||||
|
||||
The native stable OpenCode adapter requests `format.type=json_schema` with this
|
||||
envelope schema and `retryCount=0`, reads the returned structured result when
|
||||
available, and preserves the original tool transcript. Local validation remains
|
||||
mandatory. V2's published prompt/generate API does not accept a schema-output
|
||||
The native stable OpenCode adapter first checks the complete format POST/readback
|
||||
contract in a separate retained, deny-all, no-inference session. Compatible native
|
||||
backends request `format.type=json_schema` with this envelope and `retryCount=0`.
|
||||
OpenCode 1.18.35 accepts a format but rejects its stored messages with
|
||||
`Expected OutputFormatJsonSchema` (even explicit `text` fails). For that specific
|
||||
incompatibility, requests omit `format`; strict local validation remains mandatory.
|
||||
Other probe failures, such as authentication or network errors, are not silently
|
||||
downgraded. Returned structured results remain preferred when available, and tool
|
||||
transcripts are preserved. V2's published prompt/generate API does not accept a schema-output
|
||||
field; that adapter retains local validation instead of sending unsupported
|
||||
fields or claiming provider-side schema enforcement.
|
||||
|
||||
Output-contract prose describes the map **inside** `output`; it never overrides
|
||||
the runtime `{outcome, output}` envelope. Printed DSML or other tool-call markup
|
||||
is not evidence of a real tool invocation and cannot be accepted as a result.
|
||||
Flattened output fields are rejected, not guessed into a new envelope. Invalid
|
||||
responses are retained even when they are not eligible for syntax repair.
|
||||
|
||||
## Format recovery is not execution retry
|
||||
|
||||
Parser errors retain their diagnostic and byte offset: for example, `missing
|
||||
@@ -46,6 +57,15 @@ and a recognized safe error category (such as session not found or service
|
||||
unavailable). Arbitrary stderr, provider messages and response bodies are not
|
||||
copied into the diagnostic. A `/wait` command failure is not evidence that the
|
||||
agent prompt failed to start, and does not trigger replay of stage work.
|
||||
Native `StructuredOutputError` is named directly, rather than reduced to a generic
|
||||
command failure. Native HTTP evidence records the actual HTTP operation/status;
|
||||
an `exit_code` of -1 there is explicitly marked as not a subprocess exit.
|
||||
Cleanup failures preserve the original generation error and active ownership.
|
||||
|
||||
For existing native sessions with format-bearing messages, the HTTP readback
|
||||
workaround is read-only CLI `opencode export`. The exported session ID must match
|
||||
the requested ID. Nothing is rewritten or removed from the database/history.
|
||||
Debug receives the original messages plus a note identifying that recovery path.
|
||||
|
||||
## Optional machine-readable restrictions
|
||||
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
# Native structured-output regression (2026-10-09)
|
||||
|
||||
The supplied `~/findings.md` describes Math Demo run #12 on another computer,
|
||||
using OpenCode 1.18.35 and `opencode-go/deepseek-v4.1-flash`. Its recorded response
|
||||
contains `StructuredOutputError` and DSML as text, with flattened output fields.
|
||||
The supplied evidence does not identify the component responsible for converting
|
||||
that provider tool call into text. That upstream cause remains unconfirmed.
|
||||
|
||||
## Confirmed independently
|
||||
|
||||
On a fresh `ghcr.io/anomalyco/opencode:1.18.35` container, without credentials or
|
||||
inference:
|
||||
|
||||
- `POST /session/ID/message` with `noReply:true` and an explicit JSON-schema
|
||||
format succeeds, but subsequent HTTP message readback fails with HTTP 400.
|
||||
- Omitting `retryCount` does not fix it. Explicit `format:{type:"text"}` also fails.
|
||||
- Omitting `format` makes both POST and message readback succeed.
|
||||
- `opencode export ID` recovers the original format-bearing messages which the
|
||||
HTTP readback cannot encode. No stored message needs to be modified.
|
||||
|
||||
The adapter now probes this contract without inference. Affected native backends
|
||||
omit the incompatible field and enforce the outcome/output envelope locally.
|
||||
Compatible native schema mode is retained and covered by the mock HTTP contract
|
||||
tests; a real schema-tool invocation on a compatible native release is **not tested**.
|
||||
No automatic stage/tool replay or DSML-to-result interpretation was added.
|
||||
|
||||
## Real provider regression
|
||||
|
||||
The opt-in server test uses an equivalent math graph, not a copy of the remote
|
||||
immutable definition (its complete source was not supplied). It deliberately
|
||||
uses small-map output-contract prose to exercise the envelope clarification.
|
||||
Using the existing local native OpenCode 1.18.35/OpenCode Go credentials, it passed:
|
||||
|
||||
- `7 + 2`: sum → multiply → parity → add_one → report; result **28**;
|
||||
`7 + 2 = 9; 9 * 3 = 27; 27 + 1 = 28 (odd branch)`.
|
||||
- `7 + 3`: sum → multiply → parity → halve → report; result **15**;
|
||||
`7 + 3 = 10; 10 * 3 = 30; 30 / 2 = 15 (even branch)`.
|
||||
- Native retained-session readback through server evidence retrieval passed.
|
||||
|
||||
The first inference check reached result 28 but caught an ambiguous *test prompt*
|
||||
that yielded the literal word `previous` in an equation. The fixture now explicitly
|
||||
specifies numeric substitutions; the full odd/even check then passed. Published
|
||||
pipeline definitions were not rewritten.
|
||||
|
||||
Commands (supply a native CLI path on your computer):
|
||||
|
||||
```sh
|
||||
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode-1.18.35 \
|
||||
go test -count=1 -run TestStableNativeSessionContractLive -v ./internal/opencode
|
||||
|
||||
# Explicit opt-in: existing credentials, ten small paid-provider calls,
|
||||
# retained stage/probe sessions, and no project tools.
|
||||
PIPELINES_TEST_OPENCODE_V1=/path/to/opencode-1.18.35 \
|
||||
PIPELINES_TEST_NATIVE_INFERENCE=1 \
|
||||
go test -count=1 -run TestNativeMathDemoLive -v ./internal/server
|
||||
```
|
||||
|
||||
The other computer's original run/session was not accessed or modified here.
|
||||
@@ -32,6 +32,12 @@ type Connection struct {
|
||||
done chan error
|
||||
models map[string]map[string]string
|
||||
responses map[string][]byte
|
||||
formatChecked bool
|
||||
formatSupported bool
|
||||
FormatProbe *FormatProbe
|
||||
LastHTTPStatus int
|
||||
LastHTTPOperation string
|
||||
MessageReadbackDiagnostic string
|
||||
}
|
||||
|
||||
// Reference records only our ephemeral loopback service credentials, never provider auth.
|
||||
@@ -210,6 +216,8 @@ func (c *Connection) Close() {
|
||||
}
|
||||
|
||||
func (c *Connection) Native(ctx context.Context, method, path string, body any) ([]byte, error) {
|
||||
c.LastHTTPStatus = 0
|
||||
c.LastHTTPOperation = strings.ToUpper(method) + " " + path
|
||||
u, err := url.Parse(c.URL + path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -241,6 +249,7 @@ func (c *Connection) Native(ctx context.Context, method, path string, body any)
|
||||
return nil, fmt.Errorf("OpenCode native API request failed")
|
||||
}
|
||||
defer response.Body.Close()
|
||||
c.LastHTTPStatus = response.StatusCode
|
||||
if response.StatusCode < 200 || response.StatusCode >= 300 {
|
||||
// The stage adapter stores bounded, redacted failure evidence separately;
|
||||
// the public error string remains free of provider bodies and credentials.
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
package opencode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// FormatProbe is retained, deny-all and no-inference. Some native releases
|
||||
// accept a format on POST but reject their own stored user message on GET.
|
||||
type FormatProbe struct {
|
||||
SessionID string `json:"session_id"`
|
||||
Supported bool `json:"supported"`
|
||||
Diagnostic string `json:"diagnostic,omitempty"`
|
||||
Response string `json:"response,omitempty"`
|
||||
}
|
||||
|
||||
func (c *Connection) supportsFormat(ctx context.Context, format json.RawMessage) (bool, error) {
|
||||
if c.formatChecked {
|
||||
return c.formatSupported, nil
|
||||
}
|
||||
created, err := c.Native(ctx, "post", "/session", map[string]any{"title": "Pipelines structured-format compatibility probe (no inference)", "permission": []map[string]string{{"permission": "*", "pattern": "*", "action": "deny"}}})
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
var session struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if json.Unmarshal(created, &session) != nil || !sessionID.MatchString(session.ID) {
|
||||
return false, fmt.Errorf("invalid structured-format probe session")
|
||||
}
|
||||
c.FormatProbe = &FormatProbe{SessionID: session.ID}
|
||||
posted, err := c.Native(ctx, "post", "/session/"+session.ID+"/message", map[string]any{"noReply": true, "format": format, "parts": []map[string]string{{"type": "text", "text": "Compatibility probe only. No reply, inference or tool execution requested."}}})
|
||||
if err != nil {
|
||||
c.FormatProbe.Response = string(posted)
|
||||
return false, err
|
||||
}
|
||||
data, err := c.Native(ctx, "get", "/session/"+session.ID+"/message", nil)
|
||||
if err != nil {
|
||||
// Downgrade only this specific readback incompatibility, never an auth,
|
||||
// network or unrelated HTTP failure. Do not alter the stored probe.
|
||||
c.FormatProbe.Response = string(data)
|
||||
if !strings.Contains(err.Error(), "HTTP 400") || !strings.Contains(string(data), "Expected OutputFormat") {
|
||||
return false, err
|
||||
}
|
||||
c.FormatProbe.Diagnostic = "native format readback incompatibility; explicit format omitted, strict local JSON validation remains enabled"
|
||||
c.FormatProbe.Response = string(data)
|
||||
c.formatChecked = true
|
||||
return false, nil
|
||||
}
|
||||
var messages []json.RawMessage
|
||||
if json.Unmarshal(data, &messages) != nil || len(messages) == 0 {
|
||||
return false, fmt.Errorf("invalid structured-format probe readback")
|
||||
}
|
||||
c.formatChecked = true
|
||||
c.formatSupported = true
|
||||
c.FormatProbe.Supported = true
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// ExecutionError exposes a safe classification, not arbitrary provider messages,
|
||||
// prompts or credentials. Full redacted responses belong in failure evidence.
|
||||
func ExecutionError(data json.RawMessage) error {
|
||||
var failure struct {
|
||||
Name string `json:"name"`
|
||||
Tag string `json:"_tag"`
|
||||
Data struct {
|
||||
StatusCode int `json:"statusCode"`
|
||||
} `json:"data"`
|
||||
}
|
||||
if json.Unmarshal(data, &failure) != nil {
|
||||
return fmt.Errorf("OpenCode agent reported an execution error")
|
||||
}
|
||||
name := failure.Name
|
||||
if name == "" {
|
||||
name = failure.Tag
|
||||
}
|
||||
if name == "StructuredOutputError" {
|
||||
return fmt.Errorf("StructuredOutputError: Model did not produce structured output; inspect retained response for tool-call markup or result-envelope mismatch")
|
||||
}
|
||||
safeNames := map[string]bool{"APIError": true, "ProviderAuthError": true, "ContextOverflowError": true, "MessageAbortedError": true, "OutputLengthError": true, "UnknownError": true, "ServiceUnavailableError": true}
|
||||
if !safeNames[name] {
|
||||
return fmt.Errorf("OpenCode agent reported an execution error")
|
||||
}
|
||||
if failure.Data.StatusCode >= 400 && failure.Data.StatusCode <= 599 {
|
||||
return fmt.Errorf("OpenCode agent execution error: %s (HTTP %d)", name, failure.Data.StatusCode)
|
||||
}
|
||||
return fmt.Errorf("OpenCode agent execution error: %s", name)
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package opencode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestNativeFormatProbeDowngradesOnlyReadbackIncompatibility(t *testing.T) {
|
||||
for _, status := range []int{200, 400, 401, 500} {
|
||||
t.Run(http.StatusText(status), func(t *testing.T) {
|
||||
probes, generations := 0, 0
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
switch r.Method + " " + r.URL.Path {
|
||||
case "POST /session":
|
||||
probes++
|
||||
var body struct{ Permission []map[string]string }
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
if len(body.Permission) != 1 || body.Permission[0]["action"] != "deny" {
|
||||
t.Fatal("probe has tools", body)
|
||||
}
|
||||
w.Write([]byte(`{"id":"ses_probe"}`))
|
||||
case "POST /session/ses_probe/message":
|
||||
var body struct {
|
||||
NoReply bool `json:"noReply"`
|
||||
}
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
if !body.NoReply {
|
||||
t.Fatal("probe requested inference")
|
||||
}
|
||||
w.Write([]byte(`{}`))
|
||||
case "GET /session/ses_probe/message":
|
||||
w.WriteHeader(status)
|
||||
if status == 200 {
|
||||
w.Write([]byte(`[{"info":{"role":"user","format":{"type":"json_schema"}}}]`))
|
||||
} else {
|
||||
w.Write([]byte(`{"name":"BadRequest","data":{"message":"Expected OutputFormatJsonSchema at info.format"}}`))
|
||||
}
|
||||
case "POST /session/ses_primary/message":
|
||||
generations++
|
||||
var body map[string]json.RawMessage
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
if (len(body["format"]) > 0) != (status == 200) {
|
||||
t.Error("incorrect format compatibility mode", string(body["format"]))
|
||||
}
|
||||
w.Write([]byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1},"structured":{"outcome":"success","output":{"value":9}}},"parts":[]}`))
|
||||
default:
|
||||
t.Error("unexpected operation", r.Method, r.URL.Path)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
c := &Connection{Backend: "v1", URL: srv.URL, models: map[string]map[string]string{"ses_primary": {"providerID": "test", "modelID": "test"}}, responses: map[string][]byte{}}
|
||||
body := map[string]any{"prompt": "task", "format": map[string]any{"type": "json_schema", "schema": map[string]any{"type": "object"}, "retryCount": 0}}
|
||||
data, err := c.Call(context.Background(), "post", "/api/session/ses_primary/generate", body)
|
||||
if status == 401 || status == 500 {
|
||||
if err == nil || generations != 0 {
|
||||
t.Fatal("unrelated failure silently downgraded", err, generations)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || !strings.Contains(string(data), "value") || generations != 1 {
|
||||
t.Fatal(err, string(data), generations)
|
||||
}
|
||||
if c.FormatProbe == nil || c.FormatProbe.SessionID != "ses_probe" || c.FormatProbe.Supported != (status == 200) {
|
||||
t.Fatal("probe evidence lost", c.FormatProbe)
|
||||
}
|
||||
if _, err = c.Call(context.Background(), "post", "/api/session/ses_primary/generate", body); err != nil || probes != 1 {
|
||||
t.Fatal("capability probe was not cached", err, probes)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNativeStructuredFailureIsClearAndDoesNotReplayGeneration(t *testing.T) {
|
||||
response := []byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1},"error":{"name":"StructuredOutputError","data":{"message":"Model did not produce structured output; secret-token"}}},"parts":[{"type":"text","text":"<|DSML|invoke name=\"StructuredOutput\">not JSON</|DSML|invoke>"}]}`)
|
||||
_, err := messageText(response)
|
||||
if err == nil || !strings.Contains(err.Error(), "StructuredOutputError: Model did not produce structured output") || strings.Contains(err.Error(), "secret-token") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, partialErr := messageText([]byte(strings.Replace(string(response), `"completed":1`, `"completed":0`, 1))); partialErr == nil || !strings.Contains(partialErr.Error(), "StructuredOutputError") {
|
||||
t.Fatal("incomplete timestamp hid provider failure", partialErr)
|
||||
}
|
||||
c := &Connection{Backend: "v1", responses: map[string][]byte{"ses_failure": response}}
|
||||
retained, err := c.Call(context.Background(), "post", "/api/experimental/session/ses_failure/wait", nil)
|
||||
if err == nil || string(retained) != string(response) {
|
||||
t.Fatal("wait failure discarded provider response", err, string(retained))
|
||||
}
|
||||
}
|
||||
@@ -52,11 +52,31 @@ func TestStableNativeSessionContractLive(t *testing.T) {
|
||||
if len(session.Permission) != 3 || session.Permission[2]["permission"] != "bash" {
|
||||
t.Fatal("scoped native permission contract not retained", string(data))
|
||||
}
|
||||
// Validate the real native JSON-schema request without provider inference.
|
||||
_, err = c.Native(ctx, "post", "/session/"+result.Data.ID+"/message", map[string]any{"noReply": true, "format": map[string]any{"type": "json_schema", "retryCount": 0, "schema": map[string]any{"type": "object", "required": []string{"outcome", "output"}, "additionalProperties": false, "properties": map[string]any{"outcome": map[string]any{"type": "string", "enum": []string{"success", "error"}}, "output": map[string]any{"type": "object"}}}}, "parts": []map[string]string{{"type": "text", "text": "No inference; validate structured format."}}})
|
||||
// Validate the complete POST -> retained GET contract, not merely POST
|
||||
// acceptance. The probe itself never runs a model or tools.
|
||||
format, _ := json.Marshal(map[string]any{"type": "json_schema", "retryCount": 0, "schema": map[string]any{"type": "object", "required": []string{"outcome", "output"}, "additionalProperties": false, "properties": map[string]any{"outcome": map[string]any{"type": "string", "enum": []string{"success", "error"}}, "output": map[string]any{"type": "object", "additionalProperties": true}}}})
|
||||
supported, err := c.supportsFormat(ctx, format)
|
||||
if err != nil {
|
||||
t.Fatal("native format capability probe failed", err)
|
||||
}
|
||||
payload := map[string]any{"noReply": true, "parts": []map[string]string{{"type": "text", "text": "No inference; validate retained message readback."}}}
|
||||
if supported {
|
||||
payload["format"] = json.RawMessage(format)
|
||||
}
|
||||
_, err = c.Native(ctx, "post", "/session/"+result.Data.ID+"/message", payload)
|
||||
if err != nil {
|
||||
t.Fatal("native schema request rejected", err)
|
||||
}
|
||||
if _, err = c.Native(ctx, "get", "/session/"+result.Data.ID+"/message", nil); err != nil {
|
||||
t.Fatal("effective format breaks retained-message readback", err)
|
||||
}
|
||||
t.Logf("schema mode supported: %v; retained compatibility probe: %s", supported, c.FormatProbe.SessionID)
|
||||
if !supported {
|
||||
recovered, exportErr := c.Messages(ctx, c.FormatProbe.SessionID)
|
||||
if exportErr != nil || !json.Valid(recovered) || c.MessageReadbackDiagnostic == "" {
|
||||
t.Fatal("cannot recover old format-bearing messages via read-only export", exportErr, string(recovered))
|
||||
}
|
||||
}
|
||||
ref, err := c.Reference()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
package opencode
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type boundedOutput struct {
|
||||
bytes.Buffer
|
||||
overflow bool
|
||||
limit int
|
||||
}
|
||||
|
||||
func (b *boundedOutput) Write(p []byte) (int, error) {
|
||||
n := len(p)
|
||||
remaining := b.limit - b.Len()
|
||||
if n > remaining {
|
||||
b.overflow = true
|
||||
p = p[:remaining]
|
||||
}
|
||||
_, _ = b.Buffer.Write(p)
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// Messages preserves old sessions with native OutputFormat readback failures:
|
||||
// the CLI export reads the local stored data without that HTTP encoder. It does
|
||||
// not regenerate, repair the database, delete history or change conversations.
|
||||
func (c *Connection) Messages(ctx context.Context, id string) ([]byte, error) {
|
||||
if !sessionID.MatchString(id) {
|
||||
return nil, fmt.Errorf("invalid session ID")
|
||||
}
|
||||
c.MessageReadbackDiagnostic = ""
|
||||
data, err := c.Native(ctx, "get", "/session/"+id+"/message", nil)
|
||||
if err == nil {
|
||||
return data, nil
|
||||
}
|
||||
if !strings.Contains(err.Error(), "HTTP 400") || !strings.Contains(string(data), "Expected OutputFormat") {
|
||||
return data, err
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return data, ctx.Err()
|
||||
}
|
||||
executable := c.Executable
|
||||
if executable == "" {
|
||||
executable = Executable()
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, executable, "export", id)
|
||||
cmd.Dir = c.Directory
|
||||
stdout, stderr := &boundedOutput{limit: MaxResponse}, &boundedOutput{limit: 8 << 10}
|
||||
cmd.Stdout = stdout
|
||||
cmd.Stderr = stderr
|
||||
if exportErr := cmd.Run(); exportErr != nil {
|
||||
if ctx.Err() != nil {
|
||||
return data, ctx.Err()
|
||||
}
|
||||
return data, fmt.Errorf("%w; read-only native export fallback failed", err)
|
||||
}
|
||||
if stdout.overflow {
|
||||
return data, fmt.Errorf("%w; native export exceeds 1 MiB evidence limit", err)
|
||||
}
|
||||
var exported struct {
|
||||
Info struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"info"`
|
||||
Messages json.RawMessage `json:"messages"`
|
||||
}
|
||||
if json.Unmarshal(stdout.Bytes(), &exported) != nil || exported.Info.ID != id || len(exported.Messages) == 0 || exported.Messages[0] != '[' {
|
||||
return data, fmt.Errorf("%w; native export returned invalid or mismatched session evidence", err)
|
||||
}
|
||||
c.MessageReadbackDiagnostic = "Native HTTP message readback rejected stored OutputFormat (HTTP 400); read-only CLI export recovered the original messages without modifying the session."
|
||||
return exported.Messages, nil
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package opencode
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestOldFormatReadbackRecoveredWithoutRewritingSession(t *testing.T) {
|
||||
for _, wrongID := range []bool{false, true} {
|
||||
bin := filepath.Join(t.TempDir(), "opencode")
|
||||
id := "ses_old"
|
||||
if wrongID {
|
||||
id = "ses_unrelated"
|
||||
}
|
||||
export := `{"info":{"id":"` + id + `"},"messages":[{"info":{"role":"assistant","error":{"name":"StructuredOutputError"}},"parts":[{"type":"text","text":"original DSML evidence"}]}]}`
|
||||
if err := os.WriteFile(bin, []byte("#!/bin/sh\n[ \"$1\" = export ] || exit 8\n[ \"$2\" = ses_old ] || exit 9\necho '"+export+"'\n"), 0700); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != "GET" {
|
||||
t.Error("evidence reader changed stored session", r.Method)
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(400)
|
||||
w.Write([]byte(`{"name":"BadRequest","data":{"message":"Expected OutputFormatJsonSchema at info.format"}}`))
|
||||
}))
|
||||
c := &Connection{Backend: "v1", URL: srv.URL, Executable: bin}
|
||||
data, err := c.Messages(context.Background(), "ses_old")
|
||||
srv.Close()
|
||||
if wrongID {
|
||||
if err == nil {
|
||||
t.Fatal("unrelated session export accepted")
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err != nil || !json.Valid(data) || !strings.Contains(string(data), "original DSML evidence") || c.MessageReadbackDiagnostic == "" {
|
||||
t.Fatal(err, string(data))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -36,11 +36,14 @@ func messageText(data []byte) (string, error) {
|
||||
} `json:"info"`
|
||||
Parts []struct{ Type, Text string } `json:"parts"`
|
||||
}
|
||||
if json.Unmarshal(data, &message) != nil || message.Info.Role != "assistant" || message.Info.Time.Completed == 0 {
|
||||
if json.Unmarshal(data, &message) != nil || message.Info.Role != "assistant" {
|
||||
return "", fmt.Errorf("OpenCode returned no completed assistant result")
|
||||
}
|
||||
if len(message.Info.Error) > 0 && string(message.Info.Error) != "null" {
|
||||
return "", fmt.Errorf("OpenCode agent reported an execution error")
|
||||
return "", ExecutionError(message.Info.Error)
|
||||
}
|
||||
if message.Info.Time.Completed == 0 {
|
||||
return "", fmt.Errorf("OpenCode returned no completed assistant result")
|
||||
}
|
||||
if len(message.Info.Structured) > 0 && string(message.Info.Structured) != "null" && (message.Info.Finish == "stop" || message.Info.Finish == "tool-calls") {
|
||||
return string(message.Info.Structured), nil
|
||||
@@ -159,7 +162,17 @@ func (c *Connection) Call(ctx context.Context, method, path string, body any) ([
|
||||
}
|
||||
payload := map[string]any{"model": c.models[id], "parts": []map[string]string{{"type": "text", "text": text}}}
|
||||
if len(request.Format) > 0 {
|
||||
payload["format"] = request.Format
|
||||
supported, err := c.supportsFormat(ctx, request.Format)
|
||||
if err != nil {
|
||||
var evidence []byte
|
||||
if c.FormatProbe != nil {
|
||||
evidence = []byte(c.FormatProbe.Response)
|
||||
}
|
||||
return evidence, fmt.Errorf("native structured-format compatibility check: %w", err)
|
||||
}
|
||||
if supported {
|
||||
payload["format"] = request.Format
|
||||
}
|
||||
}
|
||||
response, err := c.Native(ctx, "post", "/session/"+id+"/message", payload)
|
||||
if err != nil {
|
||||
@@ -176,11 +189,14 @@ func (c *Connection) Call(ctx context.Context, method, path string, body any) ([
|
||||
return json.Marshal(map[string]any{"data": map[string]string{"text": result}})
|
||||
case "wait":
|
||||
_, err := messageText(c.responses[id])
|
||||
return nil, err
|
||||
case "message":
|
||||
data, err := c.Native(ctx, "get", "/session/"+id+"/message", nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return c.responses[id], err
|
||||
}
|
||||
return nil, nil
|
||||
case "message":
|
||||
data, err := c.Messages(ctx, id)
|
||||
if err != nil {
|
||||
return data, err
|
||||
}
|
||||
var messages []struct {
|
||||
Info json.RawMessage `json:"info"`
|
||||
|
||||
@@ -18,11 +18,18 @@ func TestNativeSessionModelPermissionsGenerationAndAbort(t *testing.T) {
|
||||
}
|
||||
switch r.Method + " " + r.URL.Path {
|
||||
case "POST /session":
|
||||
var raw map[string]json.RawMessage
|
||||
json.NewDecoder(r.Body).Decode(&raw)
|
||||
if strings.Contains(string(raw["title"]), "compatibility probe") {
|
||||
w.Write([]byte(`{"id":"ses_probe"}`))
|
||||
return
|
||||
}
|
||||
var body struct {
|
||||
Model map[string]string
|
||||
Permission []map[string]string
|
||||
}
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
data, _ := json.Marshal(raw)
|
||||
json.Unmarshal(data, &body)
|
||||
if body.Model["id"] != "model" || body.Model["providerID"] != "provider" || body.Permission[0]["action"] != "deny" || body.Permission[1]["permission"] != "bash" {
|
||||
t.Error("model or scoped permissions lost", body)
|
||||
}
|
||||
@@ -43,6 +50,18 @@ func TestNativeSessionModelPermissionsGenerationAndAbort(t *testing.T) {
|
||||
}
|
||||
prompted = true
|
||||
w.Write([]byte(`{"info":{"role":"assistant","finish":"stop","time":{"completed":1}},"parts":[{"type":"text","text":"result"}]}`))
|
||||
case "POST /session/ses_probe/message":
|
||||
var body struct {
|
||||
NoReply bool `json:"noReply"`
|
||||
Format map[string]any
|
||||
}
|
||||
json.NewDecoder(r.Body).Decode(&body)
|
||||
if !body.NoReply || body.Format["retryCount"] != float64(0) {
|
||||
t.Error("probe requested inference or retries", body)
|
||||
}
|
||||
w.Write([]byte(`{}`))
|
||||
case "GET /session/ses_probe/message":
|
||||
w.Write([]byte(`[{"info":{"role":"user"}}]`))
|
||||
case "GET /session/ses_native/message":
|
||||
w.Write([]byte(`[{"info":{"role":"assistant","finish":"stop","time":{"completed":1}},"parts":[{"type":"text","text":"result"}]}]`))
|
||||
case "POST /session/ses_native/abort":
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
package server
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/pipeline"
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
// Explicitly opt in: this uses existing native-provider credentials, makes ten
|
||||
// small model calls, and retains all stage/probe sessions. No tools are enabled.
|
||||
func TestNativeMathDemoLive(t *testing.T) {
|
||||
executable := os.Getenv("PIPELINES_TEST_OPENCODE_V1")
|
||||
if executable == "" || os.Getenv("PIPELINES_TEST_NATIVE_INFERENCE") != "1" {
|
||||
t.Skip("set PIPELINES_TEST_OPENCODE_V1 and PIPELINES_TEST_NATIVE_INFERENCE=1 for real native math inference")
|
||||
}
|
||||
t.Setenv("PIPELINES_OPENCODE", executable)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Minute)
|
||||
defer cancel()
|
||||
directory := t.TempDir()
|
||||
store, err := pipeline.Open(filepath.Join(directory, "math.db"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer store.Close()
|
||||
var source strings.Builder
|
||||
add := func(key, output, prompt, edges string) {
|
||||
fmt.Fprintf(&source, "# stage %s\n\n```output\n%s\n```\n\n```prompt\n%s\n```\n\n```tools\n```\n\n```configs\nprovider=opencode-go\nmodel=deepseek-v4.1-flash\nretries=0\n```\n\n```edges\n%s\nerror -> #failure\n```\n\n", key, output, prompt, edges)
|
||||
}
|
||||
contract := "Return only the small output map {a,b,value,steps}; value is an integer, steps an array of equation strings."
|
||||
add("sum", contract, "Parse input.prompt as a + b. Compute value=a+b and steps=[\"a + b = value\"] with actual numbers. Preserve a and b.", "success -> multiply")
|
||||
add("multiply", contract, "Preserve a,b and steps. Set value=input.value*3. Append the decimal input.value, followed by ' * 3 = ', followed by the decimal new value, as one equation string. For example, input.value=9 yields '9 * 3 = 27'. Never use placeholder words in equations.", "success -> parity")
|
||||
add("parity", contract, "Do not change any output fields. Select outcome even when input.value is even, odd when it is odd.", "even -> halve\nodd -> add_one")
|
||||
add("halve", contract, "Preserve a,b and steps. Set value=input.value/2. Append the decimal input.value, ' / 2 = ', the decimal new value, and ' (even branch)' as one equation string. For example, input.value=30 yields '30 / 2 = 15 (even branch)'. Never use placeholder words in equations.", "success -> report")
|
||||
add("add_one", contract, "Preserve a,b and steps. Set value=input.value+1. Append the decimal input.value, ' + 1 = ', the decimal new value, and ' (odd branch)' as one equation string. For example, input.value=27 yields '27 + 1 = 28 (odd branch)'. Never use placeholder words in equations.", "success -> report")
|
||||
add("report", "Return only {a,b,value,steps,summary}.", "Preserve a,b,value,steps. Set summary to the exact equation strings in input.steps joined by \"; \".", "success -> #success")
|
||||
if _, err = store.Submit(ctx, "native math regression", source.String()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
items, err := store.List(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := newRunner(ctx, store)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() { cancel(); r.wg.Wait() }()
|
||||
for _, test := range []struct {
|
||||
input, value, summary string
|
||||
path []string
|
||||
}{
|
||||
{"7 + 2", "28", "7 + 2 = 9; 9 * 3 = 27; 27 + 1 = 28 (odd branch)", []string{"sum", "multiply", "parity", "add_one", "report"}},
|
||||
{"7 + 3", "15", "7 + 3 = 10; 10 * 3 = 30; 30 / 2 = 15 (even branch)", []string{"sum", "multiply", "parity", "halve", "report"}},
|
||||
} {
|
||||
run, err := r.start(protocol.Message{PipelineID: items[0].ID, Version: 1, RequestKey: "native-math-" + test.value, Workspace: directory, Input: map[string]any{"prompt": test.input}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.wg.Wait()
|
||||
run, err = store.GetRun(ctx, run.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if run.Status != "succeeded" || fmt.Sprint(run.Output["value"]) != test.value || run.Output["summary"] != test.summary {
|
||||
t.Fatalf("native math failed: input=%s status=%s diagnostic=%s output=%v", test.input, run.Status, run.Diagnostic, run.Output)
|
||||
}
|
||||
var path []string
|
||||
for _, step := range run.Steps {
|
||||
path = append(path, step.Stage)
|
||||
if step.SessionID == "" || step.AgentActive {
|
||||
t.Fatal("session not retained/stopped", step)
|
||||
}
|
||||
}
|
||||
if !reflect.DeepEqual(path, test.path) {
|
||||
t.Fatal("unexpected stage replay/path", path, test.path)
|
||||
}
|
||||
evidence, err := r.evidence(ctx, run.ID)
|
||||
if err != nil || len(evidence) == 0 {
|
||||
t.Fatal("live session evidence unavailable", err)
|
||||
}
|
||||
for _, item := range evidence {
|
||||
if item.Diagnostic != "" {
|
||||
t.Fatal("native session readback failed", item.Diagnostic)
|
||||
}
|
||||
}
|
||||
t.Logf("PASS input=%s result=%s path=%v summary=%s; native session readback passed", test.input, test.value, path, test.summary)
|
||||
}
|
||||
}
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.a-wan.net/alan/pipelines/internal/opencode"
|
||||
"strings"
|
||||
@@ -87,13 +88,13 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
||||
defer cancel()
|
||||
if err != nil {
|
||||
if stopErr := c.Interrupt(cleanup, c.SessionID); stopErr != nil {
|
||||
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection", c.SessionID)
|
||||
err = fmt.Errorf("agent interrupt failed; retained session %s needs inspection: %w", c.SessionID, errors.Join(err, stopErr))
|
||||
return
|
||||
}
|
||||
}
|
||||
if c.OnSession != nil {
|
||||
if persistErr := c.OnSession(c.SessionID, false); persistErr != nil {
|
||||
err = persistErr
|
||||
err = errors.Join(err, persistErr)
|
||||
text = ""
|
||||
return
|
||||
}
|
||||
@@ -148,7 +149,7 @@ func (c *AgentClient) Generate(ctx context.Context, config ModelConfig, prompt s
|
||||
}
|
||||
for _, m := range messages.Data {
|
||||
if len(m.Error) > 0 && string(m.Error) != "null" {
|
||||
return "", fmt.Errorf("agent reported an execution error")
|
||||
return "", opencode.ExecutionError(m.Error)
|
||||
}
|
||||
if m.Finish == "stop" || (m.Finish == "tool-calls" && m.Completed && len(m.Structured) > 0 && string(m.Structured) != "null") {
|
||||
if len(m.Structured) > 0 && string(m.Structured) != "null" {
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestAgentCleanupFailurePreservesOriginalFailureAndActiveOwnership(t *testing.T) {
|
||||
original := errors.New("StructuredOutputError: Model did not produce structured output")
|
||||
cleanup := errors.New("interrupt failed")
|
||||
prompts, releases := 0, 0
|
||||
c := &AgentClient{OpenCodeClient: OpenCodeClient{Directory: t.TempDir()}, OnSession: func(_ string, active bool) error {
|
||||
if !active {
|
||||
releases++
|
||||
}
|
||||
return nil
|
||||
}}
|
||||
c.command = func(_ context.Context, _ string, args ...string) ([]byte, error) {
|
||||
switch args[2] {
|
||||
case "/api/session":
|
||||
return []byte(`{"data":{"id":"ses_cleanup"}}`), nil
|
||||
case "/api/session/ses_cleanup/prompt":
|
||||
prompts++
|
||||
return []byte(`{}`), nil
|
||||
case "/api/experimental/session/ses_cleanup/wait":
|
||||
return nil, original
|
||||
case "/api/session/ses_cleanup/interrupt":
|
||||
return nil, cleanup
|
||||
default:
|
||||
t.Errorf("unexpected operation %v", args)
|
||||
return nil, errors.New("unexpected operation")
|
||||
}
|
||||
}
|
||||
_, err := c.Generate(context.Background(), ModelConfig{Provider: "fixture", Model: "fixture"}, "task")
|
||||
if !errors.Is(err, original) || !errors.Is(err, cleanup) || !strings.Contains(err.Error(), "ses_cleanup") || prompts != 1 || releases != 0 {
|
||||
t.Fatal("cleanup lost original diagnostic, replayed work or released active ownership", err, prompts, releases)
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package stage
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"git.a-wan.net/alan/pipelines/internal/opencode"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -10,24 +11,28 @@ import (
|
||||
// Failure evidence is bounded and credential-redacted; request bodies (which
|
||||
// contain prompts) and the subprocess environment are never recorded here.
|
||||
type OperationEvidence struct {
|
||||
Operation string `json:"operation"`
|
||||
Backend string `json:"backend"`
|
||||
Executable string `json:"executable"`
|
||||
Directory string `json:"directory"`
|
||||
Provider string `json:"provider"`
|
||||
Model string `json:"model"`
|
||||
StartedAt string `json:"started_at"`
|
||||
DurationMS int64 `json:"duration_ms"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
HTTPStatus int `json:"http_status,omitempty"`
|
||||
Diagnostic string `json:"diagnostic"`
|
||||
Stdout string `json:"stdout,omitempty"`
|
||||
Stderr string `json:"stderr,omitempty"`
|
||||
Operation string `json:"operation"`
|
||||
Backend string `json:"backend"`
|
||||
Executable string `json:"executable"`
|
||||
Directory string `json:"directory"`
|
||||
Provider string `json:"provider"`
|
||||
Model string `json:"model"`
|
||||
StartedAt string `json:"started_at"`
|
||||
DurationMS int64 `json:"duration_ms"`
|
||||
ExitCode int `json:"exit_code"`
|
||||
HTTPStatus int `json:"http_status,omitempty"`
|
||||
Diagnostic string `json:"diagnostic"`
|
||||
Stdout string `json:"stdout,omitempty"`
|
||||
Stderr string `json:"stderr,omitempty"`
|
||||
Transport string `json:"transport,omitempty"`
|
||||
NativeOperation string `json:"native_operation,omitempty"`
|
||||
ExitKind string `json:"exit_kind,omitempty"`
|
||||
}
|
||||
|
||||
type OperationLog struct {
|
||||
Provider, Model string
|
||||
Failures []OperationEvidence
|
||||
FormatChecks []opencode.FormatProbe
|
||||
}
|
||||
|
||||
var sensitiveKey = regexp.MustCompile(`(?i)^(authorization|password|secret|api[-_]?key|access[-_]?token|refresh[-_]?token|token|cookie|set-cookie|client[-_]?secret)$`)
|
||||
@@ -37,7 +42,9 @@ var credentialURL = regexp.MustCompile(`(https?://)[^/\s:@]+:[^/\s@]+@`)
|
||||
|
||||
func RedactEvidence(text string) string {
|
||||
var data any
|
||||
if json.Unmarshal([]byte(text), &data) == nil {
|
||||
decoder := json.NewDecoder(strings.NewReader(text))
|
||||
decoder.UseNumber()
|
||||
if json.Valid([]byte(text)) && decoder.Decode(&data) == nil {
|
||||
var clean func(any)
|
||||
clean = func(v any) {
|
||||
switch x := v.(type) {
|
||||
@@ -91,6 +98,13 @@ func (l *OperationLog) record(e OperationEvidence, started time.Time) {
|
||||
return
|
||||
}
|
||||
e.Provider, e.Model = l.Provider, l.Model
|
||||
if e.Backend == "v1" {
|
||||
e.Transport = "native_http"
|
||||
e.ExitKind = "not a subprocess exit"
|
||||
} else {
|
||||
e.Transport = "cli_subprocess"
|
||||
e.ExitKind = "subprocess exit code"
|
||||
}
|
||||
e.StartedAt = started.UTC().Format(time.RFC3339Nano)
|
||||
e.DurationMS = time.Since(started).Milliseconds()
|
||||
e.Diagnostic = RedactEvidence(e.Diagnostic)
|
||||
@@ -102,7 +116,7 @@ func (l *OperationLog) record(e OperationEvidence, started time.Time) {
|
||||
}
|
||||
|
||||
func (l *OperationLog) Transcript(original string) string {
|
||||
if l == nil || len(l.Failures) == 0 {
|
||||
if l == nil || (len(l.Failures) == 0 && len(l.FormatChecks) == 0) {
|
||||
return original
|
||||
}
|
||||
var data map[string]any
|
||||
@@ -112,7 +126,21 @@ func (l *OperationLog) Transcript(original string) string {
|
||||
data["original_transcript"] = original
|
||||
}
|
||||
}
|
||||
data["opencode_failures"] = l.Failures
|
||||
if len(l.Failures) > 0 {
|
||||
data["opencode_failures"] = l.Failures
|
||||
}
|
||||
if len(l.FormatChecks) > 0 {
|
||||
data["native_format_checks"] = l.FormatChecks
|
||||
}
|
||||
encoded, _ := json.Marshal(data)
|
||||
return string(encoded)
|
||||
}
|
||||
|
||||
func (l *OperationLog) recordFormatProbe(probe opencode.FormatProbe) {
|
||||
if l == nil || len(l.FormatChecks) >= 4 {
|
||||
return
|
||||
}
|
||||
probe.Response = RedactEvidence(probe.Response)
|
||||
probe.Diagnostic = RedactEvidence(probe.Diagnostic)
|
||||
l.FormatChecks = append(l.FormatChecks, probe)
|
||||
}
|
||||
|
||||
@@ -3,6 +3,7 @@ package stage
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"git.a-wan.net/alan/pipelines/internal/opencode"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -40,6 +41,9 @@ exit 7
|
||||
}
|
||||
|
||||
func TestEvidenceRedactionAndLimit(t *testing.T) {
|
||||
if output := RedactEvidence(`{"value":9007199254740993,"token":"private-value"}`); !strings.Contains(output, "9007199254740993") || strings.Contains(output, "private-value") {
|
||||
t.Fatal("redaction changed evidence numbers or exposed token", output)
|
||||
}
|
||||
for _, input := range []string{`{"nested":{"access_token":"private-value","password":"private-value"}}`, "Authorization: Basic private-value\n", "https://user:private-value@example.test/path?token=private-value", "api_key=private-value"} {
|
||||
if output := RedactEvidence(input); strings.Contains(output, "private-value") {
|
||||
t.Fatal(output)
|
||||
@@ -49,3 +53,12 @@ func TestEvidenceRedactionAndLimit(t *testing.T) {
|
||||
t.Fatal(len(output))
|
||||
}
|
||||
}
|
||||
|
||||
func TestSuccessfulFormatProbeIsRetainedWithoutBeingCalledAFailure(t *testing.T) {
|
||||
log := &OperationLog{}
|
||||
log.recordFormatProbe(opencode.FormatProbe{SessionID: "ses_probe", Supported: true})
|
||||
transcript := log.Transcript("")
|
||||
if !strings.Contains(transcript, "native_format_checks") || !strings.Contains(transcript, "ses_probe") || strings.Contains(transcript, "opencode_failures") {
|
||||
t.Fatal(transcript)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,10 +22,12 @@ func (c OpenCodeClient) SessionEvidence(ctx context.Context, id string) (string,
|
||||
c.Backend = connection.Backend
|
||||
var session, messages, log []byte
|
||||
logDiagnostic := ""
|
||||
readbackDiagnostic := ""
|
||||
if c.Backend == "v1" {
|
||||
session, err = connection.Native(ctx, "get", "/session/"+id, nil)
|
||||
if err == nil {
|
||||
messages, err = connection.Native(ctx, "get", "/session/"+id+"/message", nil)
|
||||
messages, err = connection.Messages(ctx, id)
|
||||
readbackDiagnostic = connection.MessageReadbackDiagnostic
|
||||
}
|
||||
} else {
|
||||
session, err = c.call(ctx, "get", "/api/session/"+id, nil)
|
||||
@@ -40,6 +42,6 @@ func (c OpenCodeClient) SessionEvidence(ctx context.Context, id string) (string,
|
||||
}
|
||||
}
|
||||
}
|
||||
data, _ := json.Marshal(map[string]string{"session": RedactEvidence(string(session)), "messages": RedactEvidence(string(messages)), "log": RedactEvidence(string(log)), "log_diagnostic": logDiagnostic})
|
||||
data, _ := json.Marshal(map[string]string{"session": RedactEvidence(string(session)), "messages": RedactEvidence(string(messages)), "log": RedactEvidence(string(log)), "log_diagnostic": logDiagnostic, "readback_diagnostic": readbackDiagnostic})
|
||||
return string(data), err
|
||||
}
|
||||
|
||||
@@ -0,0 +1,55 @@
|
||||
package stage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestToolMarkupAndFlattenedResultsAreRejectedAndRetained(t *testing.T) {
|
||||
for _, text := range []string{
|
||||
`<|DSML|invoke name="StructuredOutput">{"outcome":"success","output":{"value":9}}</|DSML|invoke>`,
|
||||
`<tool_call>{"outcome":"success","output":{"value":9}}</tool_call>`,
|
||||
`{"outcome":"success","a":7,"b":2,"value":9,"steps":["7 + 2 = 9"]}`,
|
||||
} {
|
||||
f := &repairFixture{response: text}
|
||||
_, err := (ModelStage{Client: f}).Run(context.Background(), modelDefinition(), nil)
|
||||
if err == nil || f.generateCalls != 1 || f.repairCalls != 0 {
|
||||
t.Fatal("markup/envelope error accepted or replayed", text, err, f)
|
||||
}
|
||||
if len(f.audits) != 1 || f.audits[0].Original != text || f.audits[0].Accepted {
|
||||
t.Fatal("invalid response evidence discarded", f.audits)
|
||||
}
|
||||
}
|
||||
valid := `{"outcome":"success","output":{"evidence":"<tool_call> is a quoted artifact"}}`
|
||||
if _, err := decodeModelResult(valid, modelDefinition().Edges); err != nil {
|
||||
t.Fatal("valid quoted evidence rejected", err)
|
||||
}
|
||||
f := &repairFixture{response: strings.TrimSuffix(valid, "}")}
|
||||
if result, err := (ModelStage{Client: f}).Run(context.Background(), modelDefinition(), nil); err != nil || result.Output["evidence"] != "<tool_call> is a quoted artifact" {
|
||||
t.Fatal("quoted markup blocked safe syntax repair", result, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRuntimeEnvelopeOverridesInnerOutputContract(t *testing.T) {
|
||||
def := modelDefinition()
|
||||
def.Blocks["output"] = "Return only {a,b,value,steps}."
|
||||
spec, err := (ModelStage{}).Spec(def)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prompt, err := modelPrompt(def, spec, Values{"prompt": "7 + 2"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, required := range []string{"output_contract describes ONLY", "Never flatten", "StructuredOutput", "Do not print tool-call markup"} {
|
||||
if !strings.Contains(prompt, required) {
|
||||
t.Fatal("missing envelope rule", required)
|
||||
}
|
||||
}
|
||||
result, err := decodeModelResult(`{"outcome":"success","output":{"a":7,"b":2,"value":9,"steps":["7 + 2 = 9"]}}`, def.Edges)
|
||||
if err != nil || result.Output["value"] != json.Number("9") {
|
||||
t.Fatal(result, err)
|
||||
}
|
||||
}
|
||||
@@ -49,6 +49,12 @@ later stage, input is exactly the output map from the preceding executed stage.
|
||||
Use that runtime data with the stage instructions; no input contract is declared.
|
||||
Select exactly one of allowed_outcomes. Return exactly one JSON object with these
|
||||
two keys: {"outcome":"declared outcome", "output":{...runtime output fields...}}.
|
||||
output_contract describes ONLY the fields inside output, not the outer result.
|
||||
If stage instructions ask for a small output map, put that map inside output;
|
||||
they do not replace this runtime envelope. Never flatten stage fields alongside
|
||||
outcome. If StructuredOutput is offered, its arguments must be the complete
|
||||
{"outcome":"declared outcome","output":{...}} object, not individual output fields.
|
||||
Do not print tool-call markup (including DSML) as text or simulate a tool call.
|
||||
The output must be an object, not null. Do not include Markdown, commentary, or
|
||||
destination stage names. Preserve handoff context when the stage instructions require it.
|
||||
This call has no tools. Do not invent file edits, command results, screenshots, or
|
||||
@@ -63,6 +69,9 @@ Stage definition and runtime input:
|
||||
// Never pick a winner among multiple objects or salvage nested JSON from a
|
||||
// malformed outer object. The extracted object still gets strict validation.
|
||||
func stageResultText(text string) string {
|
||||
if !json.Valid([]byte(strings.TrimSpace(text))) && toolMarkup(text) {
|
||||
return text
|
||||
}
|
||||
remaining := strings.TrimSpace(text)
|
||||
// A single surrounding fence is presentation, even if its JSON is incomplete.
|
||||
for _, prefix := range []string{"```json\n", "```\n"} {
|
||||
@@ -103,7 +112,44 @@ func stageResultText(text string) string {
|
||||
return text
|
||||
}
|
||||
|
||||
func toolMarkup(text string) bool {
|
||||
depth, inString, escape := 0, false, false
|
||||
for i := 0; i < len(text); i++ {
|
||||
if inString {
|
||||
if escape {
|
||||
escape = false
|
||||
} else if text[i] == '\\' {
|
||||
escape = true
|
||||
} else if text[i] == '"' {
|
||||
inString = false
|
||||
}
|
||||
continue
|
||||
}
|
||||
if depth > 0 && text[i] == '"' {
|
||||
inString = true
|
||||
continue
|
||||
}
|
||||
for _, marker := range []string{"<|DSML|", "<|DSML|", "<tool_call>", "<invoke name=", "<function_calls>"} {
|
||||
if strings.HasPrefix(text[i:], marker) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
switch text[i] {
|
||||
case '{', '[':
|
||||
depth++
|
||||
case '}', ']':
|
||||
if depth > 0 {
|
||||
depth--
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func decodeModelResult(text string, edges map[string]string) (Result, error) {
|
||||
if !json.Valid([]byte(text)) && toolMarkup(text) {
|
||||
return Result{}, fmt.Errorf("%w: provider returned tool-call markup as text; expected the outcome/output JSON envelope", ErrInvalidModelResult)
|
||||
}
|
||||
// Preflight syntax to retain the real parser diagnostic and byte position,
|
||||
// rather than replacing truncation with a generic "invalid result fields".
|
||||
var syntax json.RawMessage
|
||||
|
||||
@@ -63,16 +63,27 @@ func (c OpenCodeClient) call(ctx context.Context, method, path string, body any)
|
||||
if c.connection != nil && c.connection.Backend == "v1" {
|
||||
started := time.Now()
|
||||
data, err := c.connection.Call(ctx, method, path, body)
|
||||
if probe := c.connection.FormatProbe; probe != nil {
|
||||
c.Operations.recordFormatProbe(*probe)
|
||||
if !probe.Supported && probe.Diagnostic != "" {
|
||||
c.Operations.record(OperationEvidence{Operation: "get /session/" + probe.SessionID + "/message (format compatibility probe)", Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: 400, Diagnostic: probe.Diagnostic, Stdout: probe.Response}, started)
|
||||
}
|
||||
c.connection.FormatProbe = nil
|
||||
}
|
||||
if err != nil {
|
||||
status := 0
|
||||
if match := httpStatusPattern.FindStringSubmatch(err.Error()); len(match) > 0 {
|
||||
status, _ = strconv.Atoi(match[1])
|
||||
}
|
||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: status, Diagnostic: err.Error(), Stdout: string(data)}, started)
|
||||
evidenceStatus := status
|
||||
if evidenceStatus == 0 {
|
||||
evidenceStatus = c.connection.LastHTTPStatus
|
||||
}
|
||||
c.Operations.record(OperationEvidence{Operation: method + " " + path, Backend: c.Backend, Executable: opencode.Executable(), Directory: c.Directory, ExitCode: -1, HTTPStatus: evidenceStatus, NativeOperation: c.connection.LastHTTPOperation, Diagnostic: err.Error(), Stdout: string(data)}, started)
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err}
|
||||
return nil, &openCodeError{operation: method + " " + path, status: status, cause: err, diagnostic: err.Error()}
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
@@ -80,20 +80,27 @@ func repairModelResult(ctx context.Context, client ModelGenerator, config ModelC
|
||||
return Result{}, err
|
||||
}
|
||||
original := stageResultText(text)
|
||||
record := func(candidate, method string, accepted bool) {
|
||||
if recorder, ok := client.(repairRecorder); ok {
|
||||
bounded := original
|
||||
if len(bounded) > 32<<10 {
|
||||
bounded = bounded[:32<<10] + "\n[truncated]"
|
||||
}
|
||||
recorder.RecordFormatRepair(FormatRepair{Original: bounded, Candidate: candidate, Diagnostic: parseError.Error(), Accepted: accepted, Method: method})
|
||||
}
|
||||
}
|
||||
if len(original) > 32<<10 {
|
||||
record("", "validation", false)
|
||||
return Result{}, fmt.Errorf("%w; format repair skipped: response exceeds 32 KiB repair bound", parseError)
|
||||
}
|
||||
if json.Valid([]byte(original)) {
|
||||
record("", "validation", false)
|
||||
return Result{}, parseError
|
||||
}
|
||||
if tokens, ok := jsonLexemes(original); !ok || len(tokens) == 0 || tokens[0] != "{" {
|
||||
record("", "validation", false)
|
||||
return Result{}, fmt.Errorf("%w; unsafe to repair incomplete values or a non-object response", parseError)
|
||||
}
|
||||
record := func(candidate, method string, accepted bool) {
|
||||
if recorder, ok := client.(repairRecorder); ok {
|
||||
recorder.RecordFormatRepair(FormatRepair{Original: original, Candidate: candidate, Diagnostic: parseError.Error(), Accepted: accepted, Method: method})
|
||||
}
|
||||
}
|
||||
// The most common truncation has one unique repair: close the existing
|
||||
// containers. This cannot change a verdict or any completed evidence value.
|
||||
if candidate, ok := closeJSONContainers(original); ok {
|
||||
|
||||
Reference in New Issue
Block a user