Add opt-in live thinking-level checks and notes

This commit is contained in:
2026-11-09 20:22:33 +00:00
parent 66e6a7cb73
commit 9246c168ac
3 changed files with 225 additions and 0 deletions
+98
View File
@@ -0,0 +1,98 @@
# Chat thinking level
A per-chat thinking (reasoning effort) control in the conversation header,
beside the model and mode controls: "Thinking: High". Material opens a menu,
Cupertino an action sheet; both list only the levels the chat's current (or
saved next-turn) model accepts and check the current one. With no levels the
control is not shown. It uses the chat's own gateway, so a chat on another
machine is set on that machine's server.
## Protocol
- Session rows (`chat_sessions`, `chat_create`, session reads) carry
`effort` (current level, `""` = harness default), `efforts`
(`[{id, name, description?}]`) and `default_effort`, only when the server
knows the row model's levels without starting a harness.
- `chat_models` entries carry `efforts`/`default_effort` per model, and the
reply carries `chat_effort` with the chat's current state.
- `chat_effort {session_id, effort}` sets a level. The server validates it
against the current model's levels; the reply echoes
`{session_id, effort, efforts, default_effort}`. Errors: unsupported harness,
model without levels, a level the model does not list.
- Levels apply from the next turn. A running turn keeps its level; the menu
says so and the wide label reads "Thinking: Low · next turn" until the turn
ends.
## Per harness
- **Codex** (app-server 0.164): levels are model/list
`supportedReasoningEfforts` for the chat's model (catalog cached for one
minute so a list does not read it per row), the default is
`defaultReasoningEffort`, and the current level is the thread's own
`reasoningEffort` from thread/read unless Pipelines stored one. The stored
level is sent as turn/start `effort` (which Codex keeps for later turns).
Pipelines stores it in `operator_sessions.chat_effort`, re-applies it
before every turn and after a restart, and `/new` copies it.
- **Claude Code** (2.1.319): levels are the initialize models'
`supportedEffortLevels` where `supportsEffort` is true (Haiku 4.5 has none,
so no control). The level is sent with the `apply_flag_settings` control
request (`{"effortLevel": level}`, `null` restores the default) before the
next turn, only when the process was told another level. It is session-only
and never written to a settings file. Stored and re-applied like Codex.
Claude Code's initialize reports no default level, so with none chosen the
label reads "Thinking: Default". The levels come from a CLI's initialize: after
a server restart, before any Claude chat has started a CLI, rows carry no
levels and the control stays hidden until a CLI initializes (opening the
model chooser or sending a turn does that).
- **OpenCode** (2.0.x, v2 API): levels are the `/api/model` entry's
`variants` for the session's model. The chosen variant is the session
model's `variant`, set with `POST /api/session/{id}/model
{model:{id, providerID, variant}}`, which OpenCode itself persists, so
Pipelines stores nothing. A session without one reports `variant:
"default"`, which is not a listed level and shows as the default. The v1
serve backend has no variants: no control.
- A model switch (Codex/Claude) keeps a stored level the new model accepts;
otherwise it becomes the new model's default (or the harness default). The
open control re-reads the levels from `chat_models` after a model switch.
## Limits
- The create-chat dialog offers no model choice (new chats start on the
server default model), so it shows no level either; set it in the header.
- If the opened row's model had no levels, the control is absent, so a switch
to a model with levels shows the control after the list re-reads (reopen).
- Phone widths show the control as an icon (its semantics label and menu
name the level), like the mode control.
## Verification (2026-11-09)
- Go: `go test ./internal/protocol ./internal/pipeline ./internal/codex
./internal/server` ok. `./internal/claude` and `./internal/opencode`: all new
tests pass; one pre-existing failure each (`TestServiceTokenIsPassedOnlyFromAPrivateFile`,
`TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork`), which fail
identically on base 8ca5a1fd.
- Flutter: `tool/memwatch/full.sh test/features/chat`: `PASS: 1123 passed, 0
failed`. `tool/analyze.sh`: clean.
- Live Claude Code (opt-in `PIPELINES_LIVE_CLAUDE_EFFORT=1 go test
./internal/claude -run TestClaudeEffortLive -v`, scratch directory, two Haiku
turns through the adapter, then `get_settings`):
```
chose low: CLI applied model=claude-haiku-5-5 effort=low
chose max: CLI applied model=claude-haiku-5-5 effort=max
haiku levels from initialize: [low medium high xhigh max]
--- PASS: TestClaudeEffortLive (5.09s)
```
- Live Codex, read-only (`PIPELINES_LIVE_CODEX_EFFORT=1 go test
./internal/codex -run TestCodexEffortCatalogLive -v`): e.g.
`gpt-6.1-sol default="low" levels=[low medium high xhigh max ultra]`,
`gpt-6-luna default="medium" levels=[low medium high xhigh max]`. No Codex
turn was run, so turn/start `effort` was verified by the adapter test only.
- Live OpenCode, no inference, isolated HOME with `opencode api --standalone`:
a new session reported `"variant":"default"`; after posting
`variant:"high"` the session read back
`{"id": "inkling-2-small-free", "providerID": "opencode", "variant": "high"}`.
No OpenCode turn was run.
- Not run: a full Pipelines server with the Flutter app against a live
harness. Browser captures used the fixture app (`?chats=2&header=1`).
+85
View File
@@ -0,0 +1,85 @@
package claude
import (
"context"
"encoding/json"
"os"
"testing"
"time"
"git.a-wan.net/alan/pipelines/internal/protocol"
)
// TestClaudeEffortLive drives the installed Claude Code through this adapter
// in a scratch directory: two short Haiku turns, each after a level change,
// and reads back the level the CLI applied (get_settings applied.effort). It
// runs inference on the signed-in account, so it is opt-in.
func TestClaudeEffortLive(t *testing.T) {
if os.Getenv("PIPELINES_LIVE_CLAUDE_EFFORT") == "" {
t.Skip("set PIPELINES_LIVE_CLAUDE_EFFORT=1 to run two short real Claude Code turns")
}
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)
defer cancel()
chat := New(ctx)
defer chat.Close()
session, err := chat.Create(ctx, "Pipelines effort live check", t.TempDir())
if err != nil {
t.Fatal(err)
}
if err := chat.SwitchModel(ctx, session.ID, Backend, "haiku"); err != nil {
t.Fatal(err)
}
frames := make(chan protocol.ChatEventFrame, 4096)
go func() {
_ = chat.Stream(ctx, session.ID, nil, func(frame protocol.ChatEventFrame) error {
select {
case frames <- frame:
default:
}
return nil
})
}()
time.Sleep(200 * time.Millisecond)
for _, level := range []string{"low", "max"} {
if err := chat.SetEffort(ctx, session.ID, level); err != nil {
t.Fatal(err)
}
if err := chat.Prompt(ctx, session.ID, "Reply with the single word OK."); err != nil {
t.Fatal(err)
}
for ended := false; !ended; {
select {
case frame := <-frames:
ended = frame.Kind == protocol.ChatEventStep && frame.Step != nil && frame.Step.Status == protocol.ChatStepEnded
case <-ctx.Done():
t.Fatal("turn did not end")
}
}
s := chat.session(session.ID)
s.mu.Lock()
proc := s.proc
s.mu.Unlock()
raw, err := proc.request(ctx, "get_settings", nil)
if err != nil {
t.Fatal(err)
}
var settings struct {
Applied struct {
Model string `json:"model"`
Effort string `json:"effort"`
} `json:"applied"`
}
if err := json.Unmarshal(raw, &settings); err != nil {
t.Fatal(err)
}
t.Logf("chose %s: CLI applied model=%s effort=%s", level, settings.Applied.Model, settings.Applied.Effort)
if settings.Applied.Effort != level {
t.Fatalf("CLI applied effort %q, want %q", settings.Applied.Effort, level)
}
}
efforts, _, err := chat.Efforts(ctx, protocol.ChatSession{ID: session.ID, Model: "haiku"})
if err != nil || len(efforts) == 0 {
t.Fatalf("haiku levels = %+v, %v", efforts, err)
}
t.Logf("haiku levels from initialize: %+v", efforts)
}
+42
View File
@@ -0,0 +1,42 @@
package codex
import (
"context"
"os"
"testing"
"time"
)
// TestCodexEffortCatalogLive reads the installed App Server's model/list
// through this adapter. It is read-only (no thread, no turn) and opt-in,
// because it attaches to the user's Codex connection host.
func TestCodexEffortCatalogLive(t *testing.T) {
if os.Getenv("PIPELINES_LIVE_CODEX_EFFORT") == "" {
t.Skip("set PIPELINES_LIVE_CODEX_EFFORT=1 to read the real Codex model catalog")
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
chat, err := Open(ctx)
if err != nil {
t.Fatal(err)
}
defer chat.Close()
catalog, err := chat.catalog(ctx)
if err != nil {
t.Fatal(err)
}
withLevels := 0
for _, model := range catalog {
ids := []string{}
for _, effort := range model.Efforts {
ids = append(ids, effort.ID)
}
t.Logf("%s default=%q levels=%v", model.ID, model.DefaultEffort, ids)
if len(model.Efforts) > 0 {
withLevels++
}
}
if withLevels == 0 {
t.Fatal("no Codex model reported reasoning efforts")
}
}