Add opt-in live thinking-level checks and notes
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# Chat thinking level
|
||||
|
||||
A per-chat thinking (reasoning effort) control in the conversation header,
|
||||
beside the model and mode controls: "Thinking: High". Material opens a menu,
|
||||
Cupertino an action sheet; both list only the levels the chat's current (or
|
||||
saved next-turn) model accepts and check the current one. With no levels the
|
||||
control is not shown. It uses the chat's own gateway, so a chat on another
|
||||
machine is set on that machine's server.
|
||||
|
||||
## Protocol
|
||||
|
||||
- Session rows (`chat_sessions`, `chat_create`, session reads) carry
|
||||
`effort` (current level, `""` = harness default), `efforts`
|
||||
(`[{id, name, description?}]`) and `default_effort`, only when the server
|
||||
knows the row model's levels without starting a harness.
|
||||
- `chat_models` entries carry `efforts`/`default_effort` per model, and the
|
||||
reply carries `chat_effort` with the chat's current state.
|
||||
- `chat_effort {session_id, effort}` sets a level. The server validates it
|
||||
against the current model's levels; the reply echoes
|
||||
`{session_id, effort, efforts, default_effort}`. Errors: unsupported harness,
|
||||
model without levels, a level the model does not list.
|
||||
- Levels apply from the next turn. A running turn keeps its level; the menu
|
||||
says so and the wide label reads "Thinking: Low · next turn" until the turn
|
||||
ends.
|
||||
|
||||
## Per harness
|
||||
|
||||
- **Codex** (app-server 0.164): levels are model/list
|
||||
`supportedReasoningEfforts` for the chat's model (catalog cached for one
|
||||
minute so a list does not read it per row), the default is
|
||||
`defaultReasoningEffort`, and the current level is the thread's own
|
||||
`reasoningEffort` from thread/read unless Pipelines stored one. The stored
|
||||
level is sent as turn/start `effort` (which Codex keeps for later turns).
|
||||
Pipelines stores it in `operator_sessions.chat_effort`, re-applies it
|
||||
before every turn and after a restart, and `/new` copies it.
|
||||
- **Claude Code** (2.1.319): levels are the initialize models'
|
||||
`supportedEffortLevels` where `supportsEffort` is true (Haiku 4.5 has none,
|
||||
so no control). The level is sent with the `apply_flag_settings` control
|
||||
request (`{"effortLevel": level}`, `null` restores the default) before the
|
||||
next turn, only when the process was told another level. It is session-only
|
||||
and never written to a settings file. Stored and re-applied like Codex.
|
||||
Claude Code's initialize reports no default level, so with none chosen the
|
||||
label reads "Thinking: Default". The levels come from a CLI's initialize: after
|
||||
a server restart, before any Claude chat has started a CLI, rows carry no
|
||||
levels and the control stays hidden until a CLI initializes (opening the
|
||||
model chooser or sending a turn does that).
|
||||
- **OpenCode** (2.0.x, v2 API): levels are the `/api/model` entry's
|
||||
`variants` for the session's model. The chosen variant is the session
|
||||
model's `variant`, set with `POST /api/session/{id}/model
|
||||
{model:{id, providerID, variant}}`, which OpenCode itself persists, so
|
||||
Pipelines stores nothing. A session without one reports `variant:
|
||||
"default"`, which is not a listed level and shows as the default. The v1
|
||||
serve backend has no variants: no control.
|
||||
- A model switch (Codex/Claude) keeps a stored level the new model accepts;
|
||||
otherwise it becomes the new model's default (or the harness default). The
|
||||
open control re-reads the levels from `chat_models` after a model switch.
|
||||
|
||||
## Limits
|
||||
|
||||
- The create-chat dialog offers no model choice (new chats start on the
|
||||
server default model), so it shows no level either; set it in the header.
|
||||
- If the opened row's model had no levels, the control is absent, so a switch
|
||||
to a model with levels shows the control after the list re-reads (reopen).
|
||||
- Phone widths show the control as an icon (its semantics label and menu
|
||||
name the level), like the mode control.
|
||||
|
||||
## Verification (2026-11-09)
|
||||
|
||||
- Go: `go test ./internal/protocol ./internal/pipeline ./internal/codex
|
||||
./internal/server` ok. `./internal/claude` and `./internal/opencode`: all new
|
||||
tests pass; one pre-existing failure each (`TestServiceTokenIsPassedOnlyFromAPrivateFile`,
|
||||
`TestStageDiscoveryRequiresPrivateRegistrationBeforeSessionWork`), which fail
|
||||
identically on base 8ca5a1fd.
|
||||
- Flutter: `tool/memwatch/full.sh test/features/chat`: `PASS: 1123 passed, 0
|
||||
failed`. `tool/analyze.sh`: clean.
|
||||
- Live Claude Code (opt-in `PIPELINES_LIVE_CLAUDE_EFFORT=1 go test
|
||||
./internal/claude -run TestClaudeEffortLive -v`, scratch directory, two Haiku
|
||||
turns through the adapter, then `get_settings`):
|
||||
|
||||
```
|
||||
chose low: CLI applied model=claude-haiku-5-5 effort=low
|
||||
chose max: CLI applied model=claude-haiku-5-5 effort=max
|
||||
haiku levels from initialize: [low medium high xhigh max]
|
||||
--- PASS: TestClaudeEffortLive (5.09s)
|
||||
```
|
||||
|
||||
- Live Codex, read-only (`PIPELINES_LIVE_CODEX_EFFORT=1 go test
|
||||
./internal/codex -run TestCodexEffortCatalogLive -v`): e.g.
|
||||
`gpt-6.1-sol default="low" levels=[low medium high xhigh max ultra]`,
|
||||
`gpt-6-luna default="medium" levels=[low medium high xhigh max]`. No Codex
|
||||
turn was run, so turn/start `effort` was verified by the adapter test only.
|
||||
- Live OpenCode, no inference, isolated HOME with `opencode api --standalone`:
|
||||
a new session reported `"variant":"default"`; after posting
|
||||
`variant:"high"` the session read back
|
||||
`{"id": "inkling-2-small-free", "providerID": "opencode", "variant": "high"}`.
|
||||
No OpenCode turn was run.
|
||||
- Not run: a full Pipelines server with the Flutter app against a live
|
||||
harness. Browser captures used the fixture app (`?chats=2&header=1`).
|
||||
@@ -0,0 +1,85 @@
|
||||
package claude
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.a-wan.net/alan/pipelines/internal/protocol"
|
||||
)
|
||||
|
||||
// TestClaudeEffortLive drives the installed Claude Code through this adapter
|
||||
// in a scratch directory: two short Haiku turns, each after a level change,
|
||||
// and reads back the level the CLI applied (get_settings applied.effort). It
|
||||
// runs inference on the signed-in account, so it is opt-in.
|
||||
func TestClaudeEffortLive(t *testing.T) {
|
||||
if os.Getenv("PIPELINES_LIVE_CLAUDE_EFFORT") == "" {
|
||||
t.Skip("set PIPELINES_LIVE_CLAUDE_EFFORT=1 to run two short real Claude Code turns")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Minute)
|
||||
defer cancel()
|
||||
chat := New(ctx)
|
||||
defer chat.Close()
|
||||
session, err := chat.Create(ctx, "Pipelines effort live check", t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := chat.SwitchModel(ctx, session.ID, Backend, "haiku"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
frames := make(chan protocol.ChatEventFrame, 4096)
|
||||
go func() {
|
||||
_ = chat.Stream(ctx, session.ID, nil, func(frame protocol.ChatEventFrame) error {
|
||||
select {
|
||||
case frames <- frame:
|
||||
default:
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}()
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
for _, level := range []string{"low", "max"} {
|
||||
if err := chat.SetEffort(ctx, session.ID, level); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := chat.Prompt(ctx, session.ID, "Reply with the single word OK."); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for ended := false; !ended; {
|
||||
select {
|
||||
case frame := <-frames:
|
||||
ended = frame.Kind == protocol.ChatEventStep && frame.Step != nil && frame.Step.Status == protocol.ChatStepEnded
|
||||
case <-ctx.Done():
|
||||
t.Fatal("turn did not end")
|
||||
}
|
||||
}
|
||||
s := chat.session(session.ID)
|
||||
s.mu.Lock()
|
||||
proc := s.proc
|
||||
s.mu.Unlock()
|
||||
raw, err := proc.request(ctx, "get_settings", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var settings struct {
|
||||
Applied struct {
|
||||
Model string `json:"model"`
|
||||
Effort string `json:"effort"`
|
||||
} `json:"applied"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, &settings); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("chose %s: CLI applied model=%s effort=%s", level, settings.Applied.Model, settings.Applied.Effort)
|
||||
if settings.Applied.Effort != level {
|
||||
t.Fatalf("CLI applied effort %q, want %q", settings.Applied.Effort, level)
|
||||
}
|
||||
}
|
||||
efforts, _, err := chat.Efforts(ctx, protocol.ChatSession{ID: session.ID, Model: "haiku"})
|
||||
if err != nil || len(efforts) == 0 {
|
||||
t.Fatalf("haiku levels = %+v, %v", efforts, err)
|
||||
}
|
||||
t.Logf("haiku levels from initialize: %+v", efforts)
|
||||
}
|
||||
@@ -0,0 +1,42 @@
|
||||
package codex
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TestCodexEffortCatalogLive reads the installed App Server's model/list
|
||||
// through this adapter. It is read-only (no thread, no turn) and opt-in,
|
||||
// because it attaches to the user's Codex connection host.
|
||||
func TestCodexEffortCatalogLive(t *testing.T) {
|
||||
if os.Getenv("PIPELINES_LIVE_CODEX_EFFORT") == "" {
|
||||
t.Skip("set PIPELINES_LIVE_CODEX_EFFORT=1 to read the real Codex model catalog")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
chat, err := Open(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer chat.Close()
|
||||
catalog, err := chat.catalog(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
withLevels := 0
|
||||
for _, model := range catalog {
|
||||
ids := []string{}
|
||||
for _, effort := range model.Efforts {
|
||||
ids = append(ids, effort.ID)
|
||||
}
|
||||
t.Logf("%s default=%q levels=%v", model.ID, model.DefaultEffort, ids)
|
||||
if len(model.Efforts) > 0 {
|
||||
withLevels++
|
||||
}
|
||||
}
|
||||
if withLevels == 0 {
|
||||
t.Fatal("no Codex model reported reasoning efforts")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user