Files
pipelines/docs/chat-verification.md
T
alan 976169e413 Record the chat end-to-end verification record
Add docs/chat-verification.md, the required record for the chat verification brief: per-flow table (flow, result, evidence path, notes), the known tree-level gaps, and the section on what could not be verified and why. Verification only; no app behaviour changed.

Checked: packaged templates (internal/author/skill/references/) do not mention this record - no template or skill reference needed updating. Shared README and topic docs are intentionally not touched (single-writer fold).
2026-10-17 18:02:00 +01:00

15 KiB

Chat verification (app chat lane, 17 Oct 2026)

The chat surface gets its own end-to-end verification brief. It is a verification-only pass: nothing was fixed and no app behaviour was changed.

This document is the brief's required record. It is written by the implementation pass, which per the brief's device policy verifies with the build and the unit tests only. The rows that need the phone read device — visual review: the visual reviewer owns the operator's Pixel 9 Pro XL and captures them; this pass did not run adb, any Gradle device task, or the device lock.

What this pass establishes, and what it does not

Established here:

  • the exact revision built, and that it is main's own build (no main feature is missing from it);
  • the build and the unit tests, with the exact commands and their observed results;
  • a source-level map of every flow the brief names to the code path that implements it and the unit test that pins it;
  • the APK published to this run's artifacts directory and the LAN URL.

Not established here (deferred to the visual reviewer, per the device policy):

  • anything that requires the phone: real-data captures, real round-trips, streaming, the approval card, scroll feel, resilience, and every on-screen rendering row marked below.

No failure is reported as observed, because no device flow was executed in this pass; the failure-triage details the brief asks for (capture, steps, exact error text) can only come from the device pass. The two capability gaps that are visible from the tree are recorded under "Known gaps" and are not device failures.

Revision and build

  • Run directory HEAD: 3ffc126. git rev-parse --short HEAD = 3ffc126 and git rev-parse --short main = 3ffc126, so this lane is at main (not behind it). The operator checkout's main is also 3ffc126 (git -C /Users/alan/repos/pipelines rev-parse --short main). No feature of main is absent from the build reviewed here — this is main's build.
  • Server reachable. curl -s http://10.0.0.75:8080/healthz → {"healthy":true}. The phone's own connection to it is the visual reviewer's to confirm.
  • APK published as pipelines-debug-wip-3ffc126.apk (18,552,316 bytes, sha256 5289b4f3d5dc6f9fd587e1e5decf075246b0e82b6fd7255335f2e27a2895aab7), served at http://10.0.0.75:8080/artifacts/298/1/pipelines-debug-wip-3ffc126.apk (HTTP/1.1 200 OK, Content-Type: application/vnd.android.package-archive, Content-Length: 18552316).

Commands and observed output

Run in android/ with JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home (JDK 17.0.20.1), ANDROID_HOME=/Users/alan/Library/Android/sdk, and the cached Gradle ~/.gradle/wrapper/dists/gradle-8.11.1-bin/bpt9gzteqjrbo1mjrsomdt32c/gradle-8.11.1/bin/gradle. Every command used --no-daemon as the brief requires.

gradle --offline --no-daemon --console=plain :app:testDebugUnitTest
  → BUILD SUCCESSFUL in 5s (41 tasks up-to-date, from the warm build)
gradle --offline --no-daemon --console=plain :app:testDebugUnitTest --rerun-tasks
  → BUILD SUCCESSFUL in 22s; testDebugUnitTest executed (22 tasks executed)
gradle --offline --no-daemon --console=plain :app:assembleDebug --rerun-tasks
  → BUILD SUCCESSFUL in 25s; app-debug.apk 18,552,316 bytes,
    sha256 5289b4f3…aab7 (byte-for-byte identical to the non-forced package)

The unit-test JUnit XMLs under android/app/build/test-results/testDebugUnitTest/ aggregate to:

60 suites / 545 tests / 0 failures / 0 errors / 0 skipped

with the chat suites green: ChatProtocolTest, ChatStreamTest, ChatApprovalTest, ChatScrollTest, ChatMultiChatStateTest, ChatSessionManagementTest, ChatListFilterTest, ChatActivityTest, ChatTranscriptTest, ChatFormatTest, ChatDirectoriesTest, ChatAttachmentTest, ChatAttachmentStateTest, MarkdownTest, ToolPanelPositionProviderTest.

Per-flow table

Result keys: Pass (unit) — the flow's logic is pinned by an executed, green unit test; Device — visual review — the on-phone behaviour and capture are the visual reviewer's; Gap (server) — the capability is absent because the brokered server has no request for it. Evidence path is a repository path; device captures will be added by the visual reviewer.

Flow Result Evidence path Notes
Create — directory picker, home first + a recent Pass (unit); device — visual review android/app/src/main/java/net/awan/pipelines/ui/chat/ChatDirectories.kt, ChatCreatePane.kt; android/app/src/test/java/net/awan/pipelines/ui/chat/ChatDirectoriesTest.kt directoryChoices puts the home first and keeps the server's recents order; ChatDirectoriesTest.homeIsFirstAndIsTheDefaultChoice, recentsKeepTheServersOrderAndCreateInTheirFullPath, anEmptyOrBlankReplyStillYieldsTheHomeChoice. On-screen picker not captured here.
Create — the model choice Gap (server) ChatCreatePane.kt (the note at the create footer); ChatFormatTest.chatHeaderMeta* No chat_models/chat_agents request exists and chat_prompt drops model/agent, so the flow says so plainly and shows only the model the server reports. Needs the server catalogue to exist before it can be verified.
Create — session appears tagged Operator Pass (unit); device — visual review ChatProtocol.kt (ChatSession.tag, TAG_OPERATOR), ChatFormat.kt sessionTagLabel; ChatSessionManagementTest.theKindChipComesFromTheServersOwnTag, ChatProtocolTest.chatSessionsCarryTheServersOwnKindTag The chip label derives from the server's own tag; an older server omitting it shows no chip.
List — recency order Pass (unit); device — visual review ChatListPane.kt; ChatListFilter.kt; ChatListFilterTest.filteringKeepsTheRecencyOrderOfTheSurvivors The list is the server's order (MRU first, internal/server/chat.go handleChatSessions); the app never re-sorts it, only narrows.
List — activity marks Pass (unit); device — visual review ChatListPane.kt SessionActivityBadge; ChatSeen.kt; ChatFormat.kt sessionActivity; ChatActivityTest (5 cases) Streaming / New / Open derivations pinned; distinct from each other and from idle.
List — search and filter Pass (unit); device — visual review ChatListPane.kt ChatListSearchControl; ChatListFilter.kt; ChatListFilterTest (9) Case-insensitive over title and directory; blank matches all; survivors keep order; the create action is never hidden.
List — rename and remove Pass (unit); device — visual review ChatListPane.kt; ChatStore.kt rename/remove/undoRemove; ChatSessionManagementTest.rename*, remove*, overrides* Both are local and honest (no server rename/delete); removal is undoable and the dialog says so.
List — no pipeline run session ever appears Pass (unit + server source); device — visual review internal/server/chat.go handleChatSessions (lists registry.OperatorSessions only); ChatStore.kt visibleSessions; ChatSessionManagementTest.theListShowsTheServersSessionsVerbatimWhenNothingIsOverridden The server filters to operator sessions; the app performs no guessing. The brief's on-phone confirmation with real runs present is the visual reviewer's.
Conversation — sending a prompt Pass (unit); device — visual review AppViewModel.kt sendChatPrompt; ChatProtocol.kt promptRequest; ChatProtocolTest.chatPromptRequestCarriesTheAttachedFileUris, chatRequestsRoundTripTheirBodies; ChatAttachmentStateTest.sendingClearsTheAttachmentsAndStillShowsThemOnTheOptimisticTurn Optimistic user turn; the wire shape is pinned.
Conversation — streaming reply arriving Pass (unit); device — visual review ChatStore.kt applyDelta/upsertPart; ChatStreamTest.deltaSequenceGrowsRenderedTextAndTogglesRunning; ChatProtocolTest.chatEventDeltaCarriesTheTargetPartAndText Deltas grow the part and mark the thread running.
Conversation — tool chip expanding Partial (position logic only); device — visual review ChatToolCalls.kt; ToolPanelPositionProviderTest (3); ChatFormatTest.prettyToolInput* The chip/panel tap, the popup overlay and the INPUT/OUTPUT/ERROR blocks are rendering that needs the device; the placement maths and the input formatting are pinned.
Conversation — diff card Partial (diff logic); device — visual review ChatDiffCard.kt; ChatFormatTest.toolEditDiff* (3), toolEditDiffSummarisesAnOversizedEditInsteadOfRenderingEveryLine +N -M counts, line kinds and the oversized summary are pinned; the rendered card is the visual reviewer's.
Conversation — approval once / always / rejected Pass (unit); device — visual review ChatApprovalCard.kt; ChatProtocol.kt replyRequest; ChatApprovalTest (10) The three decisions' wire values, the in-place settle, the server-resolved card and the failed-reply reopen are pinned.
Conversation — stopping a turn mid-stream Pass (unit, wire); device — visual review AppViewModel.kt stopChat; ChatProtocol.kt interruptRequest; ChatProtocolTest.chatInterruptRequestCarriesTheRunningSession The interrupt frame is pinned; the Stop control's live behaviour needs the device.
Switching — two sessions, transcript + scroll each Pass (unit); device — visual review ChatStore.kt open/activate/showList; MultiChatStateTest.switchingPreservesEachConversation, scrollPositionIsPerChat, draftsArePerChat Each open chat keeps messages, draft and scroll; a running chat re-syncs on return.
Scroll — opens at the newest message Pass (unit); device — visual review ChatScroll.kt; ChatTranscript.kt placement; ChatScrollTest.aFreshConversationOpensAtItsNewestMessage, anItemAppendedPastTheFoldHasNoMeasurableGapSoItMustBeBroughtIntoView A chat with no saved position lands on its newest item, instantly (no animation).
Scroll — following while at the bottom Pass (unit); device — visual review ChatTranscript.kt follow effect; ChatScrollTest.theFollowResumesAtTheBottomAndWithinTheThreshold, theFollowMeasuresAgainstTheContentNotTheLayoutThatStillLagsIt; Dimens.kt CHAT_FOLLOW_THRESHOLD_DP (32dp) Every arrival (message, delta, tool chip, grown composer, IME) follows while following.
Scroll — not moving while scrolled up Pass (unit); device — visual review ChatTranscript.kt DragInteraction handling; ChatScrollTest.theFollowDoesNotResumeWhenTheEndIsNotInViewOrTheConversationIsEmpty, aSavedPositionAtTheVeryTopIsRestoredRatherThanTreatedAsNew Only the operator's own drag stops the follow; programmatic scrolls never un-stick it.
Scroll — jump-to-latest Pass (unit); device — visual review ChatTranscript.kt SmallFloatingActionButton; ChatScrollTest.theJumpToLatestIsOfferedOnlyWhileTheOperatorIsAwayFromTheBottom Offered only while not following and something is genuinely below the fold.
Resilience — break the connection, reconnect + re-sync Device — visual review AppViewModel.kt observeConnection (streamResumed().forEach { loadChatHistory(it) }); ChatStreamTest.streamResumedSettlesRunningAndReturnsEveryOpenChat The re-sync logic on the connected edge is pinned; how the connection is broken is for the visual reviewer to choose and name, together with whether the transcript re-syncs.
States — empty, loading, error Partial (store states); device — visual review ChatListPane.kt / ChatConversationPane.kt / ChatCreatePane.kt (LoadingState/ErrorState/EmptyState); ChatStore.kt sessionsLoading/sessionsFailed/historyLoading/failed The state transitions exist and are asserted across stores/tests; the rendered states need the device.
Long conversation — hundreds of messages Partial; device — visual review AppViewModel.kt CHAT_HISTORY_LIMIT = 200; ChatStore.kt history/upsertPart History is fetched up to 200 messages; how a several-hundred-message transcript scrolls and renders needs the device.

Known gaps (from the tree, not device failures)

  • No model choice. The brokered server exposes no request to choose a chat's model and no catalogue to offer one from, and handleChatPrompt drops the model/agent the protocol carries. The create flow states this instead of shipping a dead picker. Classification if seen on the phone: server gap, by design and documented.
  • Attachments are not echoed back. The server stores a user turn as text and does not forward the prompt's files (internal/server/chat.go → broker.Prompt(ctx, sessionID, prompt.Text); internal/opencode/chat.go posts only {"text": …}). The phone shows the file from its own optimistic turn; a history round-trip will not carry it. Classification: server gap.

What could not be verified, and why

Everything that needs the phone. The brief's device policy is explicit: "Implementation and ordinary code review verify with the build and the unit tests only: do not run adb, do not run Gradle device tasks, and do not take the device lock." This is the implementation pass, so the unresolved items are, precisely:

  • every on-screen rendering and capture (create picker, list, conversation, tool panel, diff card, approval card, empty/loading/error states, long transcript);
  • every real-data round-trip against the deployed server (create a chat, send a prompt, receive a streaming reply, answer an approval, stop a turn);
  • the streaming and scroll feel (opening at the newest, following, staying put when scrolled up, jump-to-latest);
  • resilience: breaking the connection a named way and confirming the reconnect and transcript re-sync;
  • the wide-window checks (wm size 2000x2244) and the near-square unfolded shape (wm size 2076x2152), then wm size reset;
  • the per-flow evidence captures that fill the "device — visual review" rows above and the failure-triage records the brief asks for.

Why: the device belongs to the visual reviewer, and this pass changed nothing, so it has no device evidence to offer rather than a fabricated one.

Device state after this pass

This pass ran no adb command (including no screencap, no wm size, no install) and did not take the device lock. It therefore neither read nor changed the phone's installed build, its server URL or its app state, and it cannot report which revision the phone is on. The brief's own closing steps are consequently the visual reviewer's, not this pass's:

  • install and verify main's build with /private/var/folders/jp/q0qvjyg54fl_c5vhhk1ylz2m0000gn/T/opencode/deploy-app.sh main and report the revision the phone ends on;
  • leave the app pointed at the real server (ws://10.0.0.75:8080/connect, workspace /Users/alan/repos/pipelines), force-stopped and relaunched, and say so in the evidence.