Files
pipelines/app/notes/chat-transcript-cache.md
T
alan 0a19ef229f Show a reopened chat's last transcript while it refreshes
ChatScreen keeps a bounded in-memory cache of recently left transcripts
(16 chats / 4000 messages, LRU). A reopened chat is seeded from it with no
loading state, still re-subscribes and re-reads, and the newest page replaces
the cached copy while live frames received during the read are kept. /close
removes the entry; only transcripts backed by a successful read on the
current connection are written.

Adds tool/chat_reopen_probe.mjs and an optional history delay in the loopback
fixture. Measurement shows a second cost the cache does not remove: opening at
the end of the top-anchored transcript lays out every row (notes).
2026-10-24 18:14:27 +01:00

79 lines
4.3 KiB
Markdown

# Chat transcript cache
Reported problem: going list -> chat -> Back -> same or another chat reloads the
whole conversation for a few seconds each time.
## What it does
`ChatScreen` owns a `ChatTranscriptCache` (`lib/features/chat/conversation/chat_transcript_cache.dart`)
next to its scroll store. It is in memory only and lasts as long as the
destination. It is bounded to 16 conversations and 4,000 messages in total.
The least recently used entry is evicted first, and a transcript too large for
the whole bound is not kept.
- Leaving a conversation (phone page pop, wide pane switch, destination teardown)
writes the controller's transcript, cursor and `has_more`. It is written only
if a read on the current connection succeeded. After a socket gap that has not
been repaired, or after a refused read, the earlier entry is kept unchanged.
- Reopening seeds `ChatThreadController` with that entry, so there is no
loading state and the saved scroll position is applied to it immediately. The
controller still runs `chat_subscribe` and then `chat_messages`.
- `refreshCachedChatHistory` merges the newest page. The page decides which
cached messages still exist and supplies its copies and cursor. Messages that
a live frame or local send touched during the read are kept after the page,
so no frame is dropped. Until a read or frame replaces the cached transcript,
the view treats it like a pre-gap transcript: it shows no running turn or
compaction state drawn from it.
- `/close` removes the entry and the closed view never writes it back. A
gateway change clears the cache. Rename and model switch do not affect the
transcript: their row data stays in `_openedSessions` and `_acceptedModels`.
A socket replacement needs no invalidation, because every visit re-reads.
## Measured cause of the delay
`tool/chat_reopen_probe.mjs` serves the built app against the loopback fixture
(100 user/assistant pairs, 390x844, Android UA). `HISTORY_DELAY_MS` holds every
`chat_messages` reply to stand in for a slow server read. These are controlled
samples, not device timings.
| 1.5 s read delay | Reopen, before | Reopen, after |
| --- | ---: | ---: |
| 1x CPU: transcript in semantics (visits 2/3) | 1,661 / 1,641 ms | 550 / 414 ms |
| 4x CPU: frame 250 ms after tap | loading spinner | cached transcript |
Two costs add up:
1. The history read gates the first paint. Every visit waits for it, and the
real server projects the native transcript with tool state. The cache
removes this cost from reopen.
2. Rendering is itself slow for a long transcript. Opening at the end of the
top-anchored lazy `ListView` builds and lays out every row above the end
(101 first builds of the markdown view for 100 turns, with only 2 kept).
The probe recorded one main-thread long task per open of about 250-390 ms at
1x CPU and about 1-3 s at 4x CPU. Markdown parsing is not the cost: parsing
100 answers takes about 5 ms. **The cache does not remove this cost.** With
the cache it happens when the page is pushed instead of when the reply
arrives.
The real fix for (2) is a bottom-anchored transcript, for example a
`CustomScrollView` with a center sliver at the newest row. Opening would then
lay out only the visible tail. That rewrites follow, restore, older-page and
jump-control geometry, which about 66 chat geometry assertions cover, and it
changes how a short chat is aligned. It is left as a separate item rather than
mixed into this one.
## Evidence
- Probe: `HISTORY_DELAY_MS=1500 PROBE_PLATFORM=android CPU_RATE=4 node tool/chat_reopen_probe.mjs <web build> <out>`.
`runs_probe_server.mjs` gained an optional `chatHistoryDelayMs`/`onChatHistory`.
Its default replies immediately, as before.
- Captures: `build/reopen-before-d1500/visit-3-early.png` (spinner) and
`build/reopen-after-d1500/visit-3-early.png` (cached transcript), plus
`result.json` in both directories and in `build/reopen-{before,after}-cpu1/`.
- Tests: `test/features/chat/conversation/chat_transcript_cache_test.dart`
(bounds, merge, seeded controller, live frame during the read, refused read,
gap), `test/features/chat/chat_transcript_reopen_test.dart` (phone page and
wide pane), a cached restore in `chat_thread_scroll_test.dart`, and `/close`
in `chat_command_session_test.dart`.
- Not verified on a device. No APK, no deployment.