Files
pipelines/app/notes/chat-transcript-cache.md
T
alan 0a19ef229f Show a reopened chat's last transcript while it refreshes
ChatScreen keeps a bounded in-memory cache of recently left transcripts
(16 chats / 4000 messages, LRU). A reopened chat is seeded from it with no
loading state, still re-subscribes and re-reads, and the newest page replaces
the cached copy while live frames received during the read are kept. /close
removes the entry; only transcripts backed by a successful read on the
current connection are written.

Adds tool/chat_reopen_probe.mjs and an optional history delay in the loopback
fixture. Measurement shows a second cost the cache does not remove: opening at
the end of the top-anchored transcript lays out every row (notes).
2026-10-24 18:14:27 +01:00

4.3 KiB

Chat transcript cache

Reported problem: going list -> chat -> Back -> same or another chat reloads the whole conversation for a few seconds each time.

What it does

ChatScreen owns a ChatTranscriptCache (lib/features/chat/conversation/chat_transcript_cache.dart) next to its scroll store. It is in memory only and lasts as long as the destination. It is bounded to 16 conversations and 4,000 messages in total. The least recently used entry is evicted first, and a transcript too large for the whole bound is not kept.

  • Leaving a conversation (phone page pop, wide pane switch, destination teardown) writes the controller's transcript, cursor and has_more. It is written only if a read on the current connection succeeded. After a socket gap that has not been repaired, or after a refused read, the earlier entry is kept unchanged.
  • Reopening seeds ChatThreadController with that entry, so there is no loading state and the saved scroll position is applied to it immediately. The controller still runs chat_subscribe and then chat_messages.
  • refreshCachedChatHistory merges the newest page. The page decides which cached messages still exist and supplies its copies and cursor. Messages that a live frame or local send touched during the read are kept after the page, so no frame is dropped. Until a read or frame replaces the cached transcript, the view treats it like a pre-gap transcript: it shows no running turn or compaction state drawn from it.
  • /close removes the entry and the closed view never writes it back. A gateway change clears the cache. Rename and model switch do not affect the transcript: their row data stays in _openedSessions and _acceptedModels. A socket replacement needs no invalidation, because every visit re-reads.

Measured cause of the delay

tool/chat_reopen_probe.mjs serves the built app against the loopback fixture (100 user/assistant pairs, 390x844, Android UA). HISTORY_DELAY_MS holds every chat_messages reply to stand in for a slow server read. These are controlled samples, not device timings.

1.5 s read delay Reopen, before Reopen, after
1x CPU: transcript in semantics (visits 2/3) 1,661 / 1,641 ms 550 / 414 ms
4x CPU: frame 250 ms after tap loading spinner cached transcript

Two costs add up:

  1. The history read gates the first paint. Every visit waits for it, and the real server projects the native transcript with tool state. The cache removes this cost from reopen.
  2. Rendering is itself slow for a long transcript. Opening at the end of the top-anchored lazy ListView builds and lays out every row above the end (101 first builds of the markdown view for 100 turns, with only 2 kept). The probe recorded one main-thread long task per open of about 250-390 ms at 1x CPU and about 1-3 s at 4x CPU. Markdown parsing is not the cost: parsing 100 answers takes about 5 ms. The cache does not remove this cost. With the cache it happens when the page is pushed instead of when the reply arrives.

The real fix for (2) is a bottom-anchored transcript, for example a CustomScrollView with a center sliver at the newest row. Opening would then lay out only the visible tail. That rewrites follow, restore, older-page and jump-control geometry, which about 66 chat geometry assertions cover, and it changes how a short chat is aligned. It is left as a separate item rather than mixed into this one.

Evidence

  • Probe: HISTORY_DELAY_MS=1500 PROBE_PLATFORM=android CPU_RATE=4 node tool/chat_reopen_probe.mjs <web build> <out>. runs_probe_server.mjs gained an optional chatHistoryDelayMs/onChatHistory. Its default replies immediately, as before.
  • Captures: build/reopen-before-d1500/visit-3-early.png (spinner) and build/reopen-after-d1500/visit-3-early.png (cached transcript), plus result.json in both directories and in build/reopen-{before,after}-cpu1/.
  • Tests: test/features/chat/conversation/chat_transcript_cache_test.dart (bounds, merge, seeded controller, live frame during the read, refused read, gap), test/features/chat/chat_transcript_reopen_test.dart (phone page and wide pane), a cached restore in chat_thread_scroll_test.dart, and /close in chat_command_session_test.dart.
  • Not verified on a device. No APK, no deployment.