One card, two writers, no shared memory
There is exactly one Live Activity per phone — the fleet card — aggregating every session that is working or needs input. Both the app and the host server can create it, update it, and end it. Neither can observe the other's activities. Almost every bug in this subsystem is a consequence of that one fact.
The retired design was one activity per session. ActivityKit caps an app at five concurrent activities, so past five active sessions it had to truncate and drop sessions entirely. The aggregate card has no ceiling: extra sessions fold into a N More line.
The split of responsibility is about liveness, not about data:
- App Instant and precise, but only while alive.
FleetActivityControllerwatchesSessionStoredirectly and calls ActivityKit in-process. Zero latency, no APNs. Dead the moment iOS suspends the app. - Server Always awake, but can only shout through APNs. The push watcher in
src/push/watcher.tsticks every 2 s, computes the same content state from its own view of the sessions, and sendsstart/update/endpushes. It is the only path that works while the app is suspended.
The two compute the same shape deliberately: FleetActivitySnapshot (Swift, in LFGCore) and reduceFleetLiveActivity (TypeScript) are mirrors, and both derive their per-session verdict from the single shared precedence ladder — SessionDisplayState.resolve on the client, sessionDisplayState on the server. So there is no fidelity seam: whichever writer last touched the card, it says the same thing.
The server half is off unless LFG_LIVE_ACTIVITIES=1 is in the host's .env (it is, on the Pro). It is additionally gated by pushWatcherEnabled, which only pushes from the canonical port 8766 — so a scratch lfg serve an agent starts on a spare port reads the same token store but stays silent, instead of fighting the primary over one card.
What actually travels
The attributes are fixed at creation and never change. Everything the card shows lives in ContentState, which is what both writers replace wholesale on every update.
| Field | Type | Meaning |
|---|---|---|
| fleetId | String | Attributes, not state. Always "fleet". It exists so the app can find "its" card among Activity.activities. |
| working | Int | Sessions mid-turn. Counts all of them, not just the rendered rows. |
| needsInput | Int | Sessions parked on a question, permission prompt, plan approval — or a phone sign-in request, which is modelled as an ordinary prompt. |
| rows | [Row] | At most 3. Needs-input first, then oldest-first within a state. |
| more | Int | Active sessions beyond rows. Rendered as N More. |
| updatedAt | Double | Unix seconds. Deliberately excluded from every change comparison — it moves every tick and would make every tick a push. |
A Row is { sid, title, state, since }. Two details matter:
state is only ever "working" or "needsInput". Not "blocked" — on the client that word means paused and carries a different colour, and a blocked session is neither making progress nor asking anything, so it is not a card row at all. Idle sessions aren't rows either, which is why the card has no unread count: unread is a property of idle sessions, so by definition it is never active.
since is a sticky baseline, keyed by session and state. Both writers carry a since map across ticks and reuse the prior timestamp when the session is still in the same state. Change state and the timer restarts; scroll out of the visible three rows and back, and the elapsed time survives — the server tracks since for every active session, not just the rendered ones.
The elapsed labels are not pushed. ElapsedTimeText wraps a TimelineView(.periodic(by: 60)) that recomputes 2m → 3m locally from since. A card nobody is updating still counts up correctly for as long as its rows are accurate — which is exactly why a frozen card is so easy to mistake for a live one.
Every Codable conformance in LFGFleetAttributes.swift is hand-written with decodeIfPresent and a default. That is not style: an in-flight activity started by an older build gets decoded by the newer widget, and a strict decode would throw rather than render.
Two tokens, two very different lifetimes
The server cannot address a card it was not handed a token for. There are two kinds and confusing them is the root of the most persistent failure in this subsystem.
pushToStart
A property of the install, not of any card. Issued by Activity<LFGFleetAttributes>.pushToStartTokenUpdates, re-minted on essentially every app launch. It is the only way to create a card on a phone whose app is asleep. Outlives every card.
Capped at 3 per APNs environment, newest first. The live store once reached nineteen — every dev build mints one and they rot rather than die, so a start decision was blasting all nineteen.
activityUpdate
A property of one specific live card. Issued by activity.pushTokenUpdates, and it only exists while that card exists. It is the only way to update or end.
Kept at exactly one per environment: registering a new one supersedes the old, because a device only has one card, so the previous token must be a corpse.
A dead Live Activity token does not answer 410. APNs accepts it, returns 200, and silently drops the payload. So isDeadApnsToken never fires for one, nothing prunes it — and worse, the watcher counts that 200 as a delivery, advances its state, never re-sends start, and the real card stops updating until the app is opened. One corpse in the list is enough. Keeping the list to the single token that can actually be live is what makes a 200 mean something.
Registration goes to the default host only, in both LiveActivityManager and FleetActivityController. Registering with every configured host made each host's watcher push-to-start its own card — two hosts, two cards on the lock screen.
| Endpoint | Sent when | Server effect |
|---|---|---|
| POST /api/push/live-activity/start-token | ActivityKit issues a push-to-start token (roughly, every launch) | upsertLiveActivityToken, cap to newest 3 per env |
| POST /api/push/live-activity/update-token | A card exists and ActivityKit issued its token | Supersedes the previous update token and calls noteFleetActivityStarted() — proof a card is on screen, so the server adopts it instead of push-starting a second one beside it |
| POST /api/push/live-activity/ended | The app ended the card itself | noteFleetActivityEnded() — nulls the server's memory and records the veto population |
The registration payload carries no device identifier, so two phones on the same APNs environment fight over the single activityUpdate slot. Distinguishing them needs a device id on the wire.
Four surfaces, three row budgets
One ActivityConfiguration renders the card in four contexts, and the row cap is a hard physical constraint, not a taste call.
Lock screen — 3 rows
The lock screen gives an activity a fixed ~160 pt frame and centre-clips anything taller — which silently drops the header, the most informative line on the card. maxRows = 3 is a height budget, enforced identically in FleetActivitySnapshot.maxRows and MAX_FLEET_ROWS.
Dynamic Island
- Expanded — a tighter budget than the lock screen: 2 rows, with the third folded into the overflow count (
more + max(0, rows.count - 2)). - Compact — an 8 pt accent dot leading, the active total trailing, each padded 6 pt off the sensor cutout.
- Minimal — shown when the island is shared with another app. The same 8 pt dot; Eugene chose no separate treatment (2026-09-20).
Accent
Needs-input dominates. If anything is waiting on a human the dot and keyline go amber #FF9F0A; otherwise green #30D158.
Both writers stamp 100 when needsInput > 0, else 90. This ranks this app's own activities only. Which app is attached to a shared Dynamic Island and which becomes the detached bubble is iOS's choice, and no API influences it (confirmed against Apple's docs, 2026-09-20). It is kept because it is correct should the app ever run two cards.
Birth — two ways a card appears
Path A · The app creates it App
Requires the app to be running. FleetActivityController.sync() finds no existing fleet activity, computes a snapshot with activeTotal > 0, and calls:
Activity.request(attributes: LFGFleetAttributes(fleetId: "fleet"),
content: content(state), pushType: .token)
pushType: .token — not nil — is load-bearing. It is what makes the freshly created card push-capable, so ActivityKit will mint an update token, LiveActivityManager will register it, and the server can keep the card alive after the app suspends. A card created with pushType: nil freezes the instant the app goes away.
Path B · The server push-starts it Server
The only path that works on a sleeping phone. On a tick where reduceFleetLiveActivity sees active sessions and active.current == null, it emits a start action addressed to the push-to-start tokens.
update goes to the previous card's dead token, which answers 200.A start is refused when every currently-active session id is inside the population the phone last reported it ended against (FleetActivityBox.clientEnded). Without this, "phone ends card → server push-starts a replacement → phone ends that one too" ran 27–102 times a day, and because each restart is a new activity, it stacked five stale cards on the lock screen. The veto holds until the population actually changes — some active sid that wasn't in the ended set. It is cleared by a delivered start, or by the phone registering an update token. Logged once per population as start-vetoed, not once per tick.
start requires just one token to accept. Re-blasting starts until all three stale push-to-start tokens accept could stack duplicates on the real device, and an under-delivered start self-heals through token registration anyway. update and end use the opposite rule — see §07.
Update — what wakes each writer
The app App
FleetActivityController.sync() runs on four triggers:
- Launch —
configure(settings:store:)fromLFGApp.init, which arms observation and syncs immediately. - Any observed state change — a re-arming
withObservationTrackingoverstore.sessions,store.busy,store.promptsandsettings.hiddenDirs. The handler must re-arm itself;onChangefires exactly once. - Every completed refresh — the tail of
SessionStore.refresh(). - Foregrounding —
scenePhase == .activeinRootView.
hiddenDirs is in the tracked set on purpose, and the snapshot reads store.filteredSessions, not store.sessions: a card counting sessions the list refuses to show is the same two-surfaces-two-answers bug class the rest of the codebase keeps tripping over.
The app then skips no-op updates with sameRenderableContent, which compares counts, more, and each row's sid / title / state / since — and nothing else. The elapsed labels move every minute on their own; re-pushing for them would be pure churn.
The server Server
One tick every 2 s (skipped if the previous tick is still running). It observes every session, builds the content state, and compares it with sameFleetContentState — the identical field set, updatedAt pointedly excluded. Unchanged → no push, but the refreshed since map is kept in memory and zeroSince is cleared, because activity is activity.
| Server has a card? | Active total | Content changed? | Action |
|---|---|---|---|
| no | 0 | — | nothing |
| no | > 0 | — | start — unless vetoed by clientEnded |
| yes | > 0 | no | nothing; refresh since, clear zeroSince |
| yes | > 0 | yes | update |
| yes | 0 | — | end (debounce is 0) |
One quiet safeguard inside the reducer: a sid already seen this tick is skipped. Duplicate session rows are a real shape in lfg — an in-TUI /resume of a session another live process is on, or a --fork-session fork before its pidfile is readable. A duplicate would render the same session twice in conflicting states, double-count it in the tallies, and burn two of the three row slots on one session.
Death — four ways a card goes away
Ending is the only irreversible move in the whole subsystem. The update token dies with the card, and resurrection needs a push-to-start plus a background wake — the exact chain that misfires. Every end path is therefore gated.
FleetEndGate.step. The critical distinction is between zero and unknown: an unknown count must never end a card, because a background launch from a push-to-start has an empty store and reads exactly like zero.1 · The app ends it, deliberately App
The dominant path when the app is in use. sync() asks FleetEndGate.step for a verdict:
| Condition | Verdict | Effect |
|---|---|---|
countTrustworthy == false | untouched | Leave the card exactly as the server left it. No end, and no zeroed update either. |
trustworthy, activeTotal > 0 | keep | Update if renderable content differs. |
trustworthy, zero, held ≥ hold | end | activity.end(.immediate), then POST /ended. |
fleetCountIsTrustworthy is liveSessionsFetchedOnce && hosts.allSatisfy(isNotKnownDown). GRDB hydration deliberately does not count as a live fetch — it seeds the same storage, so the key set alone cannot tell a cold snapshot from a real answer.
hold is 0. It was 60 s as insurance against end→start churn; Eugene's call on 2026-09-19 was that the card should be gone the moment nothing is running, and the server's FLEET_END_DEBOUNCE_S is 0 to match. The hold machinery survives as a parameter for tests and for the day it is wanted back.
2 · The server ends it Server
A tick with a live card and total == 0 sends event: "end" carrying the zeroed content state and a dismissal-date, then nulls and unlinks its persisted state.
This is the 2026-08-23 zombie-card fix. Update and end target one token per environment, and one environment is routinely a corpse that answers 200 — so "any accepted" was routinely "only the corpse accepted": the real device's send failed, state advanced anyway, and the card froze or refused to dismiss. A shortfall now leaves active.current untouched so the next tick re-sends. Re-updating or re-ending an already-correct card is harmless, and genuine 410s prune tokens, so the requirement converges.
3 · Dedupe ends the losers Both
The app and the server both create cards and neither can see the other's, so LiveActivityManager is the only place the one-card invariant can be enforced. It runs on every activityUpdates arrival — including in the background, since a start push wakes the app — and whenever more than one active fleet activity exists it asks FleetActivityDedupe.partition for the survivor and ends the rest, cancelling their token tasks first.
Survivor is the card with the newest updatedAt: the one both writers most recently addressed. ActivityKit exposes no creation time. Ties break on the larger id, because the order of Activity.activities is not stable.
4 · The tombstone sweep App
Once, at launch, endRetiredPerSessionActivities() enumerates and ends every LFGSessionAttributes activity. A device upgrading from the per-session build can carry up to five of them, and an activity whose attributes type the app no longer declares cannot be enumerated, so it cannot be ended. Keeping the dead type declared in RetiredSessionActivity.swift — with a deliberately empty, leniently-decoding ContentState — is the only way to reach them. It is deletable once no installed build is old enough to have started one.
The lifecycle, end to end
The same machinery, walked through the situations it actually meets.
A · You start a session with the app open
- The journal delta flips
store.busy[sid]. Observation fires;sync()runs within a frame. - No card exists,
activeTotal == 1→Activity.request(pushType: .token). The card appears instantly, with no APNs round trip. activityUpdatesyields the new activity.track()subscribes to itspushTokenUpdates; the first token posts to/update-token.- The server's handler calls
noteFleetActivityStarted()and adopts the card with nocontentState. BecausesameFleetContentState(undefined, …)is false, the very next tick pushes current content to the freshly registered token rather than waiting for something to change. - Both writers are now addressing the same card, agreeing on its contents, and pushing only on renderable change.
B · You lock the phone; a session finishes
The app suspends. Its observation stops. The server's tick sees busy drop, recomputes, finds the row set changed, and sends update to the one live activityUpdate token. The card is correct on the lock screen without the app ever running. This is the path the whole server half exists for, and it works — provided step 4 above happened while the app was awake.
C · The phone is idle and the server push-starts a card
- Server sees an active session, has no card → push-to-start. The card appears.
- iOS is supposed to launch the app in the background so it can register the new card's token.
- On an idle phone this frequently does not happen — 9 of 14 starts on 2026-09-19 got no token back within 90 s.
- Every subsequent
updategoes to the previous card's dead token. APNs answers 200. The visible card freezes at its birth state, counting up its elapsed labels convincingly.
If the app is woken, the second hazard used to fire: configure() → syncNow() runs immediately at launch, when SessionStore.sessions is still []. That empty store read as "nothing active" and ended the server's card about two seconds after it appeared — 32 client-ended events in one hour. FleetEndGate's trustworthiness rule is the fix.
D · Everything finishes
If the app is alive it ends the card locally and posts /ended; the server nulls current, records the veto population, and unlinks fleet-activity-state.json. If the app is asleep the server sends end to the update token and does the same bookkeeping. Either way the card dismisses immediately — debounce is zero on both sides.
E · A host goes offline
The card is not ended and not zeroed. fleetCountIsTrustworthy goes false the moment any host is known down, so the gate returns untouched — the count is unknown, not zero. This mirrors what the session list does: rebuildSessions blanks busy and prompt for sessions on down hosts rather than letting a frozen busy: true read as "Working" forever.
F · The server restarts
fleet-activity-state.json is loaded at watcher start, so a restart does not push a duplicate start for a card the device already has, and every row keeps its elapsed baseline. A subtle race is handled explicitly: a client /ended report arriving while the load is still in flight would otherwise be overwritten a moment later by the on-disk snapshot, resurrecting a card the client just told us is dead. The fleetEndedReported flag makes the newer report win.
G · Two cards somehow exist
The moment the second one arrives through activityUpdates, dedupe ends the older one and cancels its token task. This runs in the background too, before either writer can address the wrong card.
Every guard, and the bug it exists for
None of these are defensive programming in the abstract. Each one is a shipped failure.
| Guard | Where | Without it |
|---|---|---|
| FleetEndGate trustworthiness | client | A background launch from a push-to-start has an empty store; that reads as zero and kills the server's card ~2 s after it appears. "I have to open the app" became literally true — the app was the only thing that could put a card back, because the app was what kept removing it. |
| clientEnded start veto | server | Phone ends card → server push-starts a replacement → phone ends that too. Each restart is a new activity: 27–102 a day, stacking five stale cards on the lock screen. |
| supersede() | server | A dead Live Activity token 200s forever, so nothing prunes it and its 200 is counted as a delivery. The card silently stops updating until the app is opened. |
| accepted == attempted for update / end | server | "Any accepted" meant "the corpse accepted" — the real device's send failed, state advanced anyway, card froze or refused to dismiss. |
| MAX_PUSH_TO_START _PER_ENV = 3 | server | The live store reached nineteen tokens; every start blasted all of them. Per-env so churning sandbox dev builds cannot evict the TestFlight token. |
| FleetActivityDedupe | client | Two cards on the lock screen, only the newest addressable — so the visible one is the stale one. |
| noteFleetActivityStarted on token registration | server | The server push-to-starts a second card next to the one the app just made. |
| seen.has(sid) | server | One session rendered twice in conflicting states, double-counted, burning two of three row slots. |
| updatedAt excluded from comparison | both | Every 2 s tick becomes a push. |
| default host only | client | Each configured host push-starts its own card. Two hosts, two cards. |
| pushWatcherEnabled (port 8766) | server | A scratch lfg serve an agent started on a spare port reads the same token store and pushes to the real phone, with its own idea of active.current. One was caught doing exactly that mid-investigation. |
| maxRows = 3 | both | Overheight content is centre-clipped on the lock screen, silently dropping the header. |
| fleetEndedReported | server | An /ended landing mid-load is overwritten by the on-disk snapshot, resurrecting a dead card. |
What is still open
The idle phone unfixed
The remaining structural hole. The server can only update a card whose update token the phone has handed it, and an idle phone hands over nothing. Whether the app is never launched at all (Apple documents the wake but gives no guarantee; force-quit and power state are undocumented) or is launched and its registration through the Cloudflare tunnel fails inside the short background window, the server cannot tell — and neither is under its control.
Not implemented. With a channel there is no per-card token at all: the server publishes fleet state to one channel per APNs environment and every fleet card — push-started or app-started, on every device — receives it, awake or not.
Shape: create the channel once via POST /1/apps/<bundleId>/channels on api-manage-broadcast.push.apple.com:2196 with message-storage-policy: 1 (stores the most recent message for 8 h, so a card that comes online late still gets current state); publish with POST /4/broadcasts/apps/<bundleId> plus an apns-channel-id header and the same aps body we send today. Start pushes carry input-push-channel; app-started activities use pushType: .channel(id). Prerequisite is a developer-portal toggle — "Enable Broadcast Capability" on the identifier — not an entitlement change. Sandbox and production channels are separate.
Two devices, one slot
The registration payload carries no device identifier, so two phones on the same APNs environment overwrite each other's activityUpdate token. Needs a device id on the wire.
The tombstone
RetiredSessionActivity.swift is scheduled for deletion once no installed build is old enough to have started a per-session activity. They self-expire well within a day, so this is a calendar question, not a technical one.
Reading the evidence
The failure mode here produces no errors — a dead token answers 200 — so the log deliberately records successes. Three separate "the card is stale" investigations had to be re-derived by live repro because nothing recorded what the server sent, to how many tokens, or what came back.
Files
~/.lfg/live-activity-tokens.json — the token registry, both kinds.
~/.lfg/fleet-activity-state.json — the server's memory of the live card. Unlinked on end, so the after-the-fact artifact is destroyed exactly when you want it.
Log events
- decide
- The reducer chose an action. Carries the event, token count, counts, and the row set as
sid:statepairs. - send
- One token, one APNs response. Logged on success too — a 200 does not mean the card updated.
- no-tokens
- An action was due but there was no way to address the card. Otherwise indistinguishable from "nothing to say", and it is exactly the state a suspended app leaves behind when its token rotates.
- none-accepted / partial
- The delivery threshold was not met; state was left alone so the next tick re-sends.
- client-ended
- The phone reported its card gone. The one path that nulls the card with no
decideline — without it, a laterstartreads as inexplicable. - adopted
- An update-token registration proved a card exists; the server took ownership instead of starting a second one.
- start-vetoed
- A start was due but refused. Written once per population, not once per tick.
The question that separates the two failure families: for each decide start, did an adopted follow within ~90 s? If yes and the card is still stale, the problem is delivery or content. If no — the phone never told the server how to reach the new card, and you are looking at the idle-phone gap, not a bug.
Source map
ios/LFGWidgets/LFGFleetActivityWidget.swift — the four surfaces
ios/LFGWidgets/FleetActivityViews.swift — card atoms, tokens, elapsed clock
ios/LFG/FleetActivityController.swift — the app-side writer
ios/LFG/LiveActivityManager.swift — tokens + dedupe
ios/LFG/RetiredSessionActivity.swift — the tombstone
ios/LFGCore/…/FleetActivitySnapshot.swift — pure client reducer
ios/LFGCore/…/FleetEndGate.swift — when ending is allowed
ios/LFGCore/…/FleetActivityDedupe.swift — survivor selection
src/push/watcher.ts — tick, reducer, delivery, the fleet box
src/push/liveactivity.ts — APNs payload builders
src/push/liveactivity-store.ts — token registry, cap, supersede
src/push/fleet-active-store.ts — restart persistence
src/commands/serve.ts — the three endpoints
Diagnosis history
.claude/diagnosis-live-activity-duplicate-cards-20260906.md
.claude/diagnosis-live-activity-duplicate-cards-20260918.md
.claude/diagnosis-live-activity-stale-when-backgrounded-20260919.md
.claude/diagnosis-live-activity-idle-updates-need-broadcast-channel-20260919.md
.claude/fleet-live-activity/plan.md