Engine exits and stream recovery

For SSH/Unix connection drops, also inspect unix_session_end in the listener's telemetry. It identifies the local close path separately from an engine exit; an abnormal socket close does not prove that the shared executor died. See SSH connection diagnosis for causes and upgrade ownership.

The adapter launched the engine, so a recoverable stream problem must never kill it. Three stream faults are recorded and skipped instead of fatal, native turn tracking cannot grow into a self-inflicted stop, and every engine process exit is recorded once with its cause.

Stream faults that keep the engine running

Frame recovery is provided by AppServerConnection (the spawned native engine and each spawned gateway generation). Handler recovery also applies to SocketAppServerConnection, which attaches to shared engines. The adapter's client-facing input stream keeps its 16 MiB hard limit.

For a shared socket, engine_message_failed carries transport: "socket" instead of engine, because the transport does not identify the engine's kind. A callback failure preserves pending RPC requests, subsequent messages, and other clients. If sending the fixed error itself fails, normal transport failure handling still closes that client connection. Invalid JSON, oversized frames, binary frames, and queue floods retain their existing connection limits.

Stdin error (EPIPE) and real stream errors stop the engine only when the process is still running: the adapter waits 500 ms for the engine to finish exiting, and an exit inside that window stays recorded as engine-initiated and its remaining output is allowed to drain, because the engine was already on its way out. A missing executable is stopped at once.

The separate client-facing stdio output treats EPIPE and ERR_STREAM_DESTROYED as a disconnected client, not an engine pipe failure. Pending backpressured emits settle on disconnect or close, and later emits are no-ops. Guard state belongs to the stdout stream object for that object's lifetime: a late completion from an old stream remains handled, while a replaced stream starts with independent state. Unrelated errors reject pending emits so their callers can report them; an unrelated late error with no waiting caller remains fatal. This does not change engine stdin handling or provider network retry policy.

Server callback bookkeeping

Answered server callbacks do not count toward the 256 unanswered-request limit. Ordinary native callbacks, including attestations, do not necessarily produce a serverRequest/resolved notification. Counting their completed replies as pending could eventually block later tool and approval requests.

Each engine connection keeps bounded recent fingerprints and exact completed-ID membership. Numeric IDs are compressed into adjacent ranges of actually seen IDs; an unseen lower ID is not treated as completed just because a higher one arrived first. A completed ID whose fingerprint aged out is refused rather than exposed as a new tool call. Completed-history capacity also fails closed. The bounds are 256 unanswered requests, 256 recent fingerprints per connection, 4096 disjoint completed numeric-ID ranges, and 1024 completed string IDs. IDs from different engine connections never share a history. Delivery order is not assumed to match numeric allocation order.

Threaded callbacks on shared sockets, including dynamic tool calls, retain protected uncertainty records when their replies might have been lost. Ordinary attestation traffic cannot evict those records. A matching warm replay keeps its original desktop request ID; the adapter never automatically approves a request or replays a tool result. Genuine resolution or terminal-turn cleanup releases protected records. A full protected set refuses a new callback before exposing it to the client instead of discarding an older uncertain request. That protected set has a separate 256-entry limit across pending and answered-but-unconfirmed threaded callbacks. The limits remain deliberate fail-closed safeguards, not a promise to accept unbounded unresolved work.

Native turn tracking

src/native-turns.ts tracks which native threads have a turn in progress. Entries are released by turn/completed, by the engine's own thread/closed notification, and by a thread/status/changed whose new tagged status.type is "idle" or "notLoaded". systemError is deliberately not a release: the SDK reports it while the turn is still running, so releasing there would drop a live turn on the floor. A status in no such shape changes nothing. Every release also bumps the notification revision, so a thread/read or thread/resume that was already in flight cannot re-add a thread the engine has closed. The 256-thread cap now evicts the oldest entry instead of throwing, which used to kill a healthy engine after leaked sub-agent threads filled the table:

Engine exits

• initiatedBy: "engine" — the process exited by itself (crash, clean shutdown, or death the app triggered). reason is "none", and active native turns get the existing "The Codex engine stopped while this turn was running…" failure, as before.
• initiatedBy: "adapter" — we stopped it, and reason is one of "session_end" (adapter shutdown), "idle" (an adapter that stopped its engines because it was idle after an app install), "replaced" (an old gateway generation with no loaded chats making room for a new one), "spawn_error" (the process could not be started, or it never finished initialize), "stdin_error", or "stream_error" (a broken pipe or stream on a process that stayed alive).

The event carries no stderr text, paths, thread ids or arguments — only the fields above. Only a process the adapter spawned can say why it stopped: the engines reached over a shared socket (shared native executor or shared gateway executor) have no process the adapter watches, so they emit no engine_exit at all and the reason they are given is ignored. Their loss is reported through the socket paths documented in docs/diagnostics.md.

When an exit is unexpected (initiatedBy: "engine"), correlate uptimeMs with the app's adapter restarts: the app restarts the adapter within about half a second, which starts a new native process and a fresh engine_exit series. adapter_session_end records whether the native engine was already closed when the session ended.

The shared host's control connection

A desktop adapter that shares native chats keeps one extra connection to the executor host (host.sock) next to the engine socket. That connection is what the host keys this adapter's native ownership and generation leases to, and the host drops them when it closes it, so the session cannot go on even though the host process and its engine are still running. The adapter records execution_host_control_lost (only sharedGateway; no paths or chat ids) and ends the session the way a native engine exit does, so the app reconnects within about half a second and the new session discovers the host and acquires a fresh connection. Turns already running in the host-owned engine are not reported failed by this, for the same reason a lost engine socket does not prove SDK death; they are failed only when the host confirms that exact launch died. A connection the adapter closed itself, because it was stopping, records nothing.

Surviving a host restart (1.6.2)

When the control connection closes because the host restarted, and no native turn is running, the session continues instead of ending. ChatGPT 26.1029 shows "hit a snag" whenever its app-server session ends, so this matters more than the half-second reconnect.

host restarts ──► control + engine sockets close
                    │
     native turn running? ── yes ──► session ends as above (execution_host_control_lost)
                    │ no
                    ▼
     rediscover host ─► native/acquire ─► new engine socket
     ─► replay the app's initialize ─► thread/resume {excludeTurns} per loaded native chat
                    │
         ok ──► same session continues (execution_host_reconnected)
         90 s without success ──► session ends (execution_host_reconnect_failed)

The engine socket the adapter hands out (ReconnectingNativeConnection) stays one object throughout. Requests and sends that arrive during the repair wait for it, and its exited and drained settle only when the connection really ends, so nothing in the adapter or the app sees an exit. An engine socket that closes while the control connection stays open (the engine alone stopped) waits 1 s for the control connection. If it stays open, the loss takes the existing exit and death path unchanged. A deliberate stop abandons a repair at once.

Only native chats this session started, resumed or forked and still has loaded are resumed; throwaway chats (composer predictions) are not. Callback bookkeeping from the old engine is dropped, which is safe because no turn was running. Gateway chats on the restarted host do not end the session either: their engine sockets close too, and each re-attaches on its next request (see the re-attach rules in docs/app-server.md). The reconnect telemetry carries only counts and durations.

That exit path waits twice on the desktop app, and a lost host connection bounds both waits to 250 ms. It first lets the frames the engine already sent reach the app; if the app has stopped reading, a forwarded notification parks there and the session never ends. Then, when the host confirms that this exact launch died, the exit reports each live turn with an error and a failed turn/completed — either frame can park the same way, so each is raced against the bound, and once the bound has fired the loop stops rather than blocking on the next turn's pair. Every frame is still handed to the writer before it is awaited, so a bound that had already fired cannot swallow the report of a confirmed death: the notice goes out whenever the app reads again, and the reconnecting app finds those chats through their saved history either way.

The bound is a promise of its own rather than the one a deliberate stop resolves, because that one also ends the native/death check: cutting the confirmation short would lose a genuine death on an exit that has nothing to do with this connection. It arms whether or not this adapter had to close the engine socket itself, so it also bounds an exit whose socket died first.