> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kheish.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Kheish in production

> State-root anatomy, boot and recovery sequencing, journal maintenance, deployment topologies, observability surfaces, upgrades, and backup drills for operating a Kheish daemon at production scale.

# Running Kheish in production

This page is the operator's reference for running the Kheish daemon as durable
infrastructure. It goes past "how do I start it" (that lives in
[Running the daemon](../operations/production)) and into the questions
that matter once a daemon is carrying real traffic: what actually lives on disk,
how the daemon comes back after a restart, how to keep session journals from
growing without bound, how to lay the daemon out on a host, what to watch, how
to upgrade without corrupting state, and how to prove your backups work before
you need them.

The single most important sentence to internalize: **the daemon is the durable
orchestration engine, and the state root is the durable part.** Everything an
integrator or SDK does eventually resolves to state written under one directory
tree. If you understand that tree, you understand the daemon's failure modes,
its backup boundary, and its upgrade contract. The rest of this page is built
around it.

If you are integrating an application on top of the daemon rather than running
it, read [SDKs and the control-plane API](../integrate/sdks-and-api) instead —
this page assumes you own the process.

***

## The two roots

A production daemon is defined by two filesystem roots and one network bind:

* `--state-root` — durable daemon-owned state. Sessions, runs, learnings,
  connectors, encrypted secrets, schedules, deliveries, audit trails. This is
  the thing you back up. Losing it loses the daemon's memory.
* `--workspace-root` — the filesystem the agent's shell and file tools operate
  in. This is the blast radius of tool execution. It is **not** backed up by a
  state-root snapshot and is not part of the daemon's recovery guarantee.
* `--bind` — the control-plane listen address, `127.0.0.1:4000` by default.

```mermaid theme={null}
flowchart TD
    Daemon["kheish-daemon<br/>control plane, agent loop, workers"]
    Daemon -->|"writes durable state"| State["STATE ROOT<br/>back this up<br/>it is the memory of the daemon"]
    Daemon -->|"runs shell and file tools"| Workspace["WORKSPACE ROOT<br/>blast radius<br/>ephemeral by design"]
```

Three rules follow directly and are enforced by the official container
entrypoint (it refuses to start otherwise):

1. `state_root` and `workspace_root` must be **different** directories.
2. Both must be **absolute** paths in a container context.
3. Point the workspace root at something the agent is *allowed* to mutate. A
   source checkout or a broad home directory expands the blast radius of every
   file and shell tool the agent can call.

Always be explicit about which state root you are talking to. A large fraction
of "the daemon is behaving strangely" incidents are actually "I am talking to a
different daemon, or reusing a state root from an older build."

***

## State-root anatomy

The state root is a flat directory of daemon-owned stores. Some are single JSON
documents, some are append-only journals, some are directory trees of per-entity
files. Here is the layout you will see under a busy production root, annotated
with what each entry is for and whether it is hot (rewritten constantly),
append-only (grows), or cold (rarely touched).

```text theme={null}
<state_root>/
├── daemon.lock                     advisory flock; owner of this state root
├── daemon-index.json               daemon-level index of known sessions/entities
│
├── sessions/                       ← the heart of durable state
│   └── __safe/                     safe-encoded namespace for session names
│       ├── <name>.jsonl            append-only session journal (events, checkpoints,
│       │                           permission audits, outputs)
│       ├── <name>.jsonl.vacuum-bak pre-vacuum backup, FULL SIZE, remove after verify
│       └── <name>.meta/            per-key metadata sidecars (last-wins state)
│           ├── goal.json           one small atomically-replaced file per key
│           ├── persona.json        persona snapshot bound to the session
│           ├── capability_scope.json
│           ├── credential_scope.json
│           ├── reply_targets.json
│           └── ...                 (task archive lives alongside as needed)
│
├── run-memories/                   recovered run-memory files (crash-resume context)
├── learnings/                      durable published learnings (verified/provisional)
├── learning-candidates/            captured, not-yet-published learning candidates
├── learning-policy.json            LEGACY migration input only (policy now in runtime-config)
├── skills/                         promoted procedural skills
│
├── runtime-config.json             live runtime revision stream (route/model, permission
│                                   mode, hooks, debug level, tool limits, learning policy)
├── route-inventory.json            non-secret route metadata + loaded routes-file SHA-256
├── routes-overlay.json             runtime-added model routes (no keys; slot refs only)
├── mcp-overlay.json                runtime-added MCP server overlay
│
├── auth/                           encrypted daemon-owned secret store
│   ├── global-slots.json           generic secret slots (auth_ref targets, connector creds)
│   ├── openai-slots.json           provider-scoped slots
│   ├── anthropic-slots.json
│   └── ...                         broker lease + revocation history
├── auth-store-master.key           present only if the daemon generated its own key
├── audit-signing.key               Ed25519 key for signed external-action audit
├── external-actions/               signed per-run external-action audit ledgers
│
├── control-plane-auth/
│   └── audit.jsonl                 auth_failure / auth_rate_limited / cors_origin_rejected
│
├── connector-ingress/              connector ingress bookkeeping
├── runtime-connectors.json         daemon-managed connector definitions (redacted on read)
├── daemon-deliveries.json          delivery queue + delivery ledgers (DLQ lives here)
├── slack-delivery-progress.d/      per-delivery progress markers
├── telegram-delivery-progress.d/
│
├── channels/                       public channel records, messages, leases, thread-work
├── projects/                       daemon-owned projects, members, linked channels, tasks
├── personas/                       daemon-managed persona records + compact persona index
├── assets/                         daemon-owned assets, derived text, preview images
├── derivations/                    durable derived artifacts
├── observations/                   observation sources + captured records
├── schedules/                      durable schedules and wakeups
├── flows.json                      Flow projections over runs
├── outputs/                        persisted output artifacts
├── events/                         event-stream bookkeeping
│
├── hooks.json                      daemon-wide hook config (redacted in summaries)
├── hooks-worktree/                 hook execution worktree
├── hook-dlq/                       hook dead-letter records
│
├── kheish-apply/
│   └── ledger.json                 KheishStack reconciliation ledger (ownership, digests)
│
├── capture-agents/                 provisioned host-local capture agent leases
├── spawn-policy-ledger.json        subagent spawn-policy quota ledger
├── daemon-supervisor.json          supervisor state
├── daemon-supervisor-audit.jsonl   supervisor audit
└── debug/                          debug-capture artifacts (TTL-cleaned)
```

You do not have to memorize every entry. The ones that change how you operate:

### `sessions/` — journals, sidecars, and task archives

Each session has three physical footprints, and `/v1/status` reports all three
separately under `.storage.session_storage`:

* **The journal** (`<name>.jsonl`) is append-only. It holds ordered events,
  compaction checkpoints, permission audits, and output records. It only ever
  grows during normal operation; it is compacted offline by `sessions vacuum`.
* **The metadata sidecars** (`<name>.meta/*.json`) hold last-wins control state:
  the session goal, the bound persona snapshot, the persisted capability and
  credential scopes, the reply-target defaults, and so on. Each key is its own
  atomically-replaced file. A metadata write replaces one small file; it does
  **not** append to the journal. This is the single most important performance
  property of current builds — see [performance notes](#performance-notes).
* **The terminal-task archive** holds settled background-task records.

The `__safe/` layer is a name-encoding namespace so arbitrary session names map
to safe on-disk filenames. You will not usually touch it by hand.

### Secrets, auth, and audit

`auth/` is the encrypted secret store. Route API keys, connector shared tokens,
MCP bearer secrets, and broker lease history all live here, encrypted with
`KHEISH_AUTH_STORE_MASTER_KEY` (or the `_FILE` variant). **The master key is not
stored in a recoverable form inside the state root** unless the daemon generated
its own key into `auth-store-master.key`. If you supply the key via env or file,
you must recover the same value at restore time or the store is unreadable.

`audit-signing.key` is the Ed25519 key that signs the per-run external-action
audit ledgers under `external-actions/`. Keep it with the state root across
backup and restore, or the daemon can read old signed records but cannot
continue the ledger safely.

`control-plane-auth/audit.jsonl` is your security tripwire. It records
`auth_failure`, `auth_rate_limited`, `cors_origin_rejected`, and
`cors_origin_rate_limited`. Ship it to your SIEM.

### Runtime configuration and routes

`runtime-config.json` is the durable revision stream for everything you can
change live: the daemon default route/model, permission mode, learning
automation policy, recovered run-memory policy, tool runtime limits, system
prompt, hooks, and debug level. It is append-a-revision, and the last revision
is restored on boot before the daemon accepts work. `learning-policy.json` is
legacy: it is read only as migration input if present.

`route-inventory.json` records the canonical path and SHA-256 of the routes file
loaded at startup. `doctor routes` compares that hash against the file on disk
and warns `route_file_drift` if someone edited the file without a controlled
reload. `routes-overlay.json` holds routes you added at runtime via the API;
crucially it records **only** slot references, never keys — the keys stay in the
encrypted `auth/` store under `routes.<route_id>.api_key`.

### Deliveries and the dead-letter queue

`daemon-deliveries.json` is the output delivery queue and ledger. When a
connector-backed reply cannot be delivered after retries, it is dead-lettered
here. `/v1/status` `.delivery.unresolved_dead_lettered` is the count that drives
the delivery health warning, and `POST /v1/deliveries/{id}/replay` replays one.

### The apply ledger

`kheish-apply/ledger.json` is the KheishStack reconciliation ledger: ownership,
desired digests, secret fingerprints, and a hash-chained operation history. It
is daemon-owned. Do not pass a separate `--state-root` to `stack` commands; the
daemon owns the ledger under the root it was started with.

***

## Boot and recovery sequence

Understanding boot order tells you what is safe to expect after a restart and
what is deliberately settled fail-closed. The daemon does not "resume the world
exactly as it was" — it restores durable, replayable state and settles anything
that cannot be safely replayed.

```mermaid theme={null}
flowchart TD
    Start["kheish-daemon serve or up, --state-root R"]
    Start --> S1["1. Acquire the state-root lock, flock on daemon.lock.<br/>If another daemon holds it, refuse.<br/>Same mechanism as sessions vacuum, so vacuum-while-serving fails politely."]
    S1 --> S2["2. Load runtime-config.json, restore the LAST revision.<br/>route or model, permission mode, hooks, debug level, tool limits, learning and run-memory policy.<br/>BEFORE accepting work."]
    S2 --> S3["3. Rebuild the route registry.<br/>routes file, path and SHA from route-inventory.json.<br/>plus routes-overlay.json, insertion-ordered, first stays default if the file inventory was empty.<br/>plus resolve auth_ref slots from the encrypted auth store."]
    S3 --> S4["4. Replay durable stores: sessions journal and sidecars, schedules, channels records leases stimuli thread-work, projects, personas, connectors, deliveries, learnings."]
    S4 --> S5["5. Repair run lifecycle.<br/>one wait-capable run per session keeps the active slot.<br/>extra daemon-owned replayable runs, scheduled, mailbox, observation, parent-clarification, are requeued.<br/>extra normal input runs are INTERRUPTED, not replayed.<br/>active shell tasks are SETTLED FAIL-CLOSED: status=failed, terminal_reason=daemon_restarted, recovered_on_boot=true."]
    S5 --> S6["6. Start workers, scheduler, delivery, learning, open the HTTP listener, /readyz flips to 200."]
```

What Kheish **does** restore across a restart: pending approvals, pending
structured questions, active inline skills, schedules, queued deliveries, bound
persona snapshots, session capability/credential scopes, reply-target defaults,
channel state (records, public messages, leases, stimuli, thread-work).

What Kheish **does not** promise: that an arbitrary shell process survives a
restart. It doesn't. Active shell tasks are settled fail-closed on boot so you
never mistake a killed process for a running one. After a restart, inspect:

```bash theme={null}
kheish-daemon tasks list <session_id>          # look for status: "failed"
kheish-daemon tasks get <session_id> <task_id> # terminal_reason: daemon_restarted
kheish-daemon tasks output <session_id> <task_id> --full
```

Retry a failed shell task manually only after deciding it is safe to repeat.
`task_output.retrieval_status=success` means "the daemon read the persisted
output," not "the shell command succeeded."

***

## Performance notes

Two design properties keep a busy daemon fast. Both are worth understanding
because they explain why old builds got slow and current builds do not.

### Metadata sidecars make writes O(value), not O(journal)

Session metadata (control state, persona binding, operator config, and the rest
of the last-wins keys) lives in per-key sidecar files under
`sessions/__safe/<name>.meta/`. A metadata rewrite replaces one small file
atomically. It does **not** append a full control-state snapshot to the journal.

Older builds appended a full control-state snapshot per task mutation. Only the
last snapshot per key ever mattered, so a long-running autonomous session could
accumulate gigabytes of superseded snapshots and slow every run start — the
daemon had to parse the whole journal before it could begin. The sidecar model
removes that failure class for new writes.

A single metadata value larger than 1 MiB logs a write-time warning naming the
session and key, so abnormal growth of one hot key surfaces long before the
journal becomes a problem.

### Lazy journal parsing, scan-miss cache, and tail reads

Journals are parsed lazily. The daemon does not eagerly parse every session file
on every operation:

* **Tail reads:** when persisting incremental records, the daemon reads only the
  file tail rather than re-parsing the whole journal. Metadata records in the
  tail are filtered out so they cannot break incremental replay.
* **Growing tail windows:** when it needs to find a specific record, the daemon
  walks growing tail windows starting on line boundaries instead of parsing from
  byte zero.
* **Scan-miss cache:** when a metadata key is genuinely absent, the daemon
  remembers that miss so it does not rescan the journal for the same key
  repeatedly. A subsequent sidecar write for that key invalidates the cached
  miss.
* **Torn-tail tolerance:** the parser tolerates exactly one torn line at the
  tail (a crash mid-append), and lossy UTF-8 decoding of that torn tail is safe
  because torn-tail tolerance already handles a split multi-byte character.

The practical operator takeaway: journal *reads* stay cheap even as journals
grow, but journals still grow. That is what `sessions vacuum` is for.

***

## Journal maintenance and offline vacuum

The append-only journal grows. On modern builds it grows much more slowly than
before (metadata no longer appends), but a very long-running session still
accumulates events, checkpoints, audits, and outputs. `/v1/status` will tell you
when one has grown too large.

### Detecting oversized sessions

`/v1/status` `.storage.session_storage` reports, per daemon:

* `session_count` and `total_bytes` across journals, metadata sidecars, and task
  archives
* `largest_session_id` / `largest_session_bytes`
* `oversized_session_count` and a bounded `oversized_session_ids` sample for any
  session above `oversized_threshold_bytes` (100 MiB)

Any oversized session also logs a daemon warning naming the session and
suggesting the vacuum command. So you get both a pull signal (poll `/v1/status`)
and a push signal (the daemon log).

### The vacuum command and its safety contract

```bash theme={null}
# The daemon MUST be stopped for this session's state root.
# vacuum takes the state-root lock and refuses politely while a daemon serves it.
kheish-daemon sessions vacuum <session-id> --state-root <state-root>
```

The command prints a JSON report:

* `bytes_before` / `bytes_after`
* `kept_by_type` counts per record type
* `metadata_keys_kept`
* `metadata_records_dropped`
* `torn_tail_dropped`
* `backup_path`

The safety contract, verified end to end in the daemon's own tests:

1. It takes the **same** state-root lock `serve` uses. If a daemon is serving
   that root, vacuum refuses rather than corrupting a live journal.
2. It keeps **events, checkpoints, audits, and outputs in order**. Only
   superseded metadata snapshots are dropped. The loaded session state is
   byte-for-byte equivalent after compaction.
3. The rewrite is **verified against the original before an atomic swap**.
4. The pre-vacuum journal is preserved next to the compacted one as
   `<name>.jsonl.vacuum-bak`. **This backup is full size.** Remove it before
   vacuuming the same session again, or you keep paying for the old bytes.

Field reference: a production session whose journal had grown to 1.4 GB of
superseded control-state snapshots (an older build) compacted to 12 MB in about
11 seconds, and its runs went from stalling for hours before the first tool call
to executing normally after restart.

### Vacuum runbook

```mermaid theme={null}
flowchart TD
    V1["1. SLO warning fires: oversized_session_count above 0"]
    V1 --> V2["2. Confirm the target from /v1/status .storage.session_storage using largest_session_id and oversized_session_ids"]
    V2 --> V3["3. Drain and STOP the daemon for that state root, see Upgrades"]
    V3 --> V4["4. kheish-daemon sessions vacuum the session id --state-root R"]
    V4 --> V5["5. Verify the JSON report: bytes_after much less than bytes_before, metadata_records_dropped above 0, torn_tail_dropped as expected"]
    V5 --> V6["6. Restart the daemon. Confirm /readyz=200 and runs start normally"]
    V6 --> V7["7. Remove the .jsonl.vacuum-bak backup once you trust the compaction"]
```

### The downgrade non-guarantee

Running an **older** daemon build against a state root that already has metadata
sidecars is not supported. The old build appends metadata inline; the newer
build's sidecars keep precedence on read, so those inline writes are shadowed and
effectively lost on the next upgrade. **Stay on one build per state root.** If
you must downgrade, accept that metadata writes made by the old build are lost
when you go forward again.

***

## Deployment topologies

Three shapes cover almost every production deployment. Pick the simplest one
that satisfies your exposure requirement.

### Topology A — single host, loopback only

The default and the safest. The daemon binds loopback, nothing external reaches
it, and operators use the CLI or SDK from the same host (or through an SSH
tunnel). No TLS, no CORS surface, minimal attack surface.

```mermaid theme={null}
flowchart TD
    subgraph Host["host"]
        CLI["operator CLI or SDK"]
        Daemon["kheish-daemon"]
        State["state_root"]
        Workspace["workspace_root"]
        CLI -->|"127.0.0.1:4000"| Daemon
        Daemon --> State
        Daemon --> Workspace
    end
```

Use this for automation hosts, CI runners, and any daemon that only ever talks
to same-host clients. Bearer auth is optional here but still recommended.

### Topology B — reverse-proxied with TLS

The daemon stays on loopback (or a private service network); a dedicated reverse
proxy or ingress controller terminates TLS and forwards to it. Bearer auth stays
enabled behind the proxy. This is the standard shape for exposing a daemon to a
team or an application backend.

```mermaid theme={null}
flowchart LR
    Net["Internet or VPN"]
    Proxy["Nginx or Caddy<br/>TLS terminate<br/>SSE buffering OFF"]
    Daemon["kheish-daemon<br/>127.0.0.1:4000<br/>bearer enabled"]
    Orch["orchestrator"]
    Net -->|"https to kheish.example.com"| Proxy
    Proxy -->|"plain HTTP loopback, Authorization forwarded, X-Forwarded headers set"| Daemon
    Orch -->|"/healthz and /readyz reachable, SSE streams NOT buffered"| Proxy
```

Non-negotiables for the proxy layer, all validated by the shipped Nginx fixture
smoke (`scripts/e2e/ops_reverse_proxy_config_smoke.sh`):

* forward `Authorization` unchanged; keep daemon bearer auth on
* disable buffering for `/v1/events/stream`, `/v1/sessions/*/stream`,
  `/v1/runs/*/stream`, and `/v1/flows/*/stream` (SSE breaks under buffering)
* keep CORS **out** of the proxy layer; Kheish's own CORS allowlist is
  intentionally loopback-only, and browsers should be served same-origin
* preserve request bodies for connector ingress; do not strip signature headers
* set body-size limits deliberately for your attachment/debug/output workloads
* set long idle timeouts so SSE connections survive

Validate after deploy:

```bash theme={null}
kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor
kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor --cors-origin https://kheish.example.com
kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor routes --check-auth
```

`doctor` fails closed on broken `/readyz`, malformed or buffered SSE, storage
write-probe failures, missing state-root lock ownership, unsafe CORS preflight
responses, route/provider readiness errors, and invalid hook definitions. It
also checks configured HTTP hook hostnames against DNS and warns before a hook
dispatch surprises you with a private/local address resolution.

### Topology C — container: daemon plus remote connectors

The recommended container shape is **one daemon container** plus **separate
connector services** talking to it over `remote_http`. The official daemon image
is deliberately not the home for every platform-specific sidecar runtime.

```mermaid theme={null}
flowchart LR
    subgraph Orch["orchestrator"]
        Daemon["kheish-daemon<br/>0.0.0.0:4000<br/>bearer auth<br/>MCP discovery OFF"]
        Connector["connector service<br/>webhook, slack, telegram"]
        State["state volume<br/>/var/lib/kheish/state"]
        Work["workspace volume<br/>/workspace"]
        Secrets["secrets<br/>/run/secrets/... Docker-managed, 0600 on host"]
        Daemon <-->|"remote_http /v1/connectors/external"| Connector
        Daemon --> State
        Daemon --> Work
        Daemon --> Secrets
    end
```

Container defaults baked into the official image:

* `KHEISH_BIND=0.0.0.0:4000`
* `KHEISH_STATE_ROOT=/var/lib/kheish/state`
* `KHEISH_WORKSPACE_ROOT=/workspace`
* `KHEISH_HTTP_AUTH_MODE=bearer`
* `KHEISH_MCP_DISCOVERY=disabled` (so a container never silently imports
  developer-local MCP config from `$HOME/.codex`)

The image ships the `kheish-daemon` binary plus `bash`, `ripgrep`, `procps`,
`git`, `ca-certificates`, and `tini`, runs as non-root, and treats `SIGTERM` and
`SIGINT` as graceful shutdown. It does **not** bundle arbitrary Node/Python
runtimes for external MCP servers or `child_process` connectors. If your MCP
config needs those, build a custom image or run them as separate services.

Prefer file-backed secrets in containers:

* `KHEISH_DAEMON_ADMIN_TOKEN_FILE`
* `KHEISH_DAEMON_READONLY_TOKEN_FILE`
* `KHEISH_AUTH_STORE_MASTER_KEY_FILE`

The entrypoint refuses obvious misconfiguration before the daemon starts:
relative `state_root`/`workspace_root`, identical roots, non-loopback bind
without bearer auth, unreadable token/key files, and conflicting inline-plus-file
secret inputs for the same setting.

See [Docker and containers](../operations/production) for the full
Compose walkthrough.

***

## Observability surfaces

The daemon exposes several distinct observability surfaces. Knowing which is
which keeps you from, for example, using an authenticated endpoint for a
liveness probe.

```mermaid theme={null}
flowchart TD
    Map["Observability surfaces"]
    Map --> Unauth["Unauthenticated probes<br/>outside auth middleware"]
    Map --> Snapshot["Authenticated operator snapshot"]
    Map --> Ring["Daemon log ring buffer"]
    Map --> Delivery["Delivery and DLQ"]
    Map --> Streams["Live streams SSE"]
    Map --> Audit["Security audit on disk"]
```

* **Unauthenticated probes** (outside auth middleware): `GET /healthz` liveness; `GET /readyz` readiness, 503 while draining on shutdown. Use these for Docker/K8s probes. NEVER reuse `/v1/*` for probes.
* **Authenticated operator snapshot**: `GET /v1/status` returns `health.ok`, `storage.ok` + `probes[]`, `storage.session_storage` (sizes, oversized), `provider_readiness`, `runs.{queued,running,waiting_*}`, `delivery.unresolved_dead_lettered`, `runtime.debug_level`, `health.warnings[]`.
* **Daemon log ring buffer**: `GET /v1/logs?limit=N` returns the last N of a bounded 5000-entry in-memory ring of the daemon's own tracing events (scheduler, deliveries, MCP, errors) — no shell access needed.
* **Delivery / DLQ**: `GET /v1/deliveries[?status=dead_lettered]`, `GET /v1/deliveries/dead-letter`, `POST /v1/deliveries/{id}/replay`, `GET /v1/runtime/connectors/external/metrics`.
* **Live streams (SSE)**: `GET /v1/events/stream` (global, filterable by session\_id/run\_id), `GET /v1/sessions/{id}/stream`, `GET /v1/runs/{id}/stream`, `GET /v1/flows/{id}/stream` (proxy over the root run stream).
* **Security audit (on disk)**: `<state_root>/control-plane-auth/audit.jsonl`, `<state_root>/external-actions/` (signed per-run external audit).

### `/v1/status` is the operator snapshot

`kheish-daemon status` fetches it. It is the one call to make first in almost any
investigation. It reports readiness, active runtime route, run queues and their
ages, schedules, agents, mailboxes, storage write health, provider readiness,
per-session storage footprint, and live background shell tasks.

### `/v1/logs` is a bounded ring buffer

The daemon mirrors its own filtered tracing events into an in-memory ring buffer
of **5,000 entries**. `GET /v1/logs?limit=N` returns up to `N` most recent
entries, oldest first (default 1,000). Each entry carries `timestamp_ms`,
`level`, `target` (the emitting module), `message`, and remaining `fields`.

Two properties matter operationally:

1. It is **bounded** — it can never grow the daemon's memory footprint, and old
   entries fall off the front. It is a live tail, not a durable log store. If you
   need durable logs, capture the process's stderr/stdout separately.
2. It only retains what the **active tracing filter already lets through**. If
   you filtered a target out at the tracing layer, it will not appear here.

This is what powers the **daemon logs explorer in Kheish Air**: operators get a
live, filterable view of what the whole daemon is doing — scheduler ticks,
delivery attempts, MCP child-process events, errors — without SSH access to the
process log file.

### Delivery dead letters and replay

Connector-backed output that cannot be delivered after retries is dead-lettered.
`/v1/status` `.delivery.unresolved_dead_lettered` counts DLQ entries that do not
yet have a completed replay — that is the count the delivery health warning uses.
`status.delivery.dead_lettered` is historical (lifetime count).

To recover:

```bash theme={null}
kheish-daemon deliveries dead-letter                      # inspect the DLQ
kheish-daemon deliveries list --run-id <run_id>           # scope to one run
kheish-daemon deliveries replay <delivery_id>             # create a new pending replay
```

Replays are idempotent by default: a replay creates a *new* pending delivery with
a new `delivery_id` and `replayed_from_delivery_id` pointing at the original,
which stays as audit history. A later non-forced replay returns the existing
pending/delivered/dead-lettered replay. Use `force=true` only for an intentional
second attempt.

### Health warnings

`/v1/status` `.health.warnings[]` is a structured list of the daemon's own
concerns. Codes you will see include `queued_run_lag`, stale-run codes,
oversized-session warnings, provider-readiness warnings, and delivery warnings.
Treat a non-empty warnings array as "read this before it becomes a page."

### SLO signal thresholds

Starting thresholds for production-like deployments:

| Signal                      | JSON path or probe                                                               | Warning                                   | Page                                           |
| --------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------- | ---------------------------------------------- |
| Readiness                   | `GET /readyz`                                                                    | one non-2xx sample                        | two consecutive non-2xx within 60s             |
| Health                      | `/v1/status` `.health.ok`                                                        | `false` once                              | `false` twice consecutively                    |
| Storage                     | `.storage.ok`, `.storage.probes[]`                                               | any warning diagnostic                    | `false` immediately                            |
| Session growth              | `.storage.session_storage.oversized_session_count`, `.largest_session_bytes`     | any session above 100 MiB → plan a vacuum | oversized session still growing after a vacuum |
| Provider readiness          | `.provider_readiness.active_route_ready`, `.routes[]`                            | default route warning 5 min               | default route error 5 min                      |
| Run queue lag               | `.runs.queued`, `.runs.oldest_queued_run_age_ms`                                 | older than 30 min behind                  | older than 2 h behind                          |
| Stale active runs           | `.runs.running`, `.runs.oldest_non_terminal_run_idle_ms`                         | older than 2 h without events             | older than 6 h without events                  |
| Pending approvals/questions | `.runs.waiting_for_approval`, `.runs.waiting_for_user_question`                  | older than 4 h                            | older than 24 h                                |
| Delivery DLQ                | `.delivery.unresolved_dead_lettered`                                             | any growth in staging                     | any production DLQ item                        |
| Auth/CORS limits            | `control-plane-auth/audit.jsonl` `auth_rate_limited`, `cors_origin_rate_limited` | repeated within 10 min                    | sustained 30 min                               |
| Debug capture               | `.runtime.debug_level`                                                           | `full` outside a ticketed window          | `full` for more than 30 min                    |
| State-root disk             | host FS metrics for `--state-root`                                               | above 80%                                 | above 90% or inode exhaustion                  |

Run the CI-ready probe smoke to exercise these documented paths:

```bash theme={null}
KHEISH_OPS_SLO_MODEL=gpt-5.4 bash scripts/e2e/ops_slo_probe_smoke.sh
```

***

## Live runtime configuration

Several things change without a restart, serialized through the durable
`runtime-config.json` revision stream:

```bash theme={null}
kheish-daemon runtime get                                   # current settings + revision
kheish-daemon runtime set-model openai/gpt-5.4              # daemon default route/model
kheish-daemon runtime set-permission-mode accept-edits
kheish-daemon runtime set-debug-level redacted
kheish-daemon runtime tool-limits set --file tool-limits.json
kheish-daemon runtime learning-policy set --file learning-policy.json
kheish-daemon runtime hooks get
kheish-daemon runtime revisions
kheish-daemon runtime rollback --target-revision 7 --expected-revision 12
```

Operational semantics that trip people up:

* These are **daemon-wide** and affect **future** execution. They do not
  retroactively change the pinned route, model, debug level, or tool-limit
  snapshot of an already-active tool batch.
* `runtime set-model` on a mixed-provider daemon changes the **fallback** route,
  not every session at once.
* Mutations can include `expected_revision`; a stale writer gets `409` with
  `runtime/runtime_revision_conflict`. Use this when automating changes.
* Rollback appends a *new* revision recording the restored target; it does not
  delete history. Only revisions still returned by `runtime revisions` are valid
  rollback targets (older history is pruned to `config.history_limit`).

Adding a model route at runtime persists to `routes-overlay.json` (slot ref
only) via `POST /v1/runtime/routes`; removing one uses
`DELETE /v1/runtime/routes/{route_id}`. The overlay rebuilds at every boot
alongside the operator's routes file, which it never touches. Always
`runtime get` after any mutation to confirm the applied state.

***

## Upgrade procedure

An upgrade is: build or pull the new binary/image, drain, stop, start on the
same state root, validate. The one non-obvious gotcha is the state-root lock.

### The flock gotcha

The daemon holds an advisory lock on `daemon.lock` for the whole time it owns a
state root. On shutdown, the OS releases the listening port slightly **before**
the process fully exits and drops the lock. In practice the state-root lock can
survive `SIGTERM` by roughly 2 seconds past the moment the port becomes free.

The failure mode: your orchestrator sees the port close, immediately starts the
replacement daemon, and the new daemon fails to acquire the lock because the old
one has not let go yet. This looks like a flaky restart.

```mermaid theme={null}
flowchart TD
    T1["t0 SIGTERM sent"]
    T2["t0 drain begins, /readyz returns 503"]
    T3["ingress tasks and MCP shutdown run"]
    T4["t+ listening PORT released<br/>orchestrator may wrongly start a new daemon here"]
    T5["t++ process exits, flock on daemon.lock RELEASED<br/>only now is start safe"]
    Gap["gap up to about 2s between port release and lock release<br/>do NOT race into it"]
    T1 --> T2 --> T3 --> T4 --> T5
    T4 -.-> Gap
    T5 -.-> Gap
```

Mitigations:

* Sequence upgrades: **stop old → confirm exit → start new.** Do not overlap.
* If your orchestrator retries on lock-acquire failure, give it a few seconds of
  backoff so the retry lands after the lock releases.
* Prefer `/readyz` and process-exit as your "safe to start replacement" signal,
  not "the port closed."

### Standard upgrade runbook

```mermaid theme={null}
flowchart TD
    U1["1. Build or pull the new binary or image. Record the version or digest."]
    U1 --> U2["2. Snapshot or archive the state root, see Backup. Optionally the workspace."]
    U2 --> U3["3. Drain: stop accepting new external ingress if you can. Let active runs settle, or accept that active shell tasks will be settled fail-closed."]
    U3 --> U4["4. Stop the old daemon. CONFIRM the process exited and the flock released."]
    U4 --> U5["5. Start the new daemon on the SAME state root, same routes file, same auth-store master key."]
    U5 --> U6["6. Validate: /readyz=200, status.health.ok, runtime get, doctor, doctor routes --check-auth, doctor routes --check-references."]
    U6 --> U7["7. Watch /v1/status warnings and /v1/logs for the first few minutes."]
```

Because the runtime-config revision stream restores the last revision on boot,
your live settings survive the upgrade. Because sessions restore from journal +
sidecars, in-flight approvals and questions survive. Because shell tasks are
settled fail-closed, you must reconcile any long-running shell work manually.

***

## Stacks as code — reproducible deployments

For reproducible daemon configuration, describe resources in a `Kheishfile.yaml`
(`KheishStack`, `apiVersion: kheish.ai/v1alpha1`) and reconcile it through the
daemon:

```bash theme={null}
kheish-daemon stack init --output-file Kheishfile.yaml
kheish-daemon stack validate --file Kheishfile.yaml
kheish-daemon stack plan --file Kheishfile.yaml
kheish-daemon stack diff --file Kheishfile.yaml
kheish-daemon stack apply --file Kheishfile.yaml
kheish-daemon stack verify --file Kheishfile.yaml
```

v1alpha1 reconciles secrets, runtime permission/debug settings, HTTP connectors,
personas, sessions, schedules, playbooks, and existence probes. Apply order is
semantic: secrets → runtime settings → personas → connectors → sessions →
playbooks → schedules → verification.

The reconciliation ledger lives at `<state_root>/kheish-apply/ledger.json` and
records ownership, desired digests, secret fingerprints, and a hash-chained local
operation history. Inspect it with:

```bash theme={null}
kheish-daemon stack plan --file Kheishfile.yaml
curl http://127.0.0.1:4000/v1/stacks/<ownership_id>/ledger
```

Production guardrails you should design around:

* **Fail-closed scopes.** Persona/session capability scopes, session credential
  scopes, and connector `session_policy` scopes are always validated fail-closed.
  A scope family needs an explicit allow-list or `*_deny: ["*"]`; partial
  deny-lists are rejected because they stay fail-open in the daemon. Wildcard
  allows require explicit `spec.apply.allow_wildcard_scopes: true`.
* **Startup-only config is plan-only.** Routes files, MCP profiles, and
  file-backed connector config are plan-only and block apply — the daemon cannot
  hot-reload or self-restart those safely. Change them at daemon startup.
* **Secrets are never read back.** With `value_env` you must pass
  `--allow-secret-env`; the ledger stores only a salted fingerprint. If a secret
  had to be preloaded before startup for MCP credential resolution, adopt it with
  `stack import --resource secret/<slot> --allow-secret-env` before the first
  apply so the reconciler does not rewrite it and disconnect already-loaded MCP
  servers.
* **Only `kind: http` connectors** are supported in v1alpha1, because HTTP
  connector live views can be compared without reading secret values.
* **Deletion is conservative.** `down`/`prune` only delete connectors, cancel
  schedules, and end sessions. Secrets, personas, playbooks, and runtime
  singletons are never hard-deleted automatically; secrets are daemon-global
  credentials and remain ledger-owned until you handle them explicitly.

The stack ledger is production ownership state for reconciliation. It is **not**
a remote notarized audit log and must not be treated as tamper-evident
compliance history. See [KheishStack](../operations/production) for the full
model.

***

## Backup and restore

Back up the **full state root as one unit**. Its consistency is a property of the
whole tree, not of individual files.

### What the state-root backup contains

Session and run journals; approvals, questions, task state, scheduler state;
channel, project, persona, learning, skill, asset, derivation, and observation
state; delivery queues and ledgers; not-yet-expired debug artifacts;
daemon-managed connector state; `auth/global-slots.json` and the provider slot
files; broker revocation and lease history; and `audit-signing.key`.

### What it does NOT contain

Files under `--workspace-root`. If workspace files are part of your recovery
objective, back up the workspace root as a **separate** artifact and verify
checksums for the specific paths your workflows depend on.

Keep these deployment artifacts recoverable alongside the backup — they are not
inside the state root:

* the routes file used at startup
* file-backed connector, hook, MCP, and skill-root config
* the exact `KHEISH_AUTH_STORE_MASTER_KEY` (or `_FILE`) value — **without it the
  auth store is unreadable**
* control-plane token files
* the daemon binary version or container image digest

### Backup mechanics

For the cleanest filesystem backup, stop or drain the daemon first. If your
platform offers atomic volume snapshots, snapshot the whole state root rather
than individual files.

```bash theme={null}
tar -C /var/lib/kheish/state     -czf kheish-state-$(date +%Y%m%dT%H%M%SZ).tar.gz .
tar -C /var/lib/kheish/workspace -czf kheish-workspace-$(date +%Y%m%dT%H%M%SZ).tar.gz .
sha256sum kheish-state-*.tar.gz kheish-workspace-*.tar.gz > kheish-backup.sha256
```

### Recovery objectives

| Objective       | Production target                                                                 |
| --------------- | --------------------------------------------------------------------------------- |
| State-root RPO  | last atomic snapshot or stopped-daemon archive                                    |
| Workspace RPO   | last workspace archive/checksum set for workflow-critical paths                   |
| RTO             | restore to an isolated daemon, pass validation, then promote traffic              |
| Drill cadence   | at least monthly, and after daemon/schema/auth-store/connector/route-file changes |
| Integrity       | record SHA-256 for every archive, stored outside the archive                      |
| Confidentiality | encrypt archives with your backup system or a reviewed envelope key               |
| Ownership       | restore with the same service user/group and restrictive token/key modes          |

### Restore into isolation first

Never restore directly onto the production listener. Restore into an isolated
daemon on loopback, validate, then promote.

```bash theme={null}
mkdir -p /var/lib/kheish/restore-state /var/lib/kheish/restore-workspace
tar -C /var/lib/kheish/restore-state     -xzf kheish-state-20260503T120000Z.tar.gz
tar -C /var/lib/kheish/restore-workspace -xzf kheish-workspace-20260503T120000Z.tar.gz
sha256sum -c kheish-backup.sha256
chown -R kheish:kheish /var/lib/kheish/restore-state /var/lib/kheish/restore-workspace

KHEISH_AUTH_STORE_MASTER_KEY_FILE=/run/secrets/kheish-auth-store-master-key \
kheish-daemon serve \
  --bind 127.0.0.1:4010 \
  --state-root /var/lib/kheish/restore-state \
  --workspace-root /var/lib/kheish/restore-workspace \
  --routes-file /etc/kheish/routes.toml \
  --http-auth-mode bearer \
  --http-admin-token-file /run/secrets/kheish-admin-token

kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token status
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token runtime get
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token \
  doctor routes --check-auth
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token \
  doctor routes --check-references
```

Promote traffic only after validation passes. There is a provider-free smoke
that exercises the whole backup/restore/rotation path end to end:

```bash theme={null}
bash scripts/e2e/ops_backup_restore_smoke.sh
```

### The master-key rule

Do **not** rotate `KHEISH_AUTH_STORE_MASTER_KEY` by simply changing the file.
Existing encrypted records become unreadable. Treat it as a root encryption key:
recover the original from your secret manager during restore, and plan any
re-encryption as a separate, verified migration. Preserve `audit-signing.key`
across backup and restore if you need continuity of the signed external-action
ledger.

***

## Failure drills

Evidence-first is the rule: `doctor`, then `status`, then route diagnostics, then
scoped run/session/task/delivery views. Never debug from assumptions about the
prompt.

### Drill: daemon will not become ready

```mermaid theme={null}
flowchart TD
    Q["/readyz stuck on 503, not draining?"]
    Q --> C1["Another daemon holds the state-root lock, flock?"]
    C1 --> R1["Check for a lingering old process, wait out the about 2s flock gap"]
    Q --> C2["runtime-config.json unreadable or route rebuild failing?"]
    C2 --> R2["doctor routes --check-auth, confirm auth-store master key present"]
    Q --> C3["Storage write-probe failing?"]
    C3 --> R3["status.storage.probes, check disk full, permissions, inode"]
    Q --> C4["Provider route not ready?"]
    C4 --> R4["status.provider_readiness, doctor routes --check-references"]
```

### Drill: runs are queuing and not starting

```mermaid theme={null}
flowchart TD
    Q["status.runs.queued climbing, oldest_queued_run_age_ms large?"]
    Q --> C1["Active run stuck waiting for approval or question?"]
    C1 --> R1["status.runs.waiting_for_*, approvals list, questions list"]
    Q --> C2["Oversized session slowing run start, old build symptom?"]
    C2 --> R2["status.storage.session_storage, plan a vacuum window"]
    Q --> C3["Provider route degraded?"]
    C3 --> R3["status.provider_readiness, doctor routes --canary"]
    Q --> C4["A stale non-terminal run holding the slot?"]
    C4 --> R4["status.runs.oldest_non_terminal_run_idle_ms, runs get the run id, runs cancel it if genuinely stuck"]
```

### Drill: output is not being delivered

```mermaid theme={null}
flowchart TD
    Q["Replies not arriving?"]
    Q --> C1["status.delivery.unresolved_dead_lettered above 0?"]
    C1 --> R1["deliveries dead-letter, inspect safe error codes, fix target, deliveries replay the id"]
    Q --> C2["Wrong reply target on the run?"]
    C2 --> R2["runs get the run_id, reply targets snapshot at run or task creation<br/>a session reply-target edit is prospective, not retroactive"]
    Q --> C3["Connector broken?"]
    C3 --> R3["connectors get kind and name, runtime connectors external metrics"]
```

### Incident containment quick reference

```bash theme={null}
kheish-daemon runtime set-permission-mode dont-ask   # block tools needing approval
kheish-daemon runtime set-debug-level off             # stop capturing before collecting evidence
kheish-daemon doctor
# then, scoped:
kheish-daemon runtime auth revoke-subject session:<compromised>
kheish-daemon runtime auth revoke-slot <auth_ref>
kheish-daemon sessions interrupt <session_id>
kheish-daemon runs cancel <run_id>
kheish-daemon tasks stop <session_id> <task_id>
```

See [Production runbooks](../operations/production) for the full incident
scenario checklists, rotation matrix, and TLS/reverse-proxy runbook.

***

## FAQ

**Do I need to back up the workspace root?**
Only if workspace files are part of your recovery objective. The state-root
backup does not include them. Back the workspace up as a separate artifact with
its own checksums.

**Can I vacuum a session while the daemon is serving?**
No. Vacuum takes the same state-root lock `serve` uses and refuses politely.
Stop the daemon, vacuum, restart. The pre-vacuum journal is preserved as
`<name>.jsonl.vacuum-bak` (full size — remove it after you verify).

**A restart killed my long shell task. Is that a bug?**
No. Active shell tasks are settled fail-closed on boot (`status: failed`,
`terminal_reason: daemon_restarted`, `recovered_on_boot: true`) so you never
mistake a dead process for a live one. Inspect `tasks output --full` and retry
manually only if the command is safe to repeat.

**Why did my new daemon fail to start right after I stopped the old one?**
The state-root flock can survive `SIGTERM` by up to \~2 seconds past the port
release. Sequence upgrades stop → confirm exit → start, or give the start a few
seconds of backoff.

**I changed the routes file but the daemon didn't pick it up.**
The running daemon uses the inventory it loaded at startup. `doctor routes` warns
`route_file_drift` when the on-disk file's SHA-256 differs from the loaded one.
Restart (or perform a controlled reload) to adopt the change.

**Can I run an older build against this state root to roll back?**
Not safely once metadata sidecars exist. The old build appends metadata inline,
the new build's sidecars win on read, and those inline writes are shadowed and
lost on the next upgrade. Stay on one build per state root.

**Where do the daemon's own logs live if I don't have shell access?**
`GET /v1/logs` — a bounded 5,000-entry in-memory ring of the daemon's filtered
tracing events, surfaced in the Kheish Air logs explorer. For durable logs,
capture the process's stderr/stdout separately.

**How large is "too large" for a session journal?**
`/v1/status` flags sessions above 100 MiB (`oversized_threshold_bytes`) with an
`oversized_session_count` and a daemon warning. Plan a vacuum window; escalate if
one keeps growing after a vacuum.

**Is `runtime-config.json` safe to hand-edit?**
Treat it as daemon-owned. Change runtime settings through `runtime ...` commands
so the change is a proper durable revision with rollback history. Hand-editing
bypasses the transaction and revision guarantees.

***

## Related reading

* [SDKs and the control-plane API](../integrate/sdks-and-api) — the integrator
  side of the same daemon.
* [Sessions and runs](../concepts/sessions-and-runs) — the execution model the
  state root persists.
* [Architecture](../concepts/architecture) — how the pieces fit together.
* [Security](./security) — trust zones, auth, and hardening.
* [Quickstart](../welcome/quickstart) — from zero to a first run.
* [Running the daemon](../operations/production),
  [Deployment and hardening](../operations/production),
  [Docker and containers](../operations/production),
  [Production runbooks](../operations/production),
  [Debugging and recovery](../operations/production),
  [Runtime configuration](../operations/production),
  [KheishStack](../operations/production) — the underlying operator references.