Running Kheish in production

This page is the operator’s reference for running the Kheish daemon as durable infrastructure. It goes past “how do I start it” (that lives in Running the daemon) and into the questions that matter once a daemon is carrying real traffic: what actually lives on disk, how the daemon comes back after a restart, how to keep session journals from growing without bound, how to lay the daemon out on a host, what to watch, how to upgrade without corrupting state, and how to prove your backups work before you need them. The single most important sentence to internalize: the daemon is the durable orchestration engine, and the state root is the durable part. Everything an integrator or SDK does eventually resolves to state written under one directory tree. If you understand that tree, you understand the daemon’s failure modes, its backup boundary, and its upgrade contract. The rest of this page is built around it. If you are integrating an application on top of the daemon rather than running it, read SDKs and the control-plane API instead — this page assumes you own the process.

The two roots

A production daemon is defined by two filesystem roots and one network bind:

--state-root — durable daemon-owned state. Sessions, runs, learnings, connectors, encrypted secrets, schedules, deliveries, audit trails. This is the thing you back up. Losing it loses the daemon’s memory.
--workspace-root — the filesystem the agent’s shell and file tools operate in. This is the blast radius of tool execution. It is not backed up by a state-root snapshot and is not part of the daemon’s recovery guarantee.
--bind — the control-plane listen address, 127.0.0.1:4000 by default.

Three rules follow directly and are enforced by the official container entrypoint (it refuses to start otherwise):

state_root and workspace_root must be different directories.
Both must be absolute paths in a container context.
Point the workspace root at something the agent is allowed to mutate. A source checkout or a broad home directory expands the blast radius of every file and shell tool the agent can call.

Always be explicit about which state root you are talking to. A large fraction of “the daemon is behaving strangely” incidents are actually “I am talking to a different daemon, or reusing a state root from an older build.”

State-root anatomy

The state root is a flat directory of daemon-owned stores. Some are single JSON documents, some are append-only journals, some are directory trees of per-entity files. Here is the layout you will see under a busy production root, annotated with what each entry is for and whether it is hot (rewritten constantly), append-only (grows), or cold (rarely touched).

<state_root>/
├── daemon.lock                     advisory flock; owner of this state root
├── daemon-index.json               daemon-level index of known sessions/entities
│
├── sessions/                       ← the heart of durable state
│   └── __safe/                     safe-encoded namespace for session names
│       ├── <name>.jsonl            append-only session journal (events, checkpoints,
│       │                           permission audits, outputs)
│       ├── <name>.jsonl.vacuum-bak pre-vacuum backup, FULL SIZE, remove after verify
│       └── <name>.meta/            per-key metadata sidecars (last-wins state)
│           ├── goal.json           one small atomically-replaced file per key
│           ├── persona.json        persona snapshot bound to the session
│           ├── capability_scope.json
│           ├── credential_scope.json
│           ├── reply_targets.json
│           └── ...                 (task archive lives alongside as needed)
│
├── run-memories/                   recovered run-memory files (crash-resume context)
├── learnings/                      durable published learnings (verified/provisional)
├── learning-candidates/            captured, not-yet-published learning candidates
├── learning-policy.json            LEGACY migration input only (policy now in runtime-config)
├── skills/                         promoted procedural skills
│
├── runtime-config.json             live runtime revision stream (route/model, permission
│                                   mode, hooks, debug level, tool limits, learning policy)
├── route-inventory.json            non-secret route metadata + loaded routes-file SHA-256
├── routes-overlay.json             runtime-added model routes (no keys; slot refs only)
├── mcp-overlay.json                runtime-added MCP server overlay
│
├── auth/                           encrypted daemon-owned secret store
│   ├── global-slots.json           generic secret slots (auth_ref targets, connector creds)
│   ├── openai-slots.json           provider-scoped slots
│   ├── anthropic-slots.json
│   └── ...                         broker lease + revocation history
├── auth-store-master.key           present only if the daemon generated its own key
├── audit-signing.key               Ed25519 key for signed external-action audit
├── external-actions/               signed per-run external-action audit ledgers
│
├── control-plane-auth/
│   └── audit.jsonl                 auth_failure / auth_rate_limited / cors_origin_rejected
│
├── connector-ingress/              connector ingress bookkeeping
├── runtime-connectors.json         daemon-managed connector definitions (redacted on read)
├── daemon-deliveries.json          delivery queue + delivery ledgers (DLQ lives here)
├── slack-delivery-progress.d/      per-delivery progress markers
├── telegram-delivery-progress.d/
│
├── channels/                       public channel records, messages, leases, thread-work
├── projects/                       daemon-owned projects, members, linked channels, tasks
├── personas/                       daemon-managed persona records + compact persona index
├── assets/                         daemon-owned assets, derived text, preview images
├── derivations/                    durable derived artifacts
├── observations/                   observation sources + captured records
├── schedules/                      durable schedules and wakeups
├── flows.json                      Flow projections over runs
├── outputs/                        persisted output artifacts
├── events/                         event-stream bookkeeping
│
├── hooks.json                      daemon-wide hook config (redacted in summaries)
├── hooks-worktree/                 hook execution worktree
├── hook-dlq/                       hook dead-letter records
│
├── kheish-apply/
│   └── ledger.json                 KheishStack reconciliation ledger (ownership, digests)
│
├── capture-agents/                 provisioned host-local capture agent leases
├── spawn-policy-ledger.json        subagent spawn-policy quota ledger
├── daemon-supervisor.json          supervisor state
├── daemon-supervisor-audit.jsonl   supervisor audit
└── debug/                          debug-capture artifacts (TTL-cleaned)

You do not have to memorize every entry. The ones that change how you operate:

`sessions/` — journals, sidecars, and task archives

Each session has three physical footprints, and /v1/status reports all three separately under .storage.session_storage:

The journal (<name>.jsonl) is append-only. It holds ordered events, compaction checkpoints, permission audits, and output records. It only ever grows during normal operation; it is compacted offline by sessions vacuum.
The metadata sidecars (<name>.meta/*.json) hold last-wins control state: the session goal, the bound persona snapshot, the persisted capability and credential scopes, the reply-target defaults, and so on. Each key is its own atomically-replaced file. A metadata write replaces one small file; it does not append to the journal. This is the single most important performance property of current builds — see performance notes.
The terminal-task archive holds settled background-task records.

The __safe/ layer is a name-encoding namespace so arbitrary session names map to safe on-disk filenames. You will not usually touch it by hand.

Secrets, auth, and audit

auth/ is the encrypted secret store. Route API keys, connector shared tokens, MCP bearer secrets, and broker lease history all live here, encrypted with KHEISH_AUTH_STORE_MASTER_KEY (or the _FILE variant). The master key is not stored in a recoverable form inside the state root unless the daemon generated its own key into auth-store-master.key. If you supply the key via env or file, you must recover the same value at restore time or the store is unreadable. audit-signing.key is the Ed25519 key that signs the per-run external-action audit ledgers under external-actions/. Keep it with the state root across backup and restore, or the daemon can read old signed records but cannot continue the ledger safely. control-plane-auth/audit.jsonl is your security tripwire. It records auth_failure, auth_rate_limited, cors_origin_rejected, and cors_origin_rate_limited. Ship it to your SIEM.

Runtime configuration and routes

runtime-config.json is the durable revision stream for everything you can change live: the daemon default route/model, permission mode, learning automation policy, recovered run-memory policy, tool runtime limits, system prompt, hooks, and debug level. It is append-a-revision, and the last revision is restored on boot before the daemon accepts work. learning-policy.json is legacy: it is read only as migration input if present. route-inventory.json records the canonical path and SHA-256 of the routes file loaded at startup. doctor routes compares that hash against the file on disk and warns route_file_drift if someone edited the file without a controlled reload. routes-overlay.json holds routes you added at runtime via the API; crucially it records only slot references, never keys — the keys stay in the encrypted auth/ store under routes.<route_id>.api_key.

Deliveries and the dead-letter queue

daemon-deliveries.json is the output delivery queue and ledger. When a connector-backed reply cannot be delivered after retries, it is dead-lettered here. /v1/status .delivery.unresolved_dead_lettered is the count that drives the delivery health warning, and POST /v1/deliveries/{id}/replay replays one.

The apply ledger

kheish-apply/ledger.json is the KheishStack reconciliation ledger: ownership, desired digests, secret fingerprints, and a hash-chained operation history. It is daemon-owned. Do not pass a separate --state-root to stack commands; the daemon owns the ledger under the root it was started with.

Boot and recovery sequence

Understanding boot order tells you what is safe to expect after a restart and what is deliberately settled fail-closed. The daemon does not “resume the world exactly as it was” — it restores durable, replayable state and settles anything that cannot be safely replayed. What Kheish does restore across a restart: pending approvals, pending structured questions, active inline skills, schedules, queued deliveries, bound persona snapshots, session capability/credential scopes, reply-target defaults, channel state (records, public messages, leases, stimuli, thread-work). What Kheish does not promise: that an arbitrary shell process survives a restart. It doesn’t. Active shell tasks are settled fail-closed on boot so you never mistake a killed process for a running one. After a restart, inspect:

kheish-daemon tasks list <session_id>          # look for status: "failed"
kheish-daemon tasks get <session_id> <task_id> # terminal_reason: daemon_restarted
kheish-daemon tasks output <session_id> <task_id> --full

Retry a failed shell task manually only after deciding it is safe to repeat. task_output.retrieval_status=success means “the daemon read the persisted output,” not “the shell command succeeded.”

Performance notes

Two design properties keep a busy daemon fast. Both are worth understanding because they explain why old builds got slow and current builds do not.

Metadata sidecars make writes O(value), not O(journal)

Session metadata (control state, persona binding, operator config, and the rest of the last-wins keys) lives in per-key sidecar files under sessions/__safe/<name>.meta/. A metadata rewrite replaces one small file atomically. It does not append a full control-state snapshot to the journal. Older builds appended a full control-state snapshot per task mutation. Only the last snapshot per key ever mattered, so a long-running autonomous session could accumulate gigabytes of superseded snapshots and slow every run start — the daemon had to parse the whole journal before it could begin. The sidecar model removes that failure class for new writes. A single metadata value larger than 1 MiB logs a write-time warning naming the session and key, so abnormal growth of one hot key surfaces long before the journal becomes a problem.

Lazy journal parsing, scan-miss cache, and tail reads

Journals are parsed lazily. The daemon does not eagerly parse every session file on every operation:

Tail reads: when persisting incremental records, the daemon reads only the file tail rather than re-parsing the whole journal. Metadata records in the tail are filtered out so they cannot break incremental replay.
Growing tail windows: when it needs to find a specific record, the daemon walks growing tail windows starting on line boundaries instead of parsing from byte zero.
Scan-miss cache: when a metadata key is genuinely absent, the daemon remembers that miss so it does not rescan the journal for the same key repeatedly. A subsequent sidecar write for that key invalidates the cached miss.
Torn-tail tolerance: the parser tolerates exactly one torn line at the tail (a crash mid-append), and lossy UTF-8 decoding of that torn tail is safe because torn-tail tolerance already handles a split multi-byte character.

The practical operator takeaway: journal reads stay cheap even as journals grow, but journals still grow. That is what sessions vacuum is for.

Journal maintenance and offline vacuum

The append-only journal grows. On modern builds it grows much more slowly than before (metadata no longer appends), but a very long-running session still accumulates events, checkpoints, audits, and outputs. /v1/status will tell you when one has grown too large.

Detecting oversized sessions

/v1/status .storage.session_storage reports, per daemon:

session_count and total_bytes across journals, metadata sidecars, and task archives
largest_session_id / largest_session_bytes
oversized_session_count and a bounded oversized_session_ids sample for any session above oversized_threshold_bytes (100 MiB)

Any oversized session also logs a daemon warning naming the session and suggesting the vacuum command. So you get both a pull signal (poll /v1/status) and a push signal (the daemon log).

The vacuum command and its safety contract

# The daemon MUST be stopped for this session's state root.
# vacuum takes the state-root lock and refuses politely while a daemon serves it.
kheish-daemon sessions vacuum <session-id> --state-root <state-root>

The command prints a JSON report:

bytes_before / bytes_after
kept_by_type counts per record type
metadata_keys_kept
metadata_records_dropped
torn_tail_dropped
backup_path

The safety contract, verified end to end in the daemon’s own tests:

It takes the same state-root lock serve uses. If a daemon is serving that root, vacuum refuses rather than corrupting a live journal.
It keeps events, checkpoints, audits, and outputs in order. Only superseded metadata snapshots are dropped. The loaded session state is byte-for-byte equivalent after compaction.
The rewrite is verified against the original before an atomic swap.
The pre-vacuum journal is preserved next to the compacted one as <name>.jsonl.vacuum-bak. This backup is full size. Remove it before vacuuming the same session again, or you keep paying for the old bytes.

Field reference: a production session whose journal had grown to 1.4 GB of superseded control-state snapshots (an older build) compacted to 12 MB in about 11 seconds, and its runs went from stalling for hours before the first tool call to executing normally after restart.

Vacuum runbook

The downgrade non-guarantee

Running an older daemon build against a state root that already has metadata sidecars is not supported. The old build appends metadata inline; the newer build’s sidecars keep precedence on read, so those inline writes are shadowed and effectively lost on the next upgrade. Stay on one build per state root. If you must downgrade, accept that metadata writes made by the old build are lost when you go forward again.

Deployment topologies

Three shapes cover almost every production deployment. Pick the simplest one that satisfies your exposure requirement.

Topology A — single host, loopback only

The default and the safest. The daemon binds loopback, nothing external reaches it, and operators use the CLI or SDK from the same host (or through an SSH tunnel). No TLS, no CORS surface, minimal attack surface. Use this for automation hosts, CI runners, and any daemon that only ever talks to same-host clients. Bearer auth is optional here but still recommended.

Topology B — reverse-proxied with TLS

The daemon stays on loopback (or a private service network); a dedicated reverse proxy or ingress controller terminates TLS and forwards to it. Bearer auth stays enabled behind the proxy. This is the standard shape for exposing a daemon to a team or an application backend. Non-negotiables for the proxy layer, all validated by the shipped Nginx fixture smoke (scripts/e2e/ops_reverse_proxy_config_smoke.sh):

forward Authorization unchanged; keep daemon bearer auth on
disable buffering for /v1/events/stream, /v1/sessions/*/stream, /v1/runs/*/stream, and /v1/flows/*/stream (SSE breaks under buffering)
keep CORS out of the proxy layer; Kheish’s own CORS allowlist is intentionally loopback-only, and browsers should be served same-origin
preserve request bodies for connector ingress; do not strip signature headers
set body-size limits deliberately for your attachment/debug/output workloads
set long idle timeouts so SSE connections survive

Validate after deploy:

kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor
kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor --cors-origin https://kheish.example.com
kheish-daemon --base-url https://kheish.example.com \
  --token-file /run/secrets/kheish-admin-token doctor routes --check-auth

doctor fails closed on broken /readyz, malformed or buffered SSE, storage write-probe failures, missing state-root lock ownership, unsafe CORS preflight responses, route/provider readiness errors, and invalid hook definitions. It also checks configured HTTP hook hostnames against DNS and warns before a hook dispatch surprises you with a private/local address resolution.

Topology C — container: daemon plus remote connectors

The recommended container shape is one daemon container plus separate connector services talking to it over remote_http. The official daemon image is deliberately not the home for every platform-specific sidecar runtime. Container defaults baked into the official image:

KHEISH_BIND=0.0.0.0:4000
KHEISH_STATE_ROOT=/var/lib/kheish/state
KHEISH_WORKSPACE_ROOT=/workspace
KHEISH_HTTP_AUTH_MODE=bearer
KHEISH_MCP_DISCOVERY=disabled (so a container never silently imports developer-local MCP config from $HOME/.codex)

The image ships the kheish-daemon binary plus bash, ripgrep, procps, git, ca-certificates, and tini, runs as non-root, and treats SIGTERM and SIGINT as graceful shutdown. It does not bundle arbitrary Node/Python runtimes for external MCP servers or child_process connectors. If your MCP config needs those, build a custom image or run them as separate services. Prefer file-backed secrets in containers:

KHEISH_DAEMON_ADMIN_TOKEN_FILE
KHEISH_DAEMON_READONLY_TOKEN_FILE
KHEISH_AUTH_STORE_MASTER_KEY_FILE

The entrypoint refuses obvious misconfiguration before the daemon starts: relative state_root/workspace_root, identical roots, non-loopback bind without bearer auth, unreadable token/key files, and conflicting inline-plus-file secret inputs for the same setting. See Docker and containers for the full Compose walkthrough.

Observability surfaces

The daemon exposes several distinct observability surfaces. Knowing which is which keeps you from, for example, using an authenticated endpoint for a liveness probe.

Unauthenticated probes (outside auth middleware): GET /healthz liveness; GET /readyz readiness, 503 while draining on shutdown. Use these for Docker/K8s probes. NEVER reuse /v1/* for probes.
Authenticated operator snapshot: GET /v1/status returns health.ok, storage.ok + probes[], storage.session_storage (sizes, oversized), provider_readiness, runs.{queued,running,waiting_*}, delivery.unresolved_dead_lettered, runtime.debug_level, health.warnings[].
Daemon log ring buffer: GET /v1/logs?limit=N returns the last N of a bounded 5000-entry in-memory ring of the daemon’s own tracing events (scheduler, deliveries, MCP, errors) — no shell access needed.
Delivery / DLQ: GET /v1/deliveries[?status=dead_lettered], GET /v1/deliveries/dead-letter, POST /v1/deliveries/{id}/replay, GET /v1/runtime/connectors/external/metrics.
Live streams (SSE): GET /v1/events/stream (global, filterable by session_id/run_id), GET /v1/sessions/{id}/stream, GET /v1/runs/{id}/stream, GET /v1/flows/{id}/stream (proxy over the root run stream).
Security audit (on disk): <state_root>/control-plane-auth/audit.jsonl, <state_root>/external-actions/ (signed per-run external audit).

`/v1/status` is the operator snapshot

kheish-daemon status fetches it. It is the one call to make first in almost any investigation. It reports readiness, active runtime route, run queues and their ages, schedules, agents, mailboxes, storage write health, provider readiness, per-session storage footprint, and live background shell tasks.

`/v1/logs` is a bounded ring buffer

The daemon mirrors its own filtered tracing events into an in-memory ring buffer of 5,000 entries. GET /v1/logs?limit=N returns up to N most recent entries, oldest first (default 1,000). Each entry carries timestamp_ms, level, target (the emitting module), message, and remaining fields. Two properties matter operationally:

It is bounded — it can never grow the daemon’s memory footprint, and old entries fall off the front. It is a live tail, not a durable log store. If you need durable logs, capture the process’s stderr/stdout separately.
It only retains what the active tracing filter already lets through. If you filtered a target out at the tracing layer, it will not appear here.

This is what powers the daemon logs explorer in Kheish Air: operators get a live, filterable view of what the whole daemon is doing — scheduler ticks, delivery attempts, MCP child-process events, errors — without SSH access to the process log file.

Delivery dead letters and replay

Connector-backed output that cannot be delivered after retries is dead-lettered. /v1/status .delivery.unresolved_dead_lettered counts DLQ entries that do not yet have a completed replay — that is the count the delivery health warning uses. status.delivery.dead_lettered is historical (lifetime count). To recover:

kheish-daemon deliveries dead-letter                      # inspect the DLQ
kheish-daemon deliveries list --run-id <run_id>           # scope to one run
kheish-daemon deliveries replay <delivery_id>             # create a new pending replay

Replays are idempotent by default: a replay creates a new pending delivery with a new delivery_id and replayed_from_delivery_id pointing at the original, which stays as audit history. A later non-forced replay returns the existing pending/delivered/dead-lettered replay. Use force=true only for an intentional second attempt.

Health warnings

/v1/status .health.warnings[] is a structured list of the daemon’s own concerns. Codes you will see include queued_run_lag, stale-run codes, oversized-session warnings, provider-readiness warnings, and delivery warnings. Treat a non-empty warnings array as “read this before it becomes a page.”

SLO signal thresholds

Starting thresholds for production-like deployments:

Signal	JSON path or probe	Warning	Page
Readiness	`GET /readyz`	one non-2xx sample	two consecutive non-2xx within 60s
Health	`/v1/status` `.health.ok`	`false` once	`false` twice consecutively
Storage	`.storage.ok`, `.storage.probes[]`	any warning diagnostic	`false` immediately
Session growth	`.storage.session_storage.oversized_session_count`, `.largest_session_bytes`	any session above 100 MiB → plan a vacuum	oversized session still growing after a vacuum
Provider readiness	`.provider_readiness.active_route_ready`, `.routes[]`	default route warning 5 min	default route error 5 min
Run queue lag	`.runs.queued`, `.runs.oldest_queued_run_age_ms`	older than 30 min behind	older than 2 h behind
Stale active runs	`.runs.running`, `.runs.oldest_non_terminal_run_idle_ms`	older than 2 h without events	older than 6 h without events
Pending approvals/questions	`.runs.waiting_for_approval`, `.runs.waiting_for_user_question`	older than 4 h	older than 24 h
Delivery DLQ	`.delivery.unresolved_dead_lettered`	any growth in staging	any production DLQ item
Auth/CORS limits	`control-plane-auth/audit.jsonl` `auth_rate_limited`, `cors_origin_rate_limited`	repeated within 10 min	sustained 30 min
Debug capture	`.runtime.debug_level`	`full` outside a ticketed window	`full` for more than 30 min
State-root disk	host FS metrics for `--state-root`	above 80%	above 90% or inode exhaustion

Run the CI-ready probe smoke to exercise these documented paths:

KHEISH_OPS_SLO_MODEL=gpt-5.4 bash scripts/e2e/ops_slo_probe_smoke.sh

Live runtime configuration

Several things change without a restart, serialized through the durable runtime-config.json revision stream:

kheish-daemon runtime get                                   # current settings + revision
kheish-daemon runtime set-model openai/gpt-5.4              # daemon default route/model
kheish-daemon runtime set-permission-mode accept-edits
kheish-daemon runtime set-debug-level redacted
kheish-daemon runtime tool-limits set --file tool-limits.json
kheish-daemon runtime learning-policy set --file learning-policy.json
kheish-daemon runtime hooks get
kheish-daemon runtime revisions
kheish-daemon runtime rollback --target-revision 7 --expected-revision 12

Operational semantics that trip people up:

These are daemon-wide and affect future execution. They do not retroactively change the pinned route, model, debug level, or tool-limit snapshot of an already-active tool batch.
runtime set-model on a mixed-provider daemon changes the fallback route, not every session at once.
Mutations can include expected_revision; a stale writer gets 409 with runtime/runtime_revision_conflict. Use this when automating changes.
Rollback appends a new revision recording the restored target; it does not delete history. Only revisions still returned by runtime revisions are valid rollback targets (older history is pruned to config.history_limit).

Adding a model route at runtime persists to routes-overlay.json (slot ref only) via POST /v1/runtime/routes; removing one uses DELETE /v1/runtime/routes/{route_id}. The overlay rebuilds at every boot alongside the operator’s routes file, which it never touches. Always runtime get after any mutation to confirm the applied state.

Upgrade procedure

An upgrade is: build or pull the new binary/image, drain, stop, start on the same state root, validate. The one non-obvious gotcha is the state-root lock.

The flock gotcha

The daemon holds an advisory lock on daemon.lock for the whole time it owns a state root. On shutdown, the OS releases the listening port slightly before the process fully exits and drops the lock. In practice the state-root lock can survive SIGTERM by roughly 2 seconds past the moment the port becomes free. The failure mode: your orchestrator sees the port close, immediately starts the replacement daemon, and the new daemon fails to acquire the lock because the old one has not let go yet. This looks like a flaky restart. Mitigations:

Sequence upgrades: stop old → confirm exit → start new. Do not overlap.
If your orchestrator retries on lock-acquire failure, give it a few seconds of backoff so the retry lands after the lock releases.
Prefer /readyz and process-exit as your “safe to start replacement” signal, not “the port closed.”

Standard upgrade runbook

Because the runtime-config revision stream restores the last revision on boot, your live settings survive the upgrade. Because sessions restore from journal + sidecars, in-flight approvals and questions survive. Because shell tasks are settled fail-closed, you must reconcile any long-running shell work manually.

Stacks as code — reproducible deployments

For reproducible daemon configuration, describe resources in a Kheishfile.yaml (KheishStack, apiVersion: kheish.ai/v1alpha1) and reconcile it through the daemon:

kheish-daemon stack init --output-file Kheishfile.yaml
kheish-daemon stack validate --file Kheishfile.yaml
kheish-daemon stack plan --file Kheishfile.yaml
kheish-daemon stack diff --file Kheishfile.yaml
kheish-daemon stack apply --file Kheishfile.yaml
kheish-daemon stack verify --file Kheishfile.yaml

v1alpha1 reconciles secrets, runtime permission/debug settings, HTTP connectors, personas, sessions, schedules, playbooks, and existence probes. Apply order is semantic: secrets → runtime settings → personas → connectors → sessions → playbooks → schedules → verification. The reconciliation ledger lives at <state_root>/kheish-apply/ledger.json and records ownership, desired digests, secret fingerprints, and a hash-chained local operation history. Inspect it with:

kheish-daemon stack plan --file Kheishfile.yaml
curl http://127.0.0.1:4000/v1/stacks/<ownership_id>/ledger

Production guardrails you should design around:

Fail-closed scopes. Persona/session capability scopes, session credential scopes, and connector session_policy scopes are always validated fail-closed. A scope family needs an explicit allow-list or *_deny: ["*"]; partial deny-lists are rejected because they stay fail-open in the daemon. Wildcard allows require explicit spec.apply.allow_wildcard_scopes: true.
Startup-only config is plan-only. Routes files, MCP profiles, and file-backed connector config are plan-only and block apply — the daemon cannot hot-reload or self-restart those safely. Change them at daemon startup.
Secrets are never read back. With value_env you must pass --allow-secret-env; the ledger stores only a salted fingerprint. If a secret had to be preloaded before startup for MCP credential resolution, adopt it with stack import --resource secret/<slot> --allow-secret-env before the first apply so the reconciler does not rewrite it and disconnect already-loaded MCP servers.
Only kind: http connectors are supported in v1alpha1, because HTTP connector live views can be compared without reading secret values.
Deletion is conservative. down/prune only delete connectors, cancel schedules, and end sessions. Secrets, personas, playbooks, and runtime singletons are never hard-deleted automatically; secrets are daemon-global credentials and remain ledger-owned until you handle them explicitly.

The stack ledger is production ownership state for reconciliation. It is not a remote notarized audit log and must not be treated as tamper-evident compliance history. See KheishStack for the full model.

Backup and restore

Back up the full state root as one unit. Its consistency is a property of the whole tree, not of individual files.

What the state-root backup contains

Session and run journals; approvals, questions, task state, scheduler state; channel, project, persona, learning, skill, asset, derivation, and observation state; delivery queues and ledgers; not-yet-expired debug artifacts; daemon-managed connector state; auth/global-slots.json and the provider slot files; broker revocation and lease history; and audit-signing.key.

What it does NOT contain

Files under --workspace-root. If workspace files are part of your recovery objective, back up the workspace root as a separate artifact and verify checksums for the specific paths your workflows depend on. Keep these deployment artifacts recoverable alongside the backup — they are not inside the state root:

the routes file used at startup
file-backed connector, hook, MCP, and skill-root config
the exact KHEISH_AUTH_STORE_MASTER_KEY (or _FILE) value — without it the auth store is unreadable
control-plane token files
the daemon binary version or container image digest

Backup mechanics

For the cleanest filesystem backup, stop or drain the daemon first. If your platform offers atomic volume snapshots, snapshot the whole state root rather than individual files.

tar -C /var/lib/kheish/state     -czf kheish-state-$(date +%Y%m%dT%H%M%SZ).tar.gz .
tar -C /var/lib/kheish/workspace -czf kheish-workspace-$(date +%Y%m%dT%H%M%SZ).tar.gz .
sha256sum kheish-state-*.tar.gz kheish-workspace-*.tar.gz > kheish-backup.sha256

Recovery objectives

Objective	Production target
State-root RPO	last atomic snapshot or stopped-daemon archive
Workspace RPO	last workspace archive/checksum set for workflow-critical paths
RTO	restore to an isolated daemon, pass validation, then promote traffic
Drill cadence	at least monthly, and after daemon/schema/auth-store/connector/route-file changes
Integrity	record SHA-256 for every archive, stored outside the archive
Confidentiality	encrypt archives with your backup system or a reviewed envelope key
Ownership	restore with the same service user/group and restrictive token/key modes

Restore into isolation first

Never restore directly onto the production listener. Restore into an isolated daemon on loopback, validate, then promote.

mkdir -p /var/lib/kheish/restore-state /var/lib/kheish/restore-workspace
tar -C /var/lib/kheish/restore-state     -xzf kheish-state-20260503T120000Z.tar.gz
tar -C /var/lib/kheish/restore-workspace -xzf kheish-workspace-20260503T120000Z.tar.gz
sha256sum -c kheish-backup.sha256
chown -R kheish:kheish /var/lib/kheish/restore-state /var/lib/kheish/restore-workspace

KHEISH_AUTH_STORE_MASTER_KEY_FILE=/run/secrets/kheish-auth-store-master-key \
kheish-daemon serve \
  --bind 127.0.0.1:4010 \
  --state-root /var/lib/kheish/restore-state \
  --workspace-root /var/lib/kheish/restore-workspace \
  --routes-file /etc/kheish/routes.toml \
  --http-auth-mode bearer \
  --http-admin-token-file /run/secrets/kheish-admin-token

kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token status
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token runtime get
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token \
  doctor routes --check-auth
kheish-daemon --base-url http://127.0.0.1:4010 --token-file /run/secrets/kheish-admin-token \
  doctor routes --check-references

Promote traffic only after validation passes. There is a provider-free smoke that exercises the whole backup/restore/rotation path end to end:

bash scripts/e2e/ops_backup_restore_smoke.sh

The master-key rule

Do not rotate KHEISH_AUTH_STORE_MASTER_KEY by simply changing the file. Existing encrypted records become unreadable. Treat it as a root encryption key: recover the original from your secret manager during restore, and plan any re-encryption as a separate, verified migration. Preserve audit-signing.key across backup and restore if you need continuity of the signed external-action ledger.

Failure drills

Evidence-first is the rule: doctor, then status, then route diagnostics, then scoped run/session/task/delivery views. Never debug from assumptions about the prompt.

Drill: daemon will not become ready

Drill: runs are queuing and not starting

Drill: output is not being delivered

Incident containment quick reference

kheish-daemon runtime set-permission-mode dont-ask   # block tools needing approval
kheish-daemon runtime set-debug-level off             # stop capturing before collecting evidence
kheish-daemon doctor
# then, scoped:
kheish-daemon runtime auth revoke-subject session:<compromised>
kheish-daemon runtime auth revoke-slot <auth_ref>
kheish-daemon sessions interrupt <session_id>
kheish-daemon runs cancel <run_id>
kheish-daemon tasks stop <session_id> <task_id>

See Production runbooks for the full incident scenario checklists, rotation matrix, and TLS/reverse-proxy runbook.

FAQ

Do I need to back up the workspace root? Only if workspace files are part of your recovery objective. The state-root backup does not include them. Back the workspace up as a separate artifact with its own checksums. Can I vacuum a session while the daemon is serving? No. Vacuum takes the same state-root lock serve uses and refuses politely. Stop the daemon, vacuum, restart. The pre-vacuum journal is preserved as <name>.jsonl.vacuum-bak (full size — remove it after you verify). A restart killed my long shell task. Is that a bug? No. Active shell tasks are settled fail-closed on boot (status: failed, terminal_reason: daemon_restarted, recovered_on_boot: true) so you never mistake a dead process for a live one. Inspect tasks output --full and retry manually only if the command is safe to repeat. Why did my new daemon fail to start right after I stopped the old one? The state-root flock can survive SIGTERM by up to ~2 seconds past the port release. Sequence upgrades stop → confirm exit → start, or give the start a few seconds of backoff. I changed the routes file but the daemon didn’t pick it up. The running daemon uses the inventory it loaded at startup. doctor routes warns route_file_drift when the on-disk file’s SHA-256 differs from the loaded one. Restart (or perform a controlled reload) to adopt the change. Can I run an older build against this state root to roll back? Not safely once metadata sidecars exist. The old build appends metadata inline, the new build’s sidecars win on read, and those inline writes are shadowed and lost on the next upgrade. Stay on one build per state root. Where do the daemon’s own logs live if I don’t have shell access? GET /v1/logs — a bounded 5,000-entry in-memory ring of the daemon’s filtered tracing events, surfaced in the Kheish Air logs explorer. For durable logs, capture the process’s stderr/stdout separately. How large is “too large” for a session journal? /v1/status flags sessions above 100 MiB (oversized_threshold_bytes) with an oversized_session_count and a daemon warning. Plan a vacuum window; escalate if one keeps growing after a vacuum. Is runtime-config.json safe to hand-edit? Treat it as daemon-owned. Change runtime settings through runtime ... commands so the change is a proper durable revision with rollback history. Hand-editing bypasses the transaction and revision guarantees.

SDKs and the control-plane API — the integrator side of the same daemon.
Sessions and runs — the execution model the state root persists.
Architecture — how the pieces fit together.
Security — trust zones, auth, and hardening.
Quickstart — from zero to a first run.
Running the daemon, Deployment and hardening, Docker and containers, Production runbooks, Debugging and recovery, Runtime configuration, KheishStack — the underlying operator references.

​Running Kheish in production

​The two roots

​State-root anatomy

​sessions/ — journals, sidecars, and task archives

​Secrets, auth, and audit

​Runtime configuration and routes

​Deliveries and the dead-letter queue

​The apply ledger

​Boot and recovery sequence

​Performance notes

​Metadata sidecars make writes O(value), not O(journal)

​Lazy journal parsing, scan-miss cache, and tail reads

​Journal maintenance and offline vacuum

​Detecting oversized sessions

​The vacuum command and its safety contract

​Vacuum runbook

​The downgrade non-guarantee

​Deployment topologies

​Topology A — single host, loopback only

​Topology B — reverse-proxied with TLS

​Topology C — container: daemon plus remote connectors

​Observability surfaces

​/v1/status is the operator snapshot

​/v1/logs is a bounded ring buffer

​Delivery dead letters and replay

​Health warnings

​SLO signal thresholds

​Live runtime configuration

​Upgrade procedure

​The flock gotcha

​Standard upgrade runbook

​Stacks as code — reproducible deployments

​Backup and restore

​What the state-root backup contains

​What it does NOT contain

​Backup mechanics

​Recovery objectives

​Restore into isolation first

​The master-key rule

​Failure drills

​Drill: daemon will not become ready

​Drill: runs are queuing and not starting

​Drill: output is not being delivered

​Incident containment quick reference

​FAQ

​Related reading