> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/control-plane-observability.md). # Control-plane observability Use this runbook when the terminal's `24H CONTROL PLANE` panel is unavailable, a protected request reports `unknown`, a signer event has `LEGACY` instead of a trace, or `/api/health` reports degraded control-plane observability. Keep execution disabled or the venue kill switch active while an order state is ambiguous. ## Normal evidence path 1. Confirm `pnpm local:health` reports the loopback production runtime healthy. 2. Unlock the local operator session and open Terminal settings. 3. Inspect `24H CONTROL PLANE`: retained count and retention, 24-hour outcome totals, p95 boundary time, selected-venue feed age, Robinhood Chain divergence, and the newest five traces. 4. Match the `FLOW / SPAN` pair beside an Arcus or Hyperliquid signer-audit row to the recent control-plane row. A flow may repeat across review, submit, cancel and reconciliation; every span must remain unique. Treat both as correlation only and reconcile the client ID or CLOID against venue state for finality. 5. Confirm the separate proposal lifecycle summary reports observed/valid/reserved/consumed/invalid counts. Use the proposal lifecycle runbook for record-level operations; trace telemetry never establishes proposal or order finality. The authenticated endpoint is `GET /api/observability`. It is private, no-store, session-rate-limited, and never accepts an account or trace query. The public `/api/health` response exposes only aggregate store/delivery state, retention, counts, and reason codes. The private view also shows rolling SLO sample state, fixed-code active alerts and recent lifecycle incidents. ## Unknown or unavailable outcome 1. Do not retry a submit or cancel merely because its HTTP outcome is unknown. 2. Record the response trace and the reviewed client ID or CLOID. 3. Inspect the signer audit under the exact account scope. A `LEGACY` row is valid historical evidence created before correlation was released; do not invent a trace for it. 4. Use the authenticated reconciliation control to establish the exact venue lifecycle state. 5. Follow the Arcus reconciliation incident or execution audit runbook when the signer and venue evidence disagree. Rejected counts include unauthenticated, expired-session, invalid-body, policy-denied, and semantic rejected-review responses. They do not mean a venue rejection unless the signer/venue lifecycle says so. ## Degraded or corrupt store The default store is `%USERPROFILE%\.nightshift\data\control-plane-observability-v1.json` on Windows or `~/.nightshift/data/control-plane-observability-v1.json` elsewhere. The stable filename contains schema v3; schema-v1/v2 events migrate by assigning each legacy span as its own root, while v2 alert/incident/delivery state is preserved. `NIGHTSHIFT_OBSERVABILITY_STORE_PATH` may override it only with an absolute path. Retention and cap use `NIGHTSHIFT_OBSERVABILITY_RETENTION_DAYS` (1-365, default 30) and `NIGHTSHIFT_OBSERVABILITY_MAX_EVENTS` (100-10,000, default 5,000). 1. Stop the supervised runtime before inspecting or replacing the file. 2. Preserve the file if it is malformed, oversized, not a regular file, or schema-invalid. Do not hand-edit event fields. 3. Verify the parent directory is private and writable and the configured path is absolute. 4. Restore a previously verified public-store backup only while the supervisor is stopped, following the backup/restore procedure in `docs/local-production.md`. 5. Restart, run `pnpm local:health`, and issue a harmless locked audit request. Confirm the retained count advances and the health state returns ready. If the store is lost, signer audit and venue reconciliation remain the capital-lifecycle evidence. Loss of this aggregate never authorizes resubmission. For receiver setup or a pending/degraded delivery incident, follow [`control-plane-alert-delivery.md`](/nytshift-docs/policy-control-and-incident-response/control-plane-alert-delivery.md). An aggregate alert recovery never proves an individual ambiguous order is final. ## Disclosure boundary The store, endpoint, dashboard, logs, and public health response must never include account addresses, symbols, order values, prices, quantities, request bodies, URLs, credentials, signatures, tokens, signer detail JSON, or database paths. Preserve only fixed operation/outcome/status/timing fields plus server-owned span/root UUIDs. Follow [`execution-flow-tracing.md`](/nytshift-docs/policy-control-and-incident-response/execution-flow-tracing.md) for lineage recovery rules. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/control-plane-observability.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.