> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/control-plane-alert-delivery.md). # Control-plane alert delivery Use this runbook when the terminal reports an active control-plane SLO alert, delivery is disabled/degraded, pending delivery grows, or Hyperliquid mainnet readiness reports `CONTROL_PLANE_ALERTING_REQUIRED`. This receiver is operational notification only. It never establishes order finality or grants execution authority. ## Configure the receiver 1. Keep Hyperliquid mainnet disabled while configuring delivery. 2. Provision an operator-controlled HTTPS endpoint that rejects duplicate `idempotency-key` values only after the first request has been durably accepted. Do not put credentials in the URL. 3. Generate a distinct random secret of at least 32 characters. Supply `NIGHTSHIFT_OBSERVABILITY_ALERT_WEBHOOK_URL` and `NIGHTSHIFT_OBSERVABILITY_ALERT_WEBHOOK_SECRET` together in the private runtime environment; never put the secret in `config.json`, the repository, browser storage, a prompt, or a URL. 4. Restart the loopback runtime and run `pnpm local:health`. Unlock the terminal and inspect `24H CONTROL PLANE`; delivery must become `RUNNING`. No test alert is synthesized because NIGHTSHIFT never fabricates incidents. 5. Before mainnet authorization, exercise the receiver in an isolated test using the documented envelope verifier, prove duplicate handling, then perform an approved real SLO/recovery drill without an order. The receiver verifies `x-nightshift-signature` as lowercase hex HMAC-SHA-256 over `.` using the shared secret. It must also verify a narrow timestamp window and require `x-nightshift-delivery-id` to equal the body `deliveryId` and `idempotency-key`. The envelope fields are `schemaVersion`, `source`, `event`, `deliveryId`, `occurredAtMs`, `kind`, `severity`, `code`, `from`, `to`, `observedValue` and `threshold`. Only `https:` URLs without userinfo, query strings or fragments are accepted. Redirects are rejected. Timeout, cadence, retry, batch, retention and cap controls are documented in `.env.example`; keep their bounded defaults unless an incident review justifies a change. ## Delivery failure 1. Keep Hyperliquid mainnet disabled. Do not retry any order because an alert delivery failed. 2. Inspect public `pnpm local:health` for delivery state, pending count and reason, then unlock Terminal settings for delivery attempts and the fixed incident code. Neither surface reveals the URL or secret. 3. Check receiver availability, TLS validity, its exact raw-body verification and idempotency behavior. A 2xx is success; non-2xx, network failure or timeout schedules a bounded retry. NIGHTSHIFT discards every response body. 4. Correct the private environment or receiver and restart the runtime if configuration changed. Expired leases become claimable; the stable delivery ID lets the receiver handle a request that succeeded remotely before local acknowledgement. 5. Confirm pending count falls to zero and state returns `RUNNING`. Preserve the checkpoint and receiver evidence for the incident record. Never hand-edit or delete pending entries. The default private store is `%USERPROFILE%\.nightshift\data\control-plane-observability-v1.json` on Windows or `~/.nightshift/data/control-plane-observability-v1.json` elsewhere; the stable filename now contains schema v3 span/root correlation while preserving schema-v2 SLO state. Use the verified public-state backup/restore workflow only while the supervisor is stopped. ## Active or recovered SLO For `CONTROL_PLANE_UNKNOWN_RATE_HIGH` or `CONTROL_PLANE_UNAVAILABLE_RATE_HIGH`, stop new execution and reconcile every ambiguous in-flight client ID/CLOID through signer audit plus exact venue state. For rejection or latency alerts, establish whether the cause is expected operator/policy rejection, authentication churn, rate pressure, signer health or venue degradation before restoring authority. An `alert-resolved` delivery means only that the rolling aggregate fell below its configured threshold after a complete sample. It never resolves an individual order. Follow the control-plane observability and venue-specific reconciliation runbooks until each capital-moving lifecycle is terminal. ## Disable delivery Remove both webhook variables from the private runtime environment and restart. This intentionally returns delivery to `DISABLED` and keeps Hyperliquid mainnet gated. It does not disable localhost, public market data, Robinhood Chain reads, or an authorized Hyperliquid testnet rehearsal. Arcus uses its separate signer-owned alert configuration. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/control-plane-alert-delivery.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.