> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/production-authority-incident.md). # Production authority incident command Use this runbook for any event that can make execution authority, venue state or recovery evidence unsafe. It coordinates the narrower provider, signer, reconciliation, delivery and rollback procedures; it does not replace them. ## Named operational roles * **Incident Commander:** the current production on-call; owns severity, timeline, stop/go decisions and the final incident record. * **Risk Owner:** owns venue exposure review and must approve any restoration of new-entry authority. * **Security Owner:** owns compromise containment, revocation, credential rotation and evidence handling. * **Execution Owner:** owns signer/store health and exact-ID venue reconciliation. * **Release Owner:** owns immutable artifact verification, service/config hashes and exact rollback. * **Communications Owner:** owns the private stakeholder/status channel and must not copy secrets or customer identifiers. The controlled on-call roster must map each role to a current person and private contact route. If a role cannot be reached, the Incident Commander keeps all new entry disabled; missing approval is never implied approval. ## Immediate containment 1. Record UTC detection time, release commit, fixed reason code and affected venue. Do not record wallets, order bodies, tokens, URLs or raw provider responses in chat or tickets. 2. Disable new entry independently: * Arcus: set `ARCUS_EXECUTION_KILL_SWITCH=true` and restart only the Arcus signer boundary. * Hyperliquid: set `HYPERLIQUID_EXECUTION_KILL_SWITCH=true`; for mainnet also set `HYPERLIQUID_MAINNET_ENABLED=false`, then restart only the Hyperliquid signer boundary. * Web/control plane: `EXECUTION_MODE=disabled` and `ARCUS_EXECUTION_MODE=disabled` prevent new review/submit surfaces while public reads remain available. 3. Keep exact cancellation and reconciliation protection available only where its separate authenticated policy permits it. A global web disable must not erase signer evidence. 4. Stop the affected signer for suspected key, token, host or binary compromise. Do not delete its database, runtime record, logs or pending alert outbox. 5. Classify every non-terminal order/cancel/transfer as `unknown` or its last authoritatively reconciled state. Never retry a capital-moving request to test recovery. ## Incident routing | Condition | Required runbook | Restoration proof | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | | RHC/provider divergence or schema drift | [Robinhood Chain provider divergence](/nytshift-docs/market-data-and-providers/provider-divergence.md), [registry mismatch](/nytshift-docs/market-data-and-providers/registry-mismatch.md), [Hyperliquid metadata drift](/nytshift-docs/market-data-and-providers/hyperliquid-metadata.md) | Independent providers agree inside the approved window; schema and registry checks pass | | Suspected signer/API key compromise | [Credential and signer rotation](/nytshift-docs/policy-control-and-incident-response/credential-key-rotation.md), [kill switch](/nytshift-docs/policy-control-and-incident-response/kill-switch.md) | Old authority revoked, new scope reviewed, store retained, new rehearsal/evidence approved | | Unknown submission/cancellation | [Unknown order outcome](/nytshift-docs/policy-control-and-incident-response/unknown-order.md), venue lifecycle runbook | Exact client ID/CLOID reaches an authoritative terminal or explicitly unresolved state | | Reconciliation outage | [Arcus reconciliation incident](/nytshift-docs/arcus/arcus-reconciliation-incident.md), [Hyperliquid lifecycle](/nytshift-docs/hyperliquid/hyperliquid-order-lifecycle.md) | Reconciliation reads are healthy and every retained ambiguous ID is accounted for | | Alert delivery outage | [Arcus alert delivery](/nytshift-docs/arcus/arcus-execution-alert-delivery.md), [control-plane delivery](/nytshift-docs/policy-control-and-incident-response/control-plane-alert-delivery.md), [provider delivery](/nytshift-docs/market-data-and-providers/provider-alert-delivery.md) | Pending outbox drains with stable delivery IDs; receiver verification passes | | Unsafe signer store/database | [Execution-store recovery](/nytshift-docs/arcus/arcus-execution-store-recovery.md), [customer database](/nytshift-docs/agents-identity-and-data/customer-database.md) | Verified backup/restore, current schema, integrity and WAL evidence; retained unknowns reconciled | | Bad web/control-plane release | [Exact release rollback](/nytshift-docs/policy-control-and-incident-response/exact-release-rollback.md) | Previous archive receipt and embedded markers verify; service, health and browser gates pass | ## Restoration gate New entry remains disabled until the Incident Commander has a written statement from the Risk, Security, Execution and Release owners that their affected controls are healthy. The evidence must include exact release, configuration hashes, store health, alert delivery, venue/provider freshness, non-terminal order inventory and the applicable rotation or rollback receipt. Mainnet restoration additionally requires the normal signed eligibility, canary and final user authorization; an incident record cannot grant it. Close the incident only after reconciliation and customer-history projection complete. Alert recovery, HTTP ACKs, balance changes and absence from one open-order list are not finality. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/policy-control-and-incident-response/production-authority-incident.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.