> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/engineering-guide/12-production-operations.md). # Production operations [← Testing](/nytshift-docs/engineering-guide/11-testing-and-verification.md) · [Documentation home](/nytshift-docs/start-here/readme.md) · Next: [Repository atlas →](/nytshift-docs/engineering-guide/13-repository-atlas.md) The approved public host is the operator-owned `nytshift.xyz` VPS. Cloudflare Workers/Pages and OpenAI Sites are explicitly not release targets. A public artifact is built and independently verified from a clean exact commit before promotion. ## Release chain ```mermaid flowchart LR G[Clean exact Git commit] --> L[Linux x64 release builder] L --> Q[Preflight + contracts + tests + build + smoke + handoff] Q --> A[Content-manifested archive] A --> R[Receipt: commit, SHA-256, tree, count, bytes] R --> V[Independent verification] V --> ST[Isolated staged smoke] ST --> P[Atomic VPS promotion] P --> H[Public health/build trust] H -->|failure| RB[Automatic prior-release rollback] ``` The repository release workflow has no SSH, DNS, service or promotion authority. Deployment is an operator-owned external step that consumes the verified artifact. ## Safe release environment Release builds force the capital posture closed: * `EXECUTION_MODE=disabled`; * `ALLOW_MAINNET=false`; * `ARCUS_EXECUTION_MODE=disabled`; * `ARCUS_ALLOW_MAINNET=false`; * withdrawal mainnet gates disabled; * no signer key/token/confirmation secret in the artifact; * production mock mode rejected; * hosting metadata checked against the VPS boundary. ## Build trust `/api/health` projects exact release identity, artifact verification, capacity and independent readiness lanes. The terminal clears its build badge on transport, JSON or semantic failure rather than retaining stale trust. Readiness lanes remain separate: | Lane | Requires | Does not grant | | -------------------- | ------------------------------------------------------------------------------------- | --------------------------- | | Release foundation | Exact commit/artifact receipt and safe health | Customer login or execution | | Customer identity | Matching Privy config + fresh production rehearsal | Operator/signer authority | | Native charts | Licensed tree + SRI/CSP + exact-release rehearsal | Market/execution authority | | RHC production data | Redundant managed HTTP/archive/WSS proof | Transaction authority | | Live order authority | Every signer, eligibility, reconciliation, alert, dead-man, canary and rehearsal gate | Agent/model key access | ## Local supervisor The local production supervisor is loopback-only and manages build freshness, web-child lifecycle and health monitoring. ```powershell pnpm local:start pnpm local:status pnpm local:health pnpm local:restart pnpm local:stop ``` Manual restart stops the previous child before rebuilding and never revives it after a failed build. ## Backups ### Public/private application stores ```powershell pnpm backup:create pnpm backup:list pnpm backup:verify -- pnpm local:stop pnpm backup:restore -- pnpm local:start ``` The manifest binds logical path, data class, bytes and SHA-256. Restore requires a stopped supervisor and preserves replaced files for rollback. The allowlist includes only reviewed activity, observability, proposal, agent, history, portfolio and customer-preference stores. ### Sensitive signer store Signer snapshots are a separate cold workflow and never enter the public backup bundle. ```powershell pnpm signer:stop pnpm hl-signer:stop pnpm signer:backup:create pnpm signer:backup:verify -- pnpm signer:backup:restore -- ``` Creation/restore serialize through an operation lock and require all signer supervisors stopped. ## Observability Public health exposes aggregate, secret-free state. Protected control-plane observability stores only server UUIDs, fixed operation/outcome codes, timestamps and latency. It excludes request bodies, order/account/receiver details and credentials. Customer product telemetry is a separate opt-in boundary. The browser can submit only fixed event names and coarse route, capability, venue, result, release and reason-code fields. The server retains anonymous hourly counters for seven days. The authenticated `/ops/observability` dashboard validates this contract before rendering and never displays raw recent event bodies. See [privacy-safe product telemetry](/nytshift-docs/policy-control-and-incident-response/privacy-safe-product-telemetry.md). Rolling SLO incidents can enter a durable HMAC outbox. Delivery rejects redirects, uses exact-body signatures and bounded retry. Hyperliquid mainnet can require healthy delivery, but testnet remains available for rehearsal when the mainnet-only gate is absent. Robinhood Chain provider supervision independently tracks HTTP outcomes, live WSS heads, chain ID, divergence, archive evidence, reconnection and stabilization. The official public RPC is diagnostic—not production redundancy. ## Capacity and retention Health names coarse free bytes, utilization, a 2 GiB operating reserve and 512 MiB next-release allowance without exposing filesystem paths. Breaching the reserve makes health unsafe. Observation never prunes a release. Capacity planning and deletion remain explicit operator procedures. ## Incident entry points | Symptom | Runbook | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | Public feed/provider degraded | [Public venue degraded](/nytshift-docs/market-data-and-providers/public-venue-degraded.md) | | Browser stream stale/reconnecting | [Public stream stale](/nytshift-docs/market-data-and-providers/public-stream-stale.md) | | RHC providers diverge | [Provider divergence](/nytshift-docs/market-data-and-providers/provider-divergence.md) | | Canonical registry drift | [Registry mismatch](/nytshift-docs/market-data-and-providers/registry-mismatch.md) | | Unknown order outcome | [Unknown order](/nytshift-docs/policy-control-and-incident-response/unknown-order.md) | | Arcus reconciliation incident | [Arcus reconciliation](/nytshift-docs/arcus/arcus-reconciliation-incident.md) | | Signer database unhealthy | [Execution-store recovery](/nytshift-docs/arcus/arcus-execution-store-recovery.md) | | Operator/session failure | [Operator session](/nytshift-docs/agents-identity-and-data/operator-session.md) | | MCP/agent incident | [BYOA operations](/nytshift-docs/agents-identity-and-data/bring-your-own-agent.md) | | Release provenance/capacity | [Public release](/nytshift-docs/release-and-readiness/public-release.md) · [Release capacity](/nytshift-docs/release-and-readiness/release-capacity.md) | | Cross-domain production authority incident | [Production authority incident command](/nytshift-docs/policy-control-and-incident-response/production-authority-incident.md) | | Suspected credential or signer compromise | [Credential and signer rotation](/nytshift-docs/policy-control-and-incident-response/credential-key-rotation.md) | | Failed promoted release | [Exact release rollback](/nytshift-docs/policy-control-and-incident-response/exact-release-rollback.md) | | Recovery exercise/sign-off | [Rehearsal record template](/nytshift-docs/policy-control-and-incident-response/rehearsal-record-template.md) | See [local-production.md](/nytshift-docs/normative-design/local-production.md) for exact supervisor/store behavior and [release-manifest.md](/nytshift-docs/evidence-and-handoff-records/release-manifest.md) for the current handoff inventory. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/engineering-guide/12-production-operations.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.