> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/normative-design/rhc-observability.md). # Robinhood Chain read-path observability The production read path should emit aggregate metrics without secrets, sensitive headers, raw RPC bodies or complete wallet histories. | Metric | Alert condition | Action | | ------------------------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------- | | `rhc_rpc_error_rate` | over 5% for 5 minutes | inspect provider status and circuit state | | `rhc_rpc_latency_p95_ms` | exceeds the operator SLO for 5 minutes | inspect provider capacity and region before changing timeout policy | | `rhc_block_divergence` | over 3 blocks | make affected context unavailable; follow provider-divergence runbook | | `rhc_websocket_head_age_ms` | reaches `RHC_WEBSOCKET_STALE_MS` | mark the stream stale, reconnect with jitter and keep readiness reset | | `rhc_websocket_reconnects` | repeated reconnects inside the metrics window | inspect provider incidents, egress and authentication | | `rhc_provider_stabilizing` | does not clear after the configured window | verify both HTTP and WSS transports remain healthy without divergence | | `rhc_oracle_age_seconds` | at or beyond reviewed heartbeat | hide numeric price; follow feed-stale runbook | | `rhc_sequencer_state` | down or recovery grace | block verified price state | | `rhc_registry_mismatch_total` | any increment | block asset context; follow registry-mismatch runbook | | `rhc_partial_portfolio_rate` | over 2% for 10 minutes | inspect multicall/token failures; keep incomplete valuation explicit | | `rhc_activity_gap_blocks` | any unresolved range after the next refresh | keep activity partial; inspect provider limits and replay the checkpoint gap | | `rhc_activity_checkpoint_age_seconds` | over 120 seconds for an actively inspected wallet | verify safe-head reads, store health and request supervision | | `rhc_activity_store_state` | corrupt, migration-required or unwritable | stop indexing; preserve the store and follow local recovery instructions | `GET /api/rhc/providers` exposes the bounded process-local window as request/success/error counts, error rate, average/p95 latency, last typed error, circuit state, WSS head age inputs, lifetime and rolling-window reconnect counts, stabilization state, active alerts, monitor state and durable event count. It never exposes URLs, request bodies or credentials. The unattended monitor starts from the loopback health probe and samples every `RHC_PROVIDER_MONITOR_INTERVAL_MS` (15 seconds by default), even when the terminal is closed. It evaluates: * provider-to-provider and HTTP-to-WSS block divergence; * all-provider loss and primary-to-fallback activation; * open or half-open RPC circuits; * HTTP error rate above `RHC_PROVIDER_ALERT_ERROR_RATE_BPS` after `RHC_PROVIDER_ALERT_MIN_REQUESTS` observations; * configured WSS transports that are connecting, reconnecting, stale or unavailable; and * `RHC_PROVIDER_ALERT_RECONNECTS` reconnects inside the rolling provider metrics window. The normal `PUBLIC_RPC_DIAGNOSTIC_ONLY` state is retained as a topology transition but is not an operational alert. Error-rate alerts do not resolve after a restart until the minimum sample count proves recovery. Reconnect-storm alerts do not resolve until a continuous healthy WSS connection spans the configured metrics window. Secret-free topology, alert-open and alert-resolved transitions are atomically persisted at `~/.nightshift/data/rhc-provider-observability-v1.json` with private file permissions. Schema v2 adds leased delivery state and migrates v1 with an empty outbox. Retention is bounded by `RHC_PROVIDER_INCIDENT_RETENTION_DAYS` and `RHC_PROVIDER_INCIDENT_MAX_EVENTS`. The store contains chain ID, reason codes, provider roles, thresholds, numeric observations, delivery IDs/status and timestamps only—never endpoints, response bodies, credentials, wallet identifiers or order data. When the HTTPS webhook and 32+ character HMAC secret are configured together, alert-open and alert-resolved transitions are enqueued atomically. Retry uses bounded exponential delay with deterministic jitter; pending entries are not discarded to meet the delivered-entry retention target. Durable leases prevent simultaneous workers from sending the same item, while receiver-side delivery-ID deduplication handles the unavoidable crash-after-acceptance window. `/api/rhc/providers/health` exposes only enabled/state/count/timing/error-code summaries. `pnpm backup:create` includes this store and the public-wallet activity store in an allowlisted, SHA-256 manifest. `pnpm backup:verify -- ` detects changed bytes or undeclared content. Restore requires the supervisor to be stopped and retains the displaced checkpoint under `~/.nightshift/recovery`; see `docs/local-production.md`. * `GET /api/rhc/providers/incidents?limit=50` returns newest-first retained lifecycle history and current alerts. * `GET /api/rhc/providers/health` returns aggregate store, monitor, threshold and retention health. * `GET /api/health` starts the unattended monitor and reports its runtime state without making upstream outages trigger a web restart loop. Each lifecycle transition is also written as a bounded JSON record to process output. The supervised localhost runtime appends that output to `~/.nightshift/nightshift.log`; wallet identifiers must be hashed or omitted from routine telemetry. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/normative-design/rhc-observability.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.