> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/arcus/arcus-execution-store-recovery.md). # Execution-store recovery Use this runbook when signer startup reports an `EXECUTION_STORE_*` failure, signer health is unavailable, or the terminal shows `MIGRATION REQUIRED`, `INTEGRITY FAILED`, or `JOURNAL UNSAFE`. Keep the Arcus kill switch active. Do not submit, cancel, clear an incident, delete rows, change `user_version`, or retry an ambiguous order while store health is unsafe. ## Healthy contract `pnpm signer:health` must report all four values before any separately authorized rehearsal: * `databaseSchemaVersion=8` * `databaseSchemaCurrent=true` * `databaseIntegrity=ok` * `databaseJournalMode=wal` The database defaults to `%USERPROFILE%\.nightshift\execution.sqlite3`. It contains sensitive account, intent, audit, reconciliation, alert-delivery and Hyperliquid order/fill/revenue metadata even though it contains no private key or webhook secret. Protect every copy accordingly. ## Make a cold backup Stop the signer first. The backup command refuses a live signer and acquires the same store-operation lock that the signer holds for its lifetime: ```powershell pnpm signer:stop pnpm signer:backup:create pnpm signer:backup:list pnpm signer:backup:verify -- ``` Creation uses SQLite's snapshot API and verifies source and result independently. The manifest binds SHA-256, bytes, schema fingerprint, migration count and integrity. A sealed backup reports journal mode `delete` because it is deliberately normalized into one portable database without sidecars. This is not a runtime downgrade: signer startup re-establishes WAL and must pass its normal health gate. Do not use the public checkpoint `backup:*` commands for this file; that allowlist intentionally covers read-only market/provider stores. Do not manually copy a live main database, omit a WAL sidecar, or add files to a sealed backup directory. Treat `%USERPROFILE%\.nightshift\signer-backups` as sensitive and move copies only through operator-controlled encrypted storage. ## Failure meanings * `APPLICATION_ID_MISMATCH` or `UNKNOWN_TABLE`: the path is not a trusted NIGHTSHIFT execution database. Correct `EXECUTION_DB_PATH`; do not adopt it. * `FUTURE_SCHEMA`: this binary is older than the database. Restore the matching/newer reviewed application; never downgrade the version marker. * `MIGRATION_HISTORY_INVALID`, `SCHEMA_INVALID`, or `TABLE_SET_INVALID`: preserve the full database set and stop. Manual DDL is not recovery. * `INTEGRITY_FAILED` or `DATABASE_INVALID`: preserve the database and storage evidence. Do not keep restarting against the only copy. ## Restore a known cold copy Only restore a snapshot that remains `verified`. Keep the signer stopped: ```powershell pnpm signer:backup:verify -- pnpm signer:backup:restore -- ``` Restore stages and re-attests the snapshot, moves the current database and any WAL/SHM sidecars into `%USERPROFILE%\.nightshift\signer-recovery\`, and verifies the installed database. It rolls the original files back on failure. `NIGHTSHIFT_SIGNER_BACKUP_ROLLBACK_FAILED` means automatic rollback could not finish; do not delete the recovery directory or start the signer. Start the testnet-only signer and require healthy migration/integrity evidence: ```powershell pnpm signer:start pnpm signer:health ``` After restore, reconcile every non-terminal `clientId`, active incident, pending cancel, and alert delivery against Arcus public state. Restoring older bytes can reintroduce a previously pending claim; it is never authority to resend. Only exact `FILLED`, `CANCELED`, or `REJECTED` lifecycle proof may clear reconciliation state. Keep the kill switch active until that audit is complete. If no known-good copy exists, leave execution disabled and escalate with the preserved database set and fixed error code. Rebuilding an empty database is not recovery because it discards exactly-once and unknown-outcome evidence. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/arcus/arcus-execution-store-recovery.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.