> For the complete documentation index, see [llms.txt](https://nytshift.gitbook.io/nytshift-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://nytshift.gitbook.io/nytshift-docs/engineering-guide/02-architecture-and-data-flow.md). # Architecture and data flow [← System overview](/nytshift-docs/engineering-guide/01-system-overview.md) · [Documentation home](/nytshift-docs/start-here/readme.md) · Next: [Trust and authority →](/nytshift-docs/engineering-guide/03-trust-and-authority.md) NYTSHIFT is a pnpm/Turborepo monorepo. The Next.js application owns the public UI and browser-facing BFF, TypeScript packages own canonical contracts and pure domain logic, and Python owns isolated signing/reconciliation adapters. Durable state is split by trust domain rather than placed in one convenient database. ![Five-stage NYTSHIFT evidence loop from fresh market data through bounded intent, AEGIS, reconciled evidence and the final user gate](https://365345826-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FD02RJbcs8ypn0hikgNUk%2Fuploads%2Fgit-blob-7abffac0c00c4fa4c4d2a69de2ba98c153af2a73%2F02-verified-loop.png?alt=media) ## Runtime topology ```mermaid flowchart TB subgraph Browser UI[Next.js terminal] WS[Pooled venue WebSockets] LP[Bounded guest preferences and PAPER cache] end subgraph Web[Next.js BFF] PUB[Public market and RHC routes] AUTH[Customer and operator routes] MCP[MCP gateway] CTRL[Review, policy and observability] CDB[(Customer PostgreSQL ledger)] end subgraph Core[Pure TypeScript domains] CONTRACTS[contracts] VENUES[venue adapters] RISK[AEGIS risk] PORT[portfolio reducer] PAPER[paper engine] AI[proposal validation] end subgraph Signers[Loopback-only Python boundary] HS[Hyperliquid signer] AS[Arcus signer] DB[(SQLite WAL and audit)] end EXT[Arcus · Lighter · Hyperliquid · RHC RPC] --> PUB EXT --> WS WS --> UI PUB --> UI UI --> LP PUB --> VENUES AUTH --> CONTRACTS MCP --> CONTRACTS CTRL --> RISK CTRL --> PORT CTRL --> PAPER AUTH --> CDB PAPER --> CDB CTRL --> HS CTRL --> AS HS --> DB AS --> DB ``` ## Major source units | Path | Responsibility | | ----------------------- | --------------------------------------------------------------------------- | | `apps/web` | Product, public and authenticated BFF routes, CSP, health and production UI | | `apps/api` | Explicitly enabled loopback development API; not a production dependency | | `packages/contracts` | Zod registry, shared types, generated JSON Schema/OpenAPI | | `packages/venues` | Strict read-only venue and Robinhood Chain adapters | | `packages/agent-client` | No-secret DPoP identity/client kit for policy-bound testnet automation | | `services/risk` | AEGIS pure deterministic evaluator | | `services/ai` | Proposal validation and abstention boundary | | `services/market-data` | Hyperliquid stream/Info normalization | | `services/portfolio` | Idempotent order/fill/funding/position reduction | | `services/paper` | Deterministic paper/replay domain | | `services/execution-py` | Isolated signer, exact-decimal validation, persistence and reconciliation | | `scripts` | Supervisors, audits, backups, release builder and handoff gates | ## Data plane ### Public REST The BFF pins venue origins, caps response bytes and concurrency, applies bounded circuit state, hashes account-rate-limit keys and emits only typed failure codes. Raw provider payloads do not flow directly into browser state. ### Public streams The browser owns one pooled socket per venue. Subscriptions are reference-counted and replayed after reconnect. Arcus and Hyperliquid books replace from complete frames; Lighter applies deltas only across exact nonce continuity. A gap preserves the last accepted state, degrades the feed and demands snapshot repair. ### Reference authority Mark, midpoint, oracle and index are different evidence types. Consumer projections decide whether a value is admissible for charts, alerts, paper, order shortcuts or evidence-only display. See [Market data and charting](/nytshift-docs/engineering-guide/04-market-data-and-charting.md). ### Local history Validated provider REST bars are the only write authority for the local candle checkpoint. The store is versioned, capped and atomically updated. An outage may expose a clearly aged snapshot, but persisted history never becomes a live provider claim. ## Control plane Protected actions start in the BFF, not in browser code. The BFF creates request spans and durable flow roots, authenticates the operator or agent capability, rebuilds fresh context, runs AEGIS and issues only short-lived one-use scopes. Signers independently revalidate their own network, account, policy, amount and persistence gates. ```mermaid sequenceDiagram participant B as Browser/client participant W as Web BFF participant A as AEGIS participant S as Isolated signer participant V as Venue B->>W: typed review request W->>W: authenticate + fresh reads W->>A: normalized intent and evidence A-->>W: allow or stop with reasons W-->>B: one-use review + disclosure B->>W: explicit confirmation W->>W: consume scope + repeat checks W->>S: authenticated exact-decimal request S->>S: validate + persist claim S->>V: at most one signed attempt V-->>S: ACK, rejection or ambiguity S-->>W: typed result W-->>B: final or reconcile-required state ``` ## State ownership | State | Owner | Why | | ----------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Watchlists, layouts, alerts | Browser-local versioned stores | Presentation preference, no capital authority | | Guest PAPER ledger | Browser-local exact schema | Local simulation with explicit open-tab limits | | Authenticated profile/PAPER history | Customer PostgreSQL ledger | Opaque customer scope, optimistic revisions, archived sessions and queryable order/fill/PnL history | | Verified testnet/live history | Strict signer reconciliation → customer PostgreSQL projection | HMAC account/provider IDs, one-customer ownership, idempotent observations, exact fills/fees/closed PnL and explicit bounded coverage; no execution authority | | MCP connection/paper stores | Web private durable stores | Revocation, scope and server-owned policy | | Customer preferences | Web private durable store | Opaque identity binding; no provider tokens | | Market history | Web private bounded store | Degraded historical continuity only | | Proposal/automation stores | Web private durable stores | Quota, lifecycle, replay and policy evidence | | Execution state | Signer SQLite WAL | Must survive ambiguity and remain outside browser/web secrets | | Public release identity | Immutable release marker/receipt | Exact commit and artifact provenance | ## Failure semantics The system does not collapse errors into a generic unavailable boolean. | State | Meaning | Allowed response | | --------- | ------------------------------------------------------ | ---------------------------------------------------- | | Missing | Evidence never existed in the current scope | Show unavailable; do not derive | | Partial | Some required fields or pages are absent | Preserve components; withhold aggregate authority | | Stale | Valid evidence exceeded its age budget | Show age/reason; block freshness-dependent consumers | | Degraded | Recovery is active or provider confidence fell | Preserve explicitly retained evidence only | | Ambiguous | More than one identity/result satisfies the lookup | Stop; require disambiguation/reconciliation | | Unknown | A capital action may or may not have reached the venue | Never retry blindly; reconcile exact identity | The exhaustive normative version lives in [architecture.md](/nytshift-docs/normative-design/architecture.md). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://nytshift.gitbook.io/nytshift-docs/engineering-guide/02-architecture-and-data-flow.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.