> ## Documentation Index
> Fetch the complete documentation index at: https://hub.ypym.app/llms.txt
> Use this file to discover all available pages before exploring further.

# DBX Operational Reliability, Multi-Tenancy & Crawl Governance

> Comprehensive guide to multi-tenant isolation, cryptographic OTP approvals, supervised scheduler lifecycles, and failure recovery in YPYM DBX.

## Executive Overview

Operating an enterprise search intelligence platform requires deterministic security, robust tenant isolation, and predictable financial control. In high-stakes production environments, crawling actions directly interact with paid third-party APIs (DataForSEO) and process sensitive client portfolio data.

**YPYM DBX (Digital Brand Experience)** incorporates a zero-trust multi-tenancy model, cryptographic OTP approvals for manual scraping operations, strict time-window enforcement, and supervised background job execution.

***

## 1. Multi-Tenant Architecture & Portfolio Isolation

YPYM DBX is engineered as a secure multi-tenant platform serving internal headquarters (YPYM HQ) and external corporate clients (e.g. Nobu Bank, Venture Studio portfolio companies) from a single unified codebase.

```mermaid theme={null}
flowchart TD
    A["Incoming Request (HTTP / Session)"] --> B["verifyUser & withRequestScope"]
    B --> C{"User Role?"}
    C -- "Super Admin" --> D["Cross-Tenant Portfolio Visibility\n• Headquarters (Internal)\n• Client Portfolios (Nobu Bank, etc.)\n• Access to /admin Clients Console"]
    C -- "Enterprise Client" --> E["Strict Single-Tenant Isolation\n• Domain & Keyword Scoping (requestTenant)\n• /admin Access Blocked\n• No Leaked Competitor or Internal Assets"]
    
    style A fill:#1A4BFF,stroke:#1A4BFF,color:#fff
    style B fill:#30FFFC,stroke:#1A4BFF,color:#000
    style C fill:#1A4BFF,stroke:#1A4BFF,color:#fff
    style D fill:#f8f9fa,stroke:#e9ecef,color:#333
    style E fill:#1A4BFF,stroke:#1A4BFF,color:#fff
```

### Key Multi-Tenancy Principles:

* **Strict Role-Gating**: Super Admin accounts have oversight across all client organizations, keyword pools, and aggregated expenditure metrics. Enterprise client accounts are isolated strictly to their assigned `tenant_id`.
* **TopBar & Navigation Hygiene**: The `/admin` ("Clients") navigation link is dynamically rendered only for authenticated Super Admins. It is completely hidden from client accounts to prevent confusion and unauthorized route exploration.
* **Portfolio Grouping on Dashboard**: In the main domains directory (`/domains`), domains are logically partitioned into clear portfolio groups:
  * **Headquarters (Internal)**: Core brand assets (e.g., `ypym.app`).
  * **Client Portfolios**: Grouped by client organization (e.g., `Nobu Bank: www.nobubank.com, nobugo.nobubank.com`).
* **Domain Identity Formatting**: All domain names are rendered in **strict lowercase** (`ypym.app` instead of capitalized `Ypym.app`), respecting standard DNS naming conventions.

***

## 2. Crawl Security & Cryptographic OTP Approval

To prevent accidental balance depletion (*rage clicks*, unauthorized runs, or duplicate batch execution), all manual crawling triggers—including manual Google SERP crawls and on-demand Multi-LLM prompt audits (*Audit All Prompts*)—require explicit confirmation via a 6-digit cryptographic OTP.

```
┌────────────────────────────────────────────────────────┐
│  Confirm Manual Crawl / Prompt Audit                   │
│                                                        │
│  Target: 15 Keywords / Prompts (domain.com)            │
│  Type: Google SERP Crawl / Multi-LLM GEO Audit         │
│                                                        │
│  Estimated Cost:                                       │
│  • Total USD: $0.0525                                  │
│  • Total IDR: Rp 940 (Benchmark Rate: $1 = Rp 17,914)  │
│                                                        │
│  6-Digit Verification Code: [  ______  ]               │
│  (Dispatched to authorized operator email)             │
│  OTP Validity: 10:00 minutes                           │
│                                                        │
│  [ Cancel ]                  [ Verify & Execute ]      │
└────────────────────────────────────────────────────────┘
```

### Operational Rules for OTP Challenges:

1. **Scope & Window Restrictions**: Manual SERP crawls (single keyword, bulk keywords, domain refresh) are restricted to the maintenance window between **11:00 AM and 11:15 AM WIB**. Multi-LLM batch audits (*Audit All Prompts*) can be executed on-demand with pre-flight cost verification and OTP approval.
2. **Context Binding**: The OTP challenge is cryptographically bound to the user session, target domain, specific keyword or prompt IDs, and estimated financial cost.
3. **Short Expiration (10 Minutes)**: Codes expire after 600 seconds. Expired or unconsumed challenges are rejected.
4. **Rate Limiting**: Maximum 5 challenge requests per hour per user with a 60-second cooldown between resend attempts.
5. **Atomic Consumption**: Challenges are consumed atomically in PostgreSQL (`crawl_challenges`), preventing double-spend or replay attacks.

***

## 3. Supervised Background Execution & Job Recovery

Crawling operations run as managed background jobs rather than synchronous HTTP requests to prevent web server timeouts on large keyword sets:

| Component | Responsibility | Recovery Behavior |
| - | - | - |
| **Dual-Phase Cron Daemon** | Dispatches Phase 1 (11:00 AM WIB SERP) and Phase 2 (11:30 AM WIB Multi-LLM GEO) | Claims daily crawl slots using atomic claim files (`cron-serp-YYYY-MM-DD.claim` and `cron-llm-YYYY-MM-DD.claim`) in `/app/data`; prevents duplicate daily executions across container restarts |
| **Job Processor** | Manages asynchronous task lifecycle (`running`, `completed`, `failed`) | Tracks progress in `crawl_jobs` table; updates status per keyword/prompt; surfaces execution records via `/api/crawl/jobs` |
| **Scheduler Supervisor** | Monitors background job health | Automatically sweeps jobs stalled for >30 minutes, marking them `interrupted` and clearing locks |
| **Circuit Breaker** | Protects DataForSEO API balance | Halts execution if provider rate limits (HTTP 429) or consecutive network failures occur |

***

## 4. UI State & Preferences Persistence

DBX ensures seamless user experiences across devices without unnecessary backend overhead:

* **Centralized Crawl History Modal**: Accessible directly from the dashboard action bar, allowing operators to inspect past Google SERP and Multi-LLM crawl jobs with engine breakdown, volume metrics, duration, and status.
* **Local Column Visibility**: Toggling keyword table columns (*Best, History, Volume, Search Console*) is managed via client-side state and persisted to browser `localStorage` (`ypym_dbx_keyword_columns`). This eliminates `403 Forbidden` errors for client users and provides instantaneous UI response without database mutations.
* **Multi-Mode Trajectory Visualization**: Operators can switch between **Trend** (Line), **Comparison** (Grouped Bar), and **Stacked** (Stacked Bar) modes on an expanded visual canvas.
* **Dynamic Countdown Ticker**: Visual countdown displays time remaining until the next automated 11:00 AM WIB crawl, transitioning into an urgent red-maroon per-second ticker during the final 59 minutes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.