> ## Documentation Index > Fetch the complete documentation index at: https://hub.ypym.app/llms.txt > Use this file to discover all available pages before exploring further. # DataForSEO Cost Governance & Crawl Mechanics (Option C) > Mathematical cost models, hybrid 11:00 AM WIB crawl scheduling, 10-keyword guardrails, and audit ledger mechanics in YPYM DBX. ## Financial Governance Philosophy In high-volume SERP tracking, scraping costs scale directly with keyword count, refresh frequency, and proxy architecture. Without strict financial guardrails, automated tools can trigger accidental balance depletion through runaway concurrent requests, duplicate refreshes, or micro-batch crawling. **YPYM DBX introduces enterprise cost governance (Option C)**: a deterministic framework that balances real-time ranking freshness with predictable API expenditure and pre-flight budget approvals. *** ## 1. Ground-Truth Provider Pricing Models (Page 2 Standard) YPYM DBX utilizes **DataForSEO v3 SERP API** as its primary scraping infrastructure. Tracking is calibrated by default to **Page 2 (Top 20 results, `depth: 20`)**, delivering strategic visibility over immediate positions while avoiding 10-page micro-billing penalties. Costs are computed using exact provider unit economics: | Scraper Tier | Coverage Depth | Unit Cost (USD) | Unit Cost (IDR @ Rp 17,914) | Cost per 1,000 Targets | Primary Use Case in DBX | | - | - | - | - | - | - | | **Google SERP Live** | **Page 2 (Top 20)** | **\$0.0035** | **Rp 62.70** | **\$3.50 / Rp 62,700** | On-Demand SERP Crawls (`Crawl SERP Keywords`) | | **Google SERP Queue** | **Page 2 (Top 20)** | **\$0.00105** | **Rp 18.80** | **\$1.05 / Rp 18,800** | Automated 11:00 AM WIB Daily Batch Crawls | | **Multi-LLM GEO Live** | **4 AI Engines** (ChatGPT, Gemini, Perplexity, Claude) | **\~\$0.0089** | **\~Rp 159.00** | **\~\$8.90 / Rp 159,000** | On-Demand Prompt Audits (`Crawl all Prompts`) | > [!NOTE] > > Benchmark exchange rate is calibrated to **1 USD = Rp 17,914** reflecting real payment settlement rates. SERP pricing includes Page 1 (\$0.0020 Live / \$0.0006 Queue) plus Page 2 (\$0.0015 Live / \$0.00045 Queue at 25% discount). Multi-LLM GEO pricing combines DataForSEO base fees (\$0.0006 x 4) plus provider token costs across ChatGPT (`gpt-4o`), Gemini (`gemini-1.5-pro`), Perplexity (`sonar`), and Claude (`claude-3-5-sonnet`). ### Live Balance Extraction DBX queries the provider account balance directly in real-time via the `/v3/appendix/user_data` endpoint: ```json theme={null} { "tasks": [ { "result": [ { "money": { "balance": 47.6543, "currency": "USD" } } ] } ] } ``` This balance is continuously displayed in the **Financial Status Bar**, ensuring operators maintain full visibility into remaining crawling reserves before initiating runs. *** ## 2. Dual-Phase Automated Crawling & Hybrid Lifecycle To eliminate unnecessary expenditure while supporting flexible ad-hoc research, DBX implements a coordinated **Dual-Phase Automated Crawling Lifecycle**: ``` ┌──────────────────────────────────────────────┐ │ Daily Automated Execution │ └──────────────────────┬───────────────────────┘ │ ┌────────────────────────┴────────────────────────┐ ▼ ▼ [ Phase 1: 11:00 AM WIB ] [ Phase 2: 11:30 AM WIB ] Google SERP + AI Overviews Multi-LLM GEO Intelligence • Bulk rank & position updates • ChatGPT, Gemini, Perplexity, Claude • Standard Priority Queue execution • Automated prompt citation extraction • Scheduled keyword promotion • Zero ad-hoc operator overhead ``` ### Key Lifecycle Rules: 1. **Phase 1 Automated Batch (11:00 AM WIB / 04:00 UTC)**: * All active keywords are automatically dispatched in bulk using efficient batching. * At the conclusion of the scheduled crawl, any pending `on_queue` keywords are promoted to active status. 2. **Phase 2 Automated LLM Batch (11:30 AM WIB / 04:30 UTC)**: * All active prompts are analyzed across the configured generative engines (ChatGPT Search, Gemini, Perplexity, Claude). * Generative citations, source domains, and brand mentions are extracted and stored into the database. 3. **Post-11:00 AM Quarantine (`on_queue`)**: * Any keywords added after 11:00 AM WIB are tagged with `state: "on_queue"`. * They do **not** trigger immediate provider calls, preventing unintended credit burns. 4. **Protected On-Demand Actions**: * Both manual SERP crawls (`Crawl SERP Keywords`) and Multi-LLM prompt audits (`Crawl all Prompts`) are safeguarded with pre-flight cost estimations and OTP confirmation. *** ## 3. Crawl Execution Schedule & Security Safeguards To prevent uncontrolled API expenditure and ensure deterministic operations, on-demand and manual crawl actions are protected by strict temporal and operational guardrails: * **Time-Window Guardrail (11:00–11:15 WIB for SERP)**: Manual SERP crawl requests (`Crawl SERP Keywords`, bulk selected keywords, and full domain refresh) are restricted to the primary daily maintenance window between **11:00 AM and 11:15 AM WIB** at **\$0.0035 / keyword**. Requests dispatched outside this 15-minute window are blocked server-side (`HTTP 409 Conflict`) to preserve budget stability and enforce synchronization with automated schedules. * **On-Demand LLM Audit Guardrail (`Crawl all Prompts`)**: On-demand Multi-LLM audits across all prompts can be initiated via the top action bar with pre-flight cost calculation at **\~\$0.0089 / prompt** (auditing 4 AI engines) and mandatory OTP confirmation. * **Single, Bulk, & Domain Refresh Support**: Operators can refresh a single keyword, a subset of selected keywords, or the entire active keyword inventory of a domain with itemized pre-flight cost calculation. * **Pre-Flight Estimation**: The system calculates the exact expenditure based on the number of targets and specific audit type before issuing a crawl request. *** ## 4. Pre-Flight Financial Approval & OTP Verification Modal All manual crawling operations (`Crawl SERP Keywords` and `Crawl all Prompts`) require mandatory cryptographic confirmation via a dedicated **Crawl OTP Approval Modal**: ``` ┌────────────────────────────────────────────────────────┐ │ Confirm Manual Crawl / Prompt Audit │ │ │ │ Target: 1 Prompt / Keyword (ypym.app) │ │ Coverage: 4 Engines (ChatGPT, Gemini, Perplexity, ...)│ │ Domain: ypym.app │ │ │ │ Estimated Cost: │ │ • SERP Crawl: Rp 63 ($0.0035 / keyword) │ │ • Multi-LLM Audit: Rp 159 (~$0.0089 / prompt) │ │ │ │ 6-Digit Verification Code: [ ______ ] │ │ (Dispatched to authorized operator email) │ │ OTP Validity: 10:00 minutes │ │ │ │ [ Cancel ] [ Verify & Execute ] │ └────────────────────────────────────────────────────────┘ ``` ### Security & Operational Rules for OTP: 1. **Binding Context**: The generated 6-digit OTP is cryptographically bound to the authenticated user, tenant ID, target domain, selected keyword or prompt IDs, and calculated financial estimate (`challengeId`). 2. **Short-Lived Expiration**: The OTP is valid for exactly 10 minutes. Unconsumed challenges are automatically invalidated. 3. **Single-Use Consumption**: Once verified, the challenge is immediately consumed in a transactional database state, preventing replay attacks. 4. **Zero Bypass**: If the OTP is invalid, expired, or submitted outside allowable windows, the crawling execution fails closed (`HTTP 403 / 409`). *** ## 5. TimescaleDB Crawl Cost Ledger & Job History Every crawl transaction is recorded immutably in the `crawl_cost_ledger` hypertable and `crawl_jobs` tracking table: | Column | Type | Description | | - | - | - | | `id` | `UUID` | Unique transaction identifier | | `domain_id` | `INTEGER` | Monitored domain reference | | `keyword_count` | `INTEGER` | Number of keywords dispatched in the batch | | `cost_usd` | `NUMERIC(10, 4)` | Exact USD billed by the provider | | `cost_idr` | `NUMERIC(12, 2)` | IDR equivalent at recorded exchange rate | | `exchange_rate` | `NUMERIC(10, 2)` | Benchmark USD/IDR rate at execution | | `trigger_type` | `VARCHAR(32)` | Execution trigger: `scheduled_cron`, `manual_queue`, or `admin_override` | | `created_at` | `TIMESTAMPTZ` | Timestamp partitioned automatically into 30-day chunks | ### Crawl Execution History Modal Operators can inspect historical execution jobs via the **Crawl History** modal directly from the dashboard action bar. The modal provides a real-time audit log of both Google SERP and Multi-LLM crawl jobs, detailing timestamps, target volumes, execution duration, and engine status. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.