DataForSEO Cost Governance & Crawl Mechanics (Option C)

Financial Governance Philosophy

In high-volume SERP tracking, scraping costs scale directly with keyword count, refresh frequency, and proxy architecture. Without strict financial guardrails, automated tools can trigger accidental balance depletion through runaway concurrent requests, duplicate refreshes, or micro-batch crawling. YPYM DBX introduces enterprise cost governance (Option C): a deterministic framework that balances real-time ranking freshness with predictable API expenditure and pre-flight budget approvals.

1. Ground-Truth Provider Pricing Models (Page 2 Standard)

YPYM DBX utilizes DataForSEO v3 SERP API as its primary scraping infrastructure. Tracking is calibrated by default to Page 2 (Top 20 results, depth: 20), delivering strategic visibility over immediate positions while avoiding 10-page micro-billing penalties. Costs are computed using exact provider unit economics:

Scraper Tier	Coverage Depth	Unit Cost (USD)	Unit Cost (IDR @ Rp 17,914)	Cost per 1,000 Targets	Primary Use Case in DBX
Google SERP Live	Page 2 (Top 20)	$0.0035	Rp 62.70	$3.50 / Rp 62,700	On-Demand SERP Crawls (`Crawl SERP Keywords`)
Google SERP Queue	Page 2 (Top 20)	$0.00105	Rp 18.80	$1.05 / Rp 18,800	Automated 11:00 AM WIB Daily Batch Crawls
Multi-LLM GEO Live	4 AI Engines (ChatGPT, Gemini, Perplexity, Claude)	~$0.0089	~Rp 159.00	~$8.90 / Rp 159,000	On-Demand Prompt Audits (`Crawl all Prompts`)

Benchmark exchange rate is calibrated to 1 USD = Rp 17,914 reflecting real payment settlement rates. SERP pricing includes Page 1 ($0.0020 Live / $0.0006 Queue) plus Page 2 ($0.0015 Live / $0.00045 Queue at 25% discount). Multi-LLM GEO pricing combines DataForSEO base fees ($0.0006 x 4) plus provider token costs across ChatGPT (gpt-4o), Gemini (gemini-1.5-pro), Perplexity (sonar), and Claude (claude-3-5-sonnet).

Live Balance Extraction

DBX queries the provider account balance directly in real-time via the /v3/appendix/user_data endpoint:

{
  "tasks": [
    {
      "result": [
        {
          "money": {
            "balance": 47.6543,
            "currency": "USD"
          }
        }
      ]
    }
  ]
}

This balance is continuously displayed in the Financial Status Bar, ensuring operators maintain full visibility into remaining crawling reserves before initiating runs.

2. Dual-Phase Automated Crawling & Hybrid Lifecycle

To eliminate unnecessary expenditure while supporting flexible ad-hoc research, DBX implements a coordinated Dual-Phase Automated Crawling Lifecycle:

                       ┌──────────────────────────────────────────────┐
                       │           Daily Automated Execution          │
                       └──────────────────────┬───────────────────────┘
                                              │
                     ┌────────────────────────┴────────────────────────┐
                     ▼                                                 ▼
       [ Phase 1: 11:00 AM WIB ]                         [ Phase 2: 11:30 AM WIB ]
       Google SERP + AI Overviews                        Multi-LLM GEO Intelligence
       • Bulk rank & position updates                    • ChatGPT, Gemini, Perplexity, Claude
       • Standard Priority Queue execution               • Automated prompt citation extraction
       • Scheduled keyword promotion                     • Zero ad-hoc operator overhead

Key Lifecycle Rules:

Phase 1 Automated Batch (11:00 AM WIB / 04:00 UTC):
- All active keywords are automatically dispatched in bulk using efficient batching.
- At the conclusion of the scheduled crawl, any pending on_queue keywords are promoted to active status.
Phase 2 Automated LLM Batch (11:30 AM WIB / 04:30 UTC):
- All active prompts are analyzed across the configured generative engines (ChatGPT Search, Gemini, Perplexity, Claude).
- Generative citations, source domains, and brand mentions are extracted and stored into the database.
Post-11:00 AM Quarantine (on_queue):
- Any keywords added after 11:00 AM WIB are tagged with state: "on_queue".
- They do not trigger immediate provider calls, preventing unintended credit burns.
Protected On-Demand Actions:
- Both manual SERP crawls (Crawl SERP Keywords) and Multi-LLM prompt audits (Crawl all Prompts) are safeguarded with pre-flight cost estimations and OTP confirmation.

3. Crawl Execution Schedule & Security Safeguards

To prevent uncontrolled API expenditure and ensure deterministic operations, on-demand and manual crawl actions are protected by strict temporal and operational guardrails:

Time-Window Guardrail (11:00–11:15 WIB for SERP): Manual SERP crawl requests (Crawl SERP Keywords, bulk selected keywords, and full domain refresh) are restricted to the primary daily maintenance window between 11:00 AM and 11:15 AM WIB at $0.0035 / keyword. Requests dispatched outside this 15-minute window are blocked server-side (HTTP 409 Conflict) to preserve budget stability and enforce synchronization with automated schedules.
On-Demand LLM Audit Guardrail (Crawl all Prompts): On-demand Multi-LLM audits across all prompts can be initiated via the top action bar with pre-flight cost calculation at ~$0.0089 / prompt (auditing 4 AI engines) and mandatory OTP confirmation.
Single, Bulk, & Domain Refresh Support: Operators can refresh a single keyword, a subset of selected keywords, or the entire active keyword inventory of a domain with itemized pre-flight cost calculation.
Pre-Flight Estimation: The system calculates the exact expenditure based on the number of targets and specific audit type before issuing a crawl request.

All manual crawling operations (Crawl SERP Keywords and Crawl all Prompts) require mandatory cryptographic confirmation via a dedicated Crawl OTP Approval Modal:

┌────────────────────────────────────────────────────────┐
│  Confirm Manual Crawl / Prompt Audit                   │
│                                                        │
│  Target: 1 Prompt / Keyword (ypym.app)                 │
│  Coverage: 4 Engines (ChatGPT, Gemini, Perplexity, ...)│
│  Domain: ypym.app                                      │
│                                                        │
│  Estimated Cost:                                       │
│  • SERP Crawl:       Rp 63   ($0.0035 / keyword)       │
│  • Multi-LLM Audit:  Rp 159  (~$0.0089 / prompt)       │
│                                                        │
│  6-Digit Verification Code: [  ______  ]               │
│  (Dispatched to authorized operator email)             │
│  OTP Validity: 10:00 minutes                           │
│                                                        │
│  [ Cancel ]                  [ Verify & Execute ]      │
└────────────────────────────────────────────────────────┘

Security & Operational Rules for OTP:

Binding Context: The generated 6-digit OTP is cryptographically bound to the authenticated user, tenant ID, target domain, selected keyword or prompt IDs, and calculated financial estimate (challengeId).
Short-Lived Expiration: The OTP is valid for exactly 10 minutes. Unconsumed challenges are automatically invalidated.
Single-Use Consumption: Once verified, the challenge is immediately consumed in a transactional database state, preventing replay attacks.
Zero Bypass: If the OTP is invalid, expired, or submitted outside allowable windows, the crawling execution fails closed (HTTP 403 / 409).

5. TimescaleDB Crawl Cost Ledger & Job History

Every crawl transaction is recorded immutably in the crawl_cost_ledger hypertable and crawl_jobs tracking table:

Column	Type	Description
`id`	`UUID`	Unique transaction identifier
`domain_id`	`INTEGER`	Monitored domain reference
`keyword_count`	`INTEGER`	Number of keywords dispatched in the batch
`cost_usd`	`NUMERIC(10, 4)`	Exact USD billed by the provider
`cost_idr`	`NUMERIC(12, 2)`	IDR equivalent at recorded exchange rate
`exchange_rate`	`NUMERIC(10, 2)`	Benchmark USD/IDR rate at execution
`trigger_type`	`VARCHAR(32)`	Execution trigger: `scheduled_cron`, `manual_queue`, or `admin_override`
`created_at`	`TIMESTAMPTZ`	Timestamp partitioned automatically into 30-day chunks

Operators can inspect historical execution jobs via the Crawl History modal directly from the dashboard action bar. The modal provides a real-time audit log of both Google SERP and Multi-LLM crawl jobs, detailing timestamps, target volumes, execution duration, and engine status.

Overview

Martech & SEO Platform

Security & Collaboration

Autonomous AI & Traffic Intelligence

A/B Test, Research

DataForSEO Cost Governance & Crawl Mechanics (Option C)

Financial Governance Philosophy

1. Ground-Truth Provider Pricing Models (Page 2 Standard)

Live Balance Extraction

2. Dual-Phase Automated Crawling & Hybrid Lifecycle

Key Lifecycle Rules:

3. Crawl Execution Schedule & Security Safeguards

Security & Operational Rules for OTP:

5. TimescaleDB Crawl Cost Ledger & Job History

​Financial Governance Philosophy

​1. Ground-Truth Provider Pricing Models (Page 2 Standard)

​Live Balance Extraction

​2. Dual-Phase Automated Crawling & Hybrid Lifecycle

​Key Lifecycle Rules:

​3. Crawl Execution Schedule & Security Safeguards

​4. Pre-Flight Financial Approval & OTP Verification Modal

​Security & Operational Rules for OTP:

​5. TimescaleDB Crawl Cost Ledger & Job History

​Crawl Execution History Modal

Financial Governance Philosophy

1. Ground-Truth Provider Pricing Models (Page 2 Standard)

Live Balance Extraction

2. Dual-Phase Automated Crawling & Hybrid Lifecycle

Key Lifecycle Rules:

3. Crawl Execution Schedule & Security Safeguards

4. Pre-Flight Financial Approval & OTP Verification Modal

Security & Operational Rules for OTP:

5. TimescaleDB Crawl Cost Ledger & Job History

Crawl Execution History Modal