Overview
The Scraper configuration panel controls the engine, scheduling, and depth parameters used to fetch organic search rankings and AI Overview positions. DBX is built around enterprise predictability: managing search engine crawling with zero unexpected cost spikes, fine-grained rate limits, and multi-tier scraping strategies.
YPYM DBX Scraper Settings panel showcasing the official DataForSEO engine, crawl frequencies, scrape delays, and intelligent pagination strategies.
1. Scraping Engine & Methods
DBX supports multiple underlying crawler backends, with DataForSEO serving as the primary production standard:In production enterprise deployments, DataForSEO is strongly recommended due to its hyper-local Indonesian criterion mapping (343 regencies/cities) and sub-penny cost efficiency ($0.0006 per SERP query).
2. Crawl Scheduling & Rate Limiting
Crawl timing directly balances fresh ranking insights with provider API expenditure.select
default:"daily"
Determines how often the platform scheduler initiates rank checks across your active keyword catalog:
- Daily: Standard production cadence (synchronized with YPYM’s daily 11:00 AM WIB crawl window).
- Every Other Day: Crawls every 48 hours for cost-optimized tracking.
- Weekly: Recommended for broad keyword catalogs with low day-to-day volatility.
- Monthly: Long-term macro position auditing.
- Never: Disables automated cron runs (manual single-keyword refreshes remain available).
select
default:"No Delay"
Enforces a throttled pause between sequential keyword requests (
No Delay, 5s, 10s, up to 30m).When using API-based engines like DataForSEO, No Delay is standard because requests are batched asynchronously. For self-hosted proxy pools, set a delay of 5s to 10s to prevent search engine IP rate limiting.boolean
default:"false"
When enabled, any keyword crawl that fails due to network timeouts, proxy errors, or transient CAPTCHAs is automatically enqueued into a dedicated retry queue.If items accumulate in the retry buffer, administrators can purge them using the Clear Failed Queue button.
3. Scrape Depth Strategies
Search engines display 10 organic results per page. DBX offers three distinct search depth strategies to balance coverage with crawling credits:Basic (Top 10)
Fastest & Most EconomicalScrapes only the first page (top 10 results). Consumes exactly 1 API credit per keyword. Ideal for core enterprise terms where your brand already ranks on page 1.
Custom (1–10 Pages)
Predictable Deep SearchCrawls a fixed number of pages (e.g., 5 pages = top 50 results checked) on every run, regardless of historical position.
Smart (Adaptive)
Intelligent Cost-SavingCrawls the page where your keyword was last observed, plus adjacent neighbor pages. Drops crawl cost significantly for keywords ranking on pages 2–5.