> ## Documentation Index
> Fetch the complete documentation index at: https://hub.ypym.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Scraper Engine & Crawl Strategy Settings

> Configure SERP scraping engines, crawl intervals, keyword delays, pagination limits, and smart crawl optimization strategies in YPYM DBX.

## Overview

The **Scraper** configuration panel controls the engine, scheduling, and depth parameters used to fetch organic search rankings and AI Overview positions.

DBX is built around enterprise predictability: managing search engine crawling with zero unexpected cost spikes, fine-grained rate limits, and multi-tier scraping strategies.

![YPYM DBX Scraper Settings](https://mintcdn.com/ypymcompany/jnPr8A4nNf-pWW8z/images/platform/dbx/scraper-setting.png?fit=max\&auto=format\&n=jnPr8A4nNf-pWW8z\&q=85\&s=ee123d67de36a21343e8f0b3475117b3)

YPYM DBX Scraper Settings panel showcasing the official DataForSEO engine, crawl frequencies, scrape delays, and intelligent pagination strategies.

***

## 1. Scraping Engine & Methods

DBX supports multiple underlying crawler backends, with **DataForSEO** serving as the primary production standard:

| Scraping Method | Status | Configuration Mode |
| :- | :- | :- |
| **DataForSEO (Official Engine)** | `Recommended` | Managed centrally through production server environment variables (`DATAFORSEO_LOGIN`, `DATAFORSEO_PASSWORD`). API credentials are locked away from the UI for security. |
| **Scraping Robot / SerpApi / Serper** | `Supported` | Configured by providing an API Key or Token directly in the input field. |
| **Custom Proxy Pool** | `Advanced` | Configured via line-separated IP addresses (`http://ip:port` or `https://user:pass@ip:port`). |

> [!NOTE]
>
> In production enterprise deployments, DataForSEO is strongly recommended due to its hyper-local Indonesian criterion mapping (343 regencies/cities) and sub-penny cost efficiency (\$0.0006 per SERP query).

***

## 2. Crawl Scheduling & Rate Limiting

Crawl timing directly balances fresh ranking insights with provider API expenditure.

```mermaid theme={null}
flowchart TD
    A["Scrape Trigger\n(Daily at 11:00 AM WIB)"] --> B{"Scrape Delay (e.g. No Delay / 5s)"}
    B --> C["Fetch Target Keyword SERP"]
    C --> D{"Scrape Success?"}
    D -- Yes --> E["Record Ranks into TimescaleDB"]
    D -- No --> F{"Auto Retry Enabled?"}
    F -- Yes --> G["Enqueue into Failed Queue"]
    F -- No --> H["Mark Keyword as Error / Unchanged"]
    
    style A fill:#1A4BFF,stroke:#1A4BFF,color:#fff
    style E fill:#30FFFC,stroke:#1A4BFF,color:#000
    style G fill:#f8f9fa,stroke:#e9ecef,color:#333
```

**Scraping Frequency** `select` (default: daily)

Determines how often the platform scheduler initiates rank checks across your active keyword catalog:

* **Daily**: Standard production cadence (synchronized with YPYM's daily 11:00 AM WIB crawl window).
* **Every Other Day**: Crawls every 48 hours for cost-optimized tracking.
* **Weekly**: Recommended for broad keyword catalogs with low day-to-day volatility.
* **Monthly**: Long-term macro position auditing.
* **Never**: Disables automated cron runs (manual single-keyword refreshes remain available).

*Note: Changing scraping frequency requires a background worker restart to update the master cron daemon.*

**Keyword Scrape Delay** `select` (default: No Delay)

Enforces a throttled pause between sequential keyword requests (`No Delay`, `5s`, `10s`, up to `30m`).

When using API-based engines like DataForSEO, **No Delay** is standard because requests are batched asynchronously. For self-hosted proxy pools, set a delay of **5s to 10s** to prevent search engine IP rate limiting.

**Auto Retry Failed Keyword Scrape** `boolean` (default: false)

When enabled, any keyword crawl that fails due to network timeouts, proxy errors, or transient CAPTCHAs is automatically enqueued into a dedicated retry queue.

If items accumulate in the retry buffer, administrators can purge them using the **Clear Failed Queue** button.

***

## 3. Scrape Depth Strategies

Search engines display 10 organic results per page. DBX offers three distinct search depth strategies to balance coverage with crawling credits:

> **Basic (Top 10)**
>
> **Fastest & Most Economical**
>
> Scrapes only the first page (top 10 results). Consumes exactly 1 API credit per keyword. Ideal for core enterprise terms where your brand already ranks on page 1.

> **Custom (1–10 Pages)**
>
> **Predictable Deep Search**
>
> Crawls a fixed number of pages (e.g., 5 pages = top 50 results checked) on every run, regardless of historical position.

> **Smart (Adaptive)**
>
> **Intelligent Cost-Saving**
>
> Crawls the page where your keyword was last observed, plus adjacent neighbor pages. Drops crawl cost significantly for keywords ranking on pages 2–5.

### Smart Strategy Fallback

When using the **Smart** strategy, enable **Full Fallback**:

```
[Toggle] Full Fallback: Scrape all pages if not found on nearby pages
```

If your keyword suffered a sharp drop and is no longer found in its expected neighborhood, DBX automatically expands the search across all 10 pages (top 100 results) to locate its new position.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.