> ## Documentation Index
> Fetch the complete documentation index at: https://hub.ypym.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Automated Keyword Mapping & Local AI Semantic Discovery

> Technical architecture and operational guide for YPYM DBX v5.5.2 Automated Keyword Mapping, landing page extraction, local Ollama Qwen 2.5 AI clustering, and dual-bid CPC Google Ads intelligence.

## Executive Overview (DBX v5.5.2)

In enterprise digital marketing, keyword research is often fragmented between isolated tools: web scrapers, third-party keyword generators, Google Ads Keyword Planner spreadsheets, and rank trackers. This disconnect results in keyword cannibalization, missed topical clusters, and uncoordinated URL targeting.

**YPYM DBX v5.5.2** introduces **Automated Keyword Mapping** (`/keyword-discovery`), a linear 4-stage pipeline that bridges product positioning to media planning and rank tracking:

1. **Zero-Third-Party Landing Page Extraction**: Captures product names, value propositions, key features, and competitor benchmarks directly from HTML markup.
2. **Local Ollama AI Semantic Engine**: Formulates foundational root seeds using self-hosted Qwen 2.5 models without sending confidential briefs to external commercial APIs.
3. **Dual-Bid CPC Intelligence**: Connects to Google Ads Keyword Planner historical data, providing exact search volumes and both `Min CPC` (low-range bid) and `Max CPC` (high-range bid) in Indonesian Rupiah (IDR).
4. **Cannibalization Guardrails & URL Slug Mapping**: Automatically maps queries to relative target slugs, distinguishing the **Primary Target** keyword from **Supporting LSI** semantic variants on shared URLs.

***

## Technical Architecture

The module operates as a decoupled, multi-tier pipeline inside the YPYM DBX platform monorepo:

```mermaid theme={null}
flowchart TD
    subgraph Input["Stage 1: Initial Brief"]
        A["Product Name & Business Category"] --> B["Target Landing Page URL"]
        B --> C["1-10 Competitor Benchmark URLs"]
    end

    subgraph Crawler["Stage 2: Semantic Crawl & AI Generation"]
        C --> D["Local HTML Crawler: Extracts H1-H3, Features, Metadata"]
        D --> E["Local Ollama Engine: Qwen 2.5 Formulates 4-5 Root Seeds"]
        E --> F["Google Ads API v26: Historical Volumes & Dual CPC Bids"]
    end

    subgraph Review["Stage 3: Review & Selection"]
        F --> G["Topical Clustering & Search Intent Classification"]
        G --> H["Relative URL Slug Mapping (Primary vs Supporting LSI)"]
        H --> I["Optimistic UI Checkbox Curation"]
    end

    subgraph Output["Stage 4: Final Approval & Export"]
        I --> J["Approved Final Keywords Table"]
        J --> K1["Download .txt (Keywords Only)"]
        J --> K2["Download .csv (All 9 Data Columns)"]
        J --> K3["Export Formatted JSON Payload"]
        J --> K4["Direct Promotion to SERP / AIO Rank Tracker"]
    end

    style Input fill:#f8faff,stroke:#1A4BFF,stroke-width:1.5px
    style Crawler fill:#f8faff,stroke:#1A4BFF,stroke-width:1.5px
    style Review fill:#f8faff,stroke:#1A4BFF,stroke-width:1.5px
    style Output fill:#f8faff,stroke:#1A4BFF,stroke-width:1.5px
```

***

## The 4-Stage Operational Workflow

The interface is structured into four sequential tabs, accessible directly above the main workspace canvas:

### 1. Initial Brief & Product Profile (Tab 1)

Users define the core product parameters:

* **Product Name & Category**: Core identifying label and industry taxonomy.
* **Landing Page URL**: The target URL crawled to extract semantic headings, copy, and positioning.
* **Target User Segment**: The primary buyer persona (e.g. UMKM, Enterprise, B2B Procurement).
* **Competitor URLs**: Between 1 and 10 competitor reference links used to expand topical coverage.
* **Brand Terms & Aliases**: Dedicated list of trademarked phrases to isolate branded from generic search queries.
* **1-Click Brief Loader**: Includes a `Load Example Brief` shortcut for instant pre-population during initial onboarding and testing.

### 2. Live Logs & Raw Candidate Extraction (Tab 2)

Once analysis starts, the crawler executes asynchronously:

* **Live Terminal Telemetry**: Displays timestamped operational logs (`[INFO]`, `[SUCCESS]`, `[WARN]`) for HTTP fetching, HTML tag extraction, Ollama inference, and Google Ads metrics matching.
* **Raw Candidates Table**: Lists all candidate terms before curation with search volume, dual CPC benchmarks (`Min CPC` and `Max CPC`), competition level, search intent, and data origin (`AI Seed` vs `Google Ads`).
* **Raw Keyword Management**: Users can delete low-relevance raw terms directly before proceeding.

### 3. Review & Keyword Curation (Tab 3)

The curation workspace allows users to filter, inspect, and approve search queries:

* **Search Intent Filters**: Segment candidates by `Commercial`, `Transactional`, `Informational`, or `Navigational` intent.
* **Topical Clusters**: Group keywords by business vertical (e.g. `Finance, Banking Cluster`, `Features & Operational`, `Brand & Core Solution`).
* **Suggested URL Slug**: Shows the relative target path (e.g. `/pinjaman/nobu-kur` or `/pinjaman/nobu-kur/pricing`).
* **Cannibalization Guardrails**: When multiple keywords share the same URL slug, DBX automatically assigns roles:
  * **`Primary Target`**: The highest-volume query assigned as the main focus keyword for page title and `H1`.
  * **`Supporting LSI`**: Secondary semantic variants intended to be woven naturally into the body copy of the same page.
* **Optimistic Checkbox Selection**: Checkbox toggles update the local interface instantly (0ms latency) while persisting state changes to PostgreSQL in the background.

### 4. Approved Final Keywords & Export Suite (Tab 4)

Approved queries transition into the final repository:

* **Executive Metrics Summary**: Real-time totals for Approved Final Keywords, Aggregated Monthly Search Volume, and Portfolio Average CPC.
* **Multi-Format Export Suite**:
  * **Download .txt**: Line-delimited keyword list for instant import into Google Ads Campaign Editor or Google Sheets.
  * **Download .csv**: Full 9-column tabular data export (`Keyword`, `Search Volume`, `Min CPC (IDR)`, `Max CPC (IDR)`, `Avg CPC (IDR)`, `Competition`, `Search Intent`, `Topical Cluster`, `Suggested URL Slug`).
  * **View JSON**: Formatted JSON modal for developer API consumption and programmatic ingest.

***

## Production Deployment & Local AI Engine Architecture

### Does AI Require a Separate Project on VPS?

**No.** The local AI engine does not require an independent Git repository or detached application deployment. DBX communicates with the AI engine over standard internal HTTP REST endpoints.

In production environments (Coolify, Docker Compose, or Bare-Metal VPS):

1. **Ollama Service Container**: Ollama runs as a container service alongside PostgreSQL, Redis, and DBX Web:
   ```yaml theme={null}
   services:
     ollama:
       image: ollama/ollama:latest
       container_name: ypym-dbx-ollama
       restart: unless-stopped
       ports:
         - "11434:11434"
       volumes:
         - ollama_storage:/root/.ollama
       environment:
         - OLLAMA_KEEP_ALIVE=24h
   ```

2. **Environment Variable Configuration**:
   Inside `apps/web/.env.production`, configure the inference endpoint:
   ```env theme={null}
   # Local Ollama Inference Endpoint
   OLLAMA_BASE_URL=http://ollama:11434
   KEYWORD_DISCOVERY_MODEL=qwen2.5:32b
   ```

3. **Recommended Model Sizing by Server Specifications**:
   * **GPU-Accelerated VPS (NVIDIA T4 / A10G / RTX 4090)**: Use `qwen2.5:32b` or `qwen2.5:14b` for deep semantic nuance and sub-2-second inference.
   * **CPU-Only VPS (8 vCPU / 16 GB RAM)**: Use `qwen2.5:7b-instruct-q4_K_M`. Requires only \~4.5 GB RAM and completes inference in 3 to 4 seconds.

4. **Zero-Downtime Deterministic Fallback**:
   If the Ollama service is temporarily restarting or downloading a model update, the backend pipeline automatically activates its built-in deterministic heuristic engine. The user's crawl and keyword generation workflow **never fails with a 500 server error**.

***

## Database Persistence & Audit Schema

Keyword mapping operations are persisted across dedicated PostgreSQL 16 tables:

| Table Name | Primary Responsibility |
| - | - |
| `keyword_products` | Stores reusable business entity profiles, categories, landing page URLs, and competitor lists scoped by `tenant_id`. |
| `keyword_discoveries` | Stores discovery lifecycle runs, input snapshots, and curatable `output_snapshot` JSON payloads. |
| `keyword_discovery_jobs` | Durable PostgreSQL job queue managing attempts, worker locks, and execution heartbeats. |
| `keyword_discovery_attempts` | Historical audit ledger logging each execution attempt, provider name, and completion status. |

***

## Summary of DBX v5.5.2 Enhancements

* **Dual-Bid CPC**: Low and high bid ranges calculated directly in IDR.
* **Pure Numerical Search Volume**: Clean digits formatted without arbitrary `/mo` labels.
* **Segmented Control Tabs**: Unified navigation matching standard DBX interfaces.
* **Optimistic State Management**: Zero-delay checkbox curation with background PostgreSQL synchronization.
* **Direct Export Options**: Clean `.txt`, full `.csv`, and developer-ready JSON.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.