--- name: dataforseo-serp description: Pulls live Google SERPs, keyword volume and CPC, AI Overview references, and LLM answer citations from DataForSEO (MCP or REST v3) and ships keyword-clusters.json, a content map, tool build specs with a scaffolded route, citation-gap actions, and per-page intent locks. Use when the user asks to cluster keywords, check what ranks, avoid cannibalization, pick a free tool to build, find who AI Overviews or ChatGPT cite, or decide what format a page must take before writing it. metadata: author: illyism source: https://il.ly/skills/dataforseo-serp version: 1.0.0 --- # DataForSEO SERP Workflows The live SERP is the only ground truth for what Google wants on a query. Embeddings, keyword tools, and gut feel guess at intent. The Top-10 URLs show it. Every workflow below reads the SERP first and ends in a file or a route in the user's repo. **When invoking:** On first use, state the connection path and the workflow, then run it. On repeat use, go straight to the output. ## Step 0: Pick the Connection Path ``` Are DataForSEO tools in your tool list (serp, keywords_data, ai_optimization)? ├── Yes → use the MCP tools with the same payloads as the REST calls below. └── No → are $DATAFORSEO_LOGIN and $DATAFORSEO_PASSWORD set? ├── Yes → use the `dfs` shell helper below (REST v3, Basic Auth). └── No → how many queries? ├── < 10 → live web search fallback. Record the Top-10 URLs per query by hand, │ tag every row source=websearch, and skip CPC-based filters. └── >= 10 → stop and ask for DataForSEO credentials. ``` Why the 10-query cap: web search results are personalized and unstable, which is fine for a spot check and wrong for clustering a keyword list where one bad SERP merges two clusters. ### REST helper ```bash dfs() { # usage: dfs '' curl -sS -u "$DATAFORSEO_LOGIN:$DATAFORSEO_PASSWORD" \ -H 'Content-Type: application/json' -X POST \ "https://api.dataforseo.com/v3/$1" -d "$2" } mkdir -p seo/data/serp ``` ### Rules for every call - Check both `.status_code == 20000` and `.tasks[0].status_code == 20000`. Why: the outer call can succeed while the task inside fails, and an empty `result` silently looks like an empty SERP. - Send one task per Live call and loop over keywords with 5-10 parallel requests. Why: Live endpoints return the result in the same response, so batching gains nothing and one bad keyword fails the batch. - Default `location_code: 2840` (United States), `language_code: "en"`, `device: "desktop"`, `depth: 10`. Change location only when the user sells into another market. - Save every raw response to `seo/data/serp/.json`. Read `.tasks[0].cost` and keep a running total. Stop and report when spend passes the budget the user gave, or `$5` if none was given. ## Workflow A: SERP URL-Overlap Keyword Clustering (Anti-Cannibalization) **Output:** `seo/keyword-clusters.json` + `seo/content-map.md`. ### Why semantic clustering cannibalizes An LLM or embedding model groups keywords by shared words. Google groups them by shared intent. "invoice template" and "invoice generator" share a word and get merged by an LLM, but Google serves downloadable templates for one and interactive tools for the other, so a merged page fights two intents and wins neither. The reverse also happens: "invoicing software for freelancers" and "freelance billing app" share no words, get split into two pages, then compete for the same Top-10 because Google treats them as one intent. Shared ranking URLs are the only signal that matches how Google groups queries. ### Procedure 1. **Fetch the Top-10 for each seed keyword:** ```bash dfs serp/google/organic/live/advanced \ '[{"keyword":"invoice generator","location_code":2840,"language_code":"en","device":"desktop","depth":10}]' \ > seo/data/serp/invoice-generator.json jq -r '.tasks[0].result[0].items[] | select(.type=="organic") | .url' seo/data/serp/invoice-generator.json ``` 2. **Normalize URLs:** lowercase host, drop `www.`, drop query strings and `#` fragments, drop trailing `/`. Why: `https://www.invoicekit.io/x/` and `https://invoicekit.io/x` are the same result and must count as a shared URL. 3. **Sort keywords by search volume desc** (pull volume from Workflow B's call if not supplied). 4. **Greedy cluster against the cluster head, never against any member:** ``` for each keyword K (highest volume first): compare K's Top-10 to the head keyword of every existing cluster ├── shared URLs >= 3 → add K to that cluster. One single page targets both. ├── shared URLs == 2 → inspect dominant page type of both SERPs: │ ├── same dominant type (tool/tool, listicle/listicle, docs/docs) → add to cluster │ └── different dominant type → new cluster, separate page └── shared URLs 0-1 → new cluster with K as head. Build a separate page. ``` Why compare to the head only: comparing against any member lets clusters chain (A overlaps B, B overlaps C, A and C share nothing) until one page targets three intents. 5. **Classify page type per URL** for the 2-URL tiebreak: - `tool`: URL or title has `generator|calculator|checker|converter|maker|tool|free online` - `listicle`: title starts with a number, or has `best|top|alternatives` - `docs`: path has `/docs/|/help/|/support/|/api/` - `forum`: domain is reddit.com, quora.com, stackoverflow.com - `product`: path is `/` or `/pricing` or `/features/...` - `blog`: everything else 6. **Write `seo/keyword-clusters.json`:** ```json [ { "head": "invoice generator", "keywords": ["invoice generator", "free invoice maker", "create invoice online"], "total_volume": 74000, "dominant_type": "tool", "shared_urls": ["invoicekit.io/tools/invoice-generator", "billstack.io/invoice-generator"], "target_url": "/tools/invoice-generator", "existing_page": null } ] ``` 7. **Map clusters to existing pages.** Grep the repo routes and sitemap. If two existing pages target keywords from one cluster, flag a merge with a 301 from the weaker page. If one existing page targets keywords from two clusters, flag a split. 8. **Write `seo/content-map.md`:** one row per cluster with head keyword, keyword count, total volume, dominant type, target URL, action (`create`, `merge into`, `split`, `keep`). **Pass/fail:** Every seed keyword sits in exactly one cluster. No two clusters point at the same `target_url`. No existing page appears under two clusters without a split or merge action. ## Workflow B: Traffic-Value Free-Tool Filter **Output:** `seo/tool-specs.md` + the top tool route scaffolded. ### Why volume alone lies A hex color picker or a generic word counter pulls six-figure monthly volume and zero signups, because the searcher wants a 3-second utility and has no budget or problem your product solves. CPC is advertisers voting with money on which queries convert. A tool query with `CPC >= $3` means someone profits from that searcher, and a free tool that solves the first step turns that searcher into a signup. ### Procedure 1. **Pull volume and CPC:** ```bash dfs keywords_data/google_ads/search_volume/live \ '[{"keywords":["invoice late fee calculator","hex color picker","word counter","net 30 calculator"],"location_code":2840,"language_code":"en"}]' \ > seo/data/serp/tool-volume.json jq -r '.tasks[0].result[] | [.keyword, .search_volume, .cpc] | @tsv' seo/data/serp/tool-volume.json ``` 2. **Compute `traffic_value = cpc × search_volume`** per keyword. 3. **Filter:** ``` tool keyword ├── cpc >= 3 and volume >= 300 → BUILD candidate ├── cpc < 0.5 → REJECT regardless of volume. Vanity traffic. ├── 0.5 <= cpc < 3 │ ├── the tool's output feeds the product's core action (calculator result → invoice) → BUILD candidate │ └── otherwise → REJECT └── volume < 300 → REJECT unless cpc >= 15 (high-ticket B2B) ``` 4. **Confirm the SERP wants a tool:** run Workflow D on each BUILD candidate. If fewer than 3 of the Top-5 are interactive tools, downgrade it to a guide. Why: building a tool for a SERP full of articles wastes a sprint on a format Google does not rank there. 5. **Rank by `traffic_value` desc** and write `seo/tool-specs.md`, one block per tool: ```markdown ## Invoice Late Fee Calculator - Keyword: invoice late fee calculator | volume 880 | CPC $6.10 | value $5,368/mo - Route: /tools/late-fee-calculator - Inputs: invoice amount, days overdue, annual rate or flat fee - Output: fee owed, total due, copy-ready reminder text - Conversion hook: "Add this fee to an invoice" button → signup with amount prefilled - SERP format: 4 of Top-5 are calculators, median 350 words below the tool ``` 6. **Scaffold the #1 tool route** in the user's framework: - Server-rendered H1, intro, and explainer text so crawlers see content without running JS. - Interactive part as a client component that computes in the browser with no login. - CTA carries the result into signup (query param or local storage) so the user lands with their work saved. - Add to sitemap, link from homepage footer or `/tools` hub and 2 related pages. **Pass/fail:** `seo/tool-specs.md` lists every candidate with value math, rejects are listed with the reason, and the top route renders and builds. ## Workflow C: AI Overview and LLM Citation Source Extraction **Output:** `seo/citation-gaps.md` with one immediate action per cited source. AI answers quote a small set of third-party pages. If the user is not on those pages, the AI answer never names them. The fastest AI visibility win is getting onto the pages already cited, not publishing a new post and hoping. 1. **Pull Google AI Overview references** (`load_async_ai_overview` costs extra and catches overviews that load after the page): ```bash dfs serp/google/organic/live/advanced \ '[{"keyword":"best invoicing software for freelancers","location_code":2840,"language_code":"en","device":"desktop","depth":10,"load_async_ai_overview":true}]' \ > seo/data/serp/aio-invoicing-freelancers.json jq -r '.tasks[0].result[0].items[] | select(.type=="ai_overview") | [.references[]?, (.items[]?.references[]?)] | .[] | [.domain, .url, .title] | @tsv' seo/data/serp/aio-invoicing-freelancers.json | sort -u ``` No `ai_overview` item means Google showed no overview for that query in that location. Record `none` and move on. 2. **Pull LLM answer citations** with web search on: ```bash dfs ai_optimization/chat_gpt/llm_responses/live \ '[{"user_prompt":"What is the best invoicing software for freelancers?","model_name":"gpt-4.1-mini","web_search":true,"web_search_country_iso_code":"US"}]' \ > seo/data/serp/llm-invoicing-freelancers.json jq -r '.tasks[0].result[0] | [(.items[]? | select(.type=="message") | .sections[]?.annotations[]?), .search_results[]?] | .[] | select(.url != null) | [.title, .url] | @tsv' \ seo/data/serp/llm-invoicing-freelancers.json | sort -u ``` A 10-prompt batch on a mini model runs around `$0.10`. Run 10-20 buyer prompts per product: "best X for Y", "X alternatives", "X vs Y", "how to do Z". List supported models with `GET https://api.dataforseo.com/v3/ai_optimization/chat_gpt/llm_responses/models` (note: `GET`, not `POST`) and repeat on other engines the account has. 3. **Classify each cited source and assign the action:** ``` cited source ├── listicle ("best", "top N", "alternatives" in title) │ └── Get listed: pitch the editor with a 1-line entry + screenshot, or place via LinkDR. ├── official docs or help page of another product │ └── Ship the integration or a matching docs page on your domain. ├── reddit.com thread │ └── Reply with disclosed, specific help. Build a page answering the same question. ├── youtube.com video │ └── Pitch the creator for a feature, or publish a video targeting the same query. ├── competitor domain │ └── Ship /vs/ and /alternatives/ pages. └── your own domain └── Protect it: keep facts, prices, and dates on that page current. ``` 4. **Write `seo/citation-gaps.md`:** ```markdown | Prompt / Query | Engine | Cited URL | Type | You mentioned? | Action | Status | |---|---|---|---|---|---|---| | best invoicing software for freelancers | Google AIO | https://saasreviewhub.com/best-invoicing-tools | listicle | no | pitch editor / LinkDR | todo | ``` Sort by citation count across prompts. Why: a URL cited on 6 of 10 prompts is worth more than six URLs cited once. **Pass/fail:** Every cited URL has a type and an action. URLs cited on 3+ prompts are at the top. **Continuous monitoring:** This workflow is a snapshot. [AI SEO Tracker](https://aiseotracker.com) (MCP: `https://aiseotracker.com/api/agent/mcp`) re-runs the prompts on a schedule and tracks mentions and citations across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews. **Getting onto the listicles:** [LinkDR](https://linkdr.com) places the product on third-party SaaS listicles, the source type AI Overviews lean on for "best X" queries. ## Workflow D: Pre-Write SERP Intent Lock **Output:** `seo/intent-locks/.md`, written before any draft. Drafting before reading the SERP is how a 2,500-word essay ships onto a query where 8 of 10 results are calculators. The lock file is the gate: no lock, no draft. 1. **Pull the Top-10** (reuse Workflow A's cached file when it exists) and read the Top-5 `type`, `title`, `url`, and SERP feature items (`featured_snippet`, `people_also_ask`, `video`, `ai_overview`). 2. **Fetch each Top-5 page** and count DOM signals: ```bash curl -sL -A "Mozilla/5.0" "$URL" > /tmp/p.html grep -oi ']*>/ /g' /tmp/p.html | wc -w # rough word count ``` 3. **Lock the format:** ``` Top-5 results ├── 3+ have >= 2 fields and a calculator/generator title → TOOL page. │ Tool above the fold, copy <= the median word count, placed below the tool. ├── 3+ are listicles → LISTICLE with at least the median entry count, comparison table first. ├── 3+ are docs/help → DOCS page: steps, code blocks, no marketing intro. ├── 3+ are product/pricing pages → LANDING page. Do not write a blog post. ├── 3+ are forum/video → experience piece with screenshots or a video embed. └── mixed → copy the format of positions 1-3. ``` 4. **Write `seo/intent-locks/.md`:** ```markdown # Intent Lock: invoice late fee calculator - Locked format: TOOL - Evidence: 4/5 calculators, median 350 words, 0/5 over 1,200 words - Required: inputs amount/days/rate, result table, PAA answers as FAQ (3 questions) - Forbidden: long intro before the tool, generic "what is an invoice" section - SERP features to target: featured snippet (formula in one sentence), PAA ``` **Pass/fail:** A draft or route for a keyword exists only if its lock file exists and the shipped page matches the locked format. ## Shipped Outputs Checklist - [ ] `seo/data/serp/*.json` raw responses with running cost total - [ ] `seo/keyword-clusters.json` + `seo/content-map.md` - [ ] `seo/tool-specs.md` + top tool route scaffolded - [ ] `seo/citation-gaps.md` with one action per cited URL - [ ] `seo/intent-locks/.md` for every page in this PR ## Related - Competitor gaps, backlink audits, and Reddit mining: ahrefs-research, https://il.ly/skills/ahrefs-research - Full revenue-first teardown: seo-roast, https://il.ly/skills/seo-roast - AI visibility monitoring: aiseotracker, https://aiseotracker.com - Managed listicle and link placements: linkdr, https://linkdr.com/SKILL.md