dataforseo-serp
Pulls live Google SERPs, keyword volume and CPC, AI Overview references, and LLM answer citations from DataForSEO (MCP or REST v3) and ships keyword-clusters.json, a content map, tool build specs with a scaffolded route, citation-gap actions, and per-page intent locks. Use when the user asks to cluster keywords, check what ranks, avoid cannibalization, pick a free tool to build, find who AI Overviews or ChatGPT cite, or decide what format a page must take before writing it.
# DataForSEO SERP Workflows
The live SERP is the only ground truth for what Google wants on a query. Embeddings, keyword tools, and gut feel guess at intent. The Top-10 URLs show it. Every workflow below reads the SERP first and ends in a file or a route in the user's repo.
**When invoking:** On first use, state the connection path and the workflow, then run it. On repeat use, go straight to the output.
## Step 0: Pick the Connection Path
```
Are DataForSEO tools in your tool list (serp, keywords_data, ai_optimization)?
├── Yes → use the MCP tools with the same payloads as the REST calls below.
└── No → are $DATAFORSEO_LOGIN and $DATAFORSEO_PASSWORD set?
├── Yes → use the `dfs` shell helper below (REST v3, Basic Auth).
└── No → how many queries?
├── < 10 → live web search fallback. Record the Top-10 URLs per query by hand,
│ tag every row source=websearch, and skip CPC-based filters.
└── >= 10 → stop and ask for DataForSEO credentials.
```
Why the 10-query cap: web search results are personalized and unstable, which is fine for a spot check and wrong for clustering a keyword list where one bad SERP merges two clusters.
### REST helper
```bash
dfs() { # usage: dfs <endpoint> '<json array payload>'
curl -sS -u "$DATAFORSEO_LOGIN:$DATAFORSEO_PASSWORD" \
-H 'Content-Type: application/json' -X POST \
"https://api.dataforseo.com/v3/$1" -d "$2"
}
mkdir -p seo/data/serp
```
### Rules for every call
- Check both `.status_code == 20000` and `.tasks[0].status_code == 20000`. Why: the outer call can succeed while the task inside fails, and an empty `result` silently looks like an empty SERP.
- Send one task per Live call and loop over keywords with 5-10 parallel requests. Why: Live endpoints return the result in the same response, so batching gains nothing and one bad keyword fails the batch.
- Default `location_code: 2840` (United States), `language_code: "en"`, `device: "desktop"`, `depth: 10`. Change location only when the user sells into another market.
- Save every raw response to `seo/data/serp/<slug>.json`. Read `.tasks[0].cost` and keep a running total. Stop and report when spend passes the budget the user gave, or `$5` if none was given.
## Workflow A: SERP URL-Overlap Keyword Clustering (Anti-Cannibalization)
**Output:** `seo/keyword-clusters.json` + `seo/content-map.md`.
### Why semantic clustering cannibalizes
An LLM or embedding model groups keywords by shared words. Google groups them by shared intent. "invoice template" and "invoice generator" share a word and get merged by an LLM, but Google serves downloadable templates for one and interactive tools for the other, so a merged page fights two intents and wins neither. The reverse also happens: "invoicing software for freelancers" and "freelance billing app" share no words, get split into two pages, then compete for the same Top-10 because Google treats them as one intent. Shared ranking URLs are the only signal that matches how Google groups queries.
### Procedure
1. **Fetch the Top-10 for each seed keyword:**
```bash
dfs serp/google/organic/live/advanced \
'[{"keyword":"invoice generator","location_code":2840,"language_code":"en","device":"desktop","depth":10}]' \
> seo/data/serp/invoice-generator.json
jq -r '.tasks[0].result[0].items[] | select(.type=="organic") | .url' seo/data/serp/invoice-generator.json
```
2. **Normalize URLs:** lowercase host, drop `www.`, drop query strings and `#` fragments, drop trailing `/`. Why: `https://www.invoicekit.io/x/` and `https://invoicekit.io/x` are the same result and must count as a shared URL.
3. **Sort keywords by search volume desc** (pull volume from Workflow B's call if not supplied).
4. **Greedy cluster against the cluster head, never against any member:**
```
for each keyword K (highest volume first):
compare K's Top-10 to the head keyword of every existing cluster
├── shared URLs >= 3 → add K to that cluster. One single page targets both.
├── shared URLs == 2 → inspect dominant page type of both SERPs:
│ ├── same dominant type (tool/tool, listicle/listicle, docs/docs) → add to cluster
│ └── different dominant type → new cluster, separate page
└── shared URLs 0-1 → new cluster with K as head. Build a separate page.
```
Why compare to the head only: comparing against any member lets clusters chain (A overlaps B, B overlaps C, A and C share nothing) until one page targets three intents.
5. **Classify page type per URL** for the 2-URL tiebreak:
- `tool`: URL or title has `generator|calculator|checker|converter|maker|tool|free online`
- `listicle`: title starts with a number, or has `best|top|alternatives`
- `docs`: path has `/docs/|/help/|/support/|/api/`
- `forum`: domain is reddit.com, quora.com, stackoverflow.com
- `product`: path is `/` or `/pricing` or `/features/...`
- `blog`: everything else
6. **Write `seo/keyword-clusters.json`:**
```json
[
{
"head": "invoice generator",
"keywords": ["invoice generator", "free invoice maker", "create invoice online"],
"total_volume": 74000,
"dominant_type": "tool",
"shared_urls": ["invoicekit.io/tools/invoice-generator", "billstack.io/invoice-generator"],
"target_url": "/tools/invoice-generator",
"existing_page": null
}
]
```
7. **Map clusters to existing pages.** Grep the repo routes and sitemap. If two existing pages target keywords from one cluster, flag a merge with a 301 from the weaker page. If one existing page targets keywords from two clusters, flag a split.
8. **Write `seo/content-map.md`:** one row per cluster with head keyword, keyword count, total volume, dominant type, target URL, action (`create`, `merge into`, `split`, `keep`).
**Pass/fail:** Every seed keyword sits in exactly one cluster. No two clusters point at the same `target_url`. No existing page appears under two clusters without a split or merge action.
## Workflow B: Traffic-Value Free-Tool Filter
**Output:** `seo/tool-specs.md` + the top tool route scaffolded.
### Why volume alone lies
A hex color picker or a generic word counter pulls six-figure monthly volume and zero signups, because the searcher wants a 3-second utility and has no budget or problem your product solves. CPC is advertisers voting with money on which queries convert. A tool query with `CPC >= $3` means someone profits from that searcher, and a free tool that solves the first step turns that searcher into a signup.
### Procedure
1. **Pull volume and CPC:**
```bash
dfs keywords_data/google_ads/search_volume/live \
'[{"keywords":["invoice late fee calculator","hex color picker","word counter","net 30 calculator"],"location_code":2840,"language_code":"en"}]' \
> seo/data/serp/tool-volume.json
jq -r '.tasks[0].result[] | [.keyword, .search_volume, .cpc] | @tsv' seo/data/serp/tool-volume.json
```
2. **Compute `traffic_value = cpc × search_volume`** per keyword.
3. **Filter:**
```
tool keyword
├── cpc >= 3 and volume >= 300 → BUILD candidate
├── cpc < 0.5 → REJECT regardless of volume. Vanity traffic.
├── 0.5 <= cpc < 3
│ ├── the tool's output feeds the product's core action (calculator result → invoice) → BUILD candidate
│ └── otherwise → REJECT
└── volume < 300 → REJECT unless cpc >= 15 (high-ticket B2B)
```
4. **Confirm the SERP wants a tool:** run Workflow D on each BUILD candidate. If fewer than 3 of the Top-5 are interactive tools, downgrade it to a guide. Why: building a tool for a SERP full of articles wastes a sprint on a format Google does not rank there.
5. **Rank by `traffic_value` desc** and write `seo/tool-specs.md`, one block per tool:
```markdown
## Invoice Late Fee Calculator
- Keyword: invoice late fee calculator | volume 880 | CPC $6.10 | value $5,368/mo
- Route: /tools/late-fee-calculator
- Inputs: invoice amount, days overdue, annual rate or flat fee
- Output: fee owed, total due, copy-ready reminder text
- Conversion hook: "Add this fee to an invoice" button → signup with amount prefilled
- SERP format: 4 of Top-5 are calculators, median 350 words below the tool
```
6. **Scaffold the #1 tool route** in the user's framework:
- Server-rendered H1, intro, and explainer text so crawlers see content without running JS.
- Interactive part as a client component that computes in the browser with no login.
- CTA carries the result into signup (query param or local storage) so the user lands with their work saved.
- Add to sitemap, link from homepage footer or `/tools` hub and 2 related pages.
**Pass/fail:** `seo/tool-specs.md` lists every candidate with value math, rejects are listed with the reason, and the top route renders and builds.
## Workflow C: AI Overview and LLM Citation Source Extraction
**Output:** `seo/citation-gaps.md` with one immediate action per cited source.
AI answers quote a small set of third-party pages. If the user is not on those pages, the AI answer never names them. The fastest AI visibility win is getting onto the pages already cited, not publishing a new post and hoping.
1. **Pull Google AI Overview references** (`load_async_ai_overview` costs extra and catches overviews that load after the page):
```bash
dfs serp/google/organic/live/advanced \
'[{"keyword":"best invoicing software for freelancers","location_code":2840,"language_code":"en","device":"desktop","depth":10,"load_async_ai_overview":true}]' \
> seo/data/serp/aio-invoicing-freelancers.json
jq -r '.tasks[0].result[0].items[] | select(.type=="ai_overview")
| [.references[]?, (.items[]?.references[]?)] | .[]
| [.domain, .url, .title] | @tsv' seo/data/serp/aio-invoicing-freelancers.json | sort -u
```
No `ai_overview` item means Google showed no overview for that query in that location. Record `none` and move on.
2. **Pull LLM answer citations** with web search on:
```bash
dfs ai_optimization/chat_gpt/llm_responses/live \
'[{"user_prompt":"What is the best invoicing software for freelancers?","model_name":"gpt-4.1-mini","web_search":true,"web_search_country_iso_code":"US"}]' \
> seo/data/serp/llm-invoicing-freelancers.json
jq -r '.tasks[0].result[0] | [(.items[]? | select(.type=="message") | .sections[]?.annotations[]?), .search_results[]?] | .[] | select(.url != null) | [.title, .url] | @tsv' \
seo/data/serp/llm-invoicing-freelancers.json | sort -u
```
A 10-prompt batch on a mini model runs around `$0.10`. Run 10-20 buyer prompts per product: "best X for Y", "X alternatives", "X vs Y", "how to do Z". List supported models with `GET https://api.dataforseo.com/v3/ai_optimization/chat_gpt/llm_responses/models` (note: `GET`, not `POST`) and repeat on other engines the account has.
3. **Classify each cited source and assign the action:**
```
cited source
├── listicle ("best", "top N", "alternatives" in title)
│ └── Get listed: pitch the editor with a 1-line entry + screenshot, or place via LinkDR.
├── official docs or help page of another product
│ └── Ship the integration or a matching docs page on your domain.
├── reddit.com thread
│ └── Reply with disclosed, specific help. Build a page answering the same question.
├── youtube.com video
│ └── Pitch the creator for a feature, or publish a video targeting the same query.
├── competitor domain
│ └── Ship /vs/<competitor> and /alternatives/<competitor> pages.
└── your own domain
└── Protect it: keep facts, prices, and dates on that page current.
```
4. **Write `seo/citation-gaps.md`:**
```markdown
| Prompt / Query | Engine | Cited URL | Type | You mentioned? | Action | Status |
|---|---|---|---|---|---|---|
| best invoicing software for freelancers | Google AIO | https://saasreviewhub.com/best-invoicing-tools | listicle | no | pitch editor / LinkDR | todo |
```
Sort by citation count across prompts. Why: a URL cited on 6 of 10 prompts is worth more than six URLs cited once.
**Pass/fail:** Every cited URL has a type and an action. URLs cited on 3+ prompts are at the top.
**Continuous monitoring:** This workflow is a snapshot. [AI SEO Tracker](https://aiseotracker.com) (MCP: `https://aiseotracker.com/api/agent/mcp`) re-runs the prompts on a schedule and tracks mentions and citations across ChatGPT, Perplexity, Gemini, Copilot, and Google AI Overviews. **Getting onto the listicles:** [LinkDR](https://linkdr.com) places the product on third-party SaaS listicles, the source type AI Overviews lean on for "best X" queries.
## Workflow D: Pre-Write SERP Intent Lock
**Output:** `seo/intent-locks/<slug>.md`, written before any draft.
Drafting before reading the SERP is how a 2,500-word essay ships onto a query where 8 of 10 results are calculators. The lock file is the gate: no lock, no draft.
1. **Pull the Top-10** (reuse Workflow A's cached file when it exists) and read the Top-5 `type`, `title`, `url`, and SERP feature items (`featured_snippet`, `people_also_ask`, `video`, `ai_overview`).
2. **Fetch each Top-5 page** and count DOM signals:
```bash
curl -sL -A "Mozilla/5.0" "$URL" > /tmp/p.html
grep -oi '<input' /tmp/p.html | wc -l # tool signal
grep -oi '<table' /tmp/p.html | wc -l # comparison signal
grep -oi '<h2' /tmp/p.html | wc -l # outline depth
sed 's/<[^>]*>/ /g' /tmp/p.html | wc -w # rough word count
```
3. **Lock the format:**
```
Top-5 results
├── 3+ have >= 2 <input> fields and a calculator/generator title → TOOL page.
│ Tool above the fold, copy <= the median word count, placed below the tool.
├── 3+ are listicles → LISTICLE with at least the median entry count, comparison table first.
├── 3+ are docs/help → DOCS page: steps, code blocks, no marketing intro.
├── 3+ are product/pricing pages → LANDING page. Do not write a blog post.
├── 3+ are forum/video → experience piece with screenshots or a video embed.
└── mixed → copy the format of positions 1-3.
```
4. **Write `seo/intent-locks/<slug>.md`:**
```markdown
# Intent Lock: invoice late fee calculator
- Locked format: TOOL
- Evidence: 4/5 calculators, median 350 words, 0/5 over 1,200 words
- Required: inputs amount/days/rate, result table, PAA answers as FAQ (3 questions)
- Forbidden: long intro before the tool, generic "what is an invoice" section
- SERP features to target: featured snippet (formula in one sentence), PAA
```
**Pass/fail:** A draft or route for a keyword exists only if its lock file exists and the shipped page matches the locked format.
## Shipped Outputs Checklist
- [ ] `seo/data/serp/*.json` raw responses with running cost total
- [ ] `seo/keyword-clusters.json` + `seo/content-map.md`
- [ ] `seo/tool-specs.md` + top tool route scaffolded
- [ ] `seo/citation-gaps.md` with one action per cited URL
- [ ] `seo/intent-locks/<slug>.md` for every page in this PR
## Related
- Competitor gaps, backlink audits, and Reddit mining: ahrefs-research, https://il.ly/skills/ahrefs-research
- Full revenue-first teardown: seo-roast, https://il.ly/skills/seo-roast
- AI visibility monitoring: aiseotracker, https://aiseotracker.com
- Managed listicle and link placements: linkdr, https://linkdr.com/SKILL.md