Screaming Frog SEO Spider?

YESA focused first version is a sensible AI-assisted build. Keep the first release narrow and polish the workflow people repeat.

BUILD FIRST

One defined user and a repeatable end-to-end job
A small, clear data model with sign-in only if it is genuinely needed
Loading, empty, error, and success states that make the workflow feel complete

LEAVE FOR LATER

Large integration marketplaces
Multi-region scale and enterprise administration
Every power-user setting from the established product

RESEARCH NOTES

What this tool actually does.

Checked 2026-09-06

Screaming Frog SEO Spider is a desktop website crawler for Windows, macOS, and Linux used by SEOs for technical audits. It discovers URLs via hyperlinks using a breadth-first algorithm, crawling raw HTML by default or rendered HTML via headless Chromium for JavaScript-heavy sites (Angular, React, Vue). It reports 300+ SEO issues: broken links, redirect chains/loops, missing/duplicate/long titles and meta descriptions, exact (md5) and near-duplicate content, robots.txt/meta robots/X-Robots-Tag directives, canonicals, hreflang, pagination, structured data, word count/readability, security headers, mixed content, and URL issues. Users can extract data via XPath, CSS Path, or regex (up to 100 extractors), run custom JavaScript, generate XML sitemaps, visualize site architecture, compare crawls, schedule audits, and integrate Google Analytics, Search Console, PageSpeed Insights, and link-metr

SOURCES

RESEARCHED BUILD PROMPT

Give this to your coding agent.

Build a focused replacement for the Screaming Frog SEO Spider's core crawl-and-audit workflow: a desktop or CLI web crawler that starts from a seed URL (or an uploaded list of URLs), discovers internal links breadth-first, and produces a technical SEO audit. User-facing requirements: a crawl progress view; tabs/filters for internal URLs, external links, images, and issues; per-URL details (status code, title, meta description, H1/H2, word count, indexability, canonical, robots directives, redirect chain); an Issues panel grouping checks into errors/warnings/opportunities with explanations; export to CSV/XLSX; crawl limits (max URLs, max depth); and a 'List mode' for crawling a supplied URL list. Technical requirements: configurable politeness (rate limit, concurrency), custom user-agent and headers, robots.txt respect toggle, redirect-chain detection, duplicate-title/description detection, exact-duplicate detection via content hash, and optional headless-browser rendering (e.g., Playwright) for JS-heavy sites. Implementation approach: Node.js/TypeScript or Python with a queue-based crawler (Playwright + cheerio), SQLite for storage, and a simple web UI or TUI; keep checks as data-driven rules so the audit is extensible. Do not copy the Screaming Frog brand, logo, proprietary assets, or its UI layout; use an original name and interface. Out of scope: API integrations (GA/GSC/PSI, link metrics), AI prompts, scheduling, crawl comparison, sitemap generation, site visualizations, and commercial licensing. Deliver a README covering setup, how to run a crawl, configuration options, and important limitations (politeness, JS rendering cost, robots compliance, scale).

Visit the official product page