AutomationCAN I VIBECODE
Diffbot?
BUILD FIRST
- A non-sensitive or read-only version of the core experience
- A manual review step wherever correctness or trust matters
- A narrow companion tool that removes a real piece of work
LEAVE FOR LATER
- Regulated decisions, payments, or irreversible actions
- Claims of correctness, compliance, or uptime that the MVP cannot support
- Sensitive integrations until security, support, and auditability are ready
RESEARCH NOTES
What this tool actually does.
Diffbot is an AI-powered web data extraction platform that converts unstructured web pages into structured JSON. Its flagship Extract API uses computer vision and NLP to classify any page into standard page types (article, product, list, etc.) and pull fields like offerPrice or author without site-specific rules, returning results in roughly 300ms. Related products: Bulk (asynchronous batch processing of URL lists), Crawl (site spidering that feeds discovered pages into Extract), Natural Language API (entities, relationships, facts, sentiment from raw text), and Knowledge Graph (10B+ crawled entities used for enrichment). It is positioned as a scalable alternative to fragile rule-based scrapers; a no-signup test drive exists at diffbot.com/testdrive. Pricing is usage-based via API tokens, and a dashboard UI supports plug-and-play use.
RESEARCHED BUILD PROMPT
Give this to your coding agent.
Build a focused replacement for Diffbot's core workflow: rule-less, ML-driven web page classification and structured extraction. Workflow: a user submits a URL (or a list of URLs, or a seed URL for crawling); the system fetches and renders the page, classifies it into a standard page type (article, product, list, etc.), extracts schema fields (e.g., title, author, offerPrice, description) into clean JSON, and returns it quickly; batch and crawl modes run asynchronously with status polling. User-facing requirements: a REST API with token auth, a simple web dashboard for single-URL testing, a JSON response schema per page type, and a README covering setup and limitations. Technical requirements: headless Chromium rendering (Playwright/Puppeteer), page-type classification via a hybrid of DOM heuristics and a fine-tuned transformer model, field extraction via DOM analysis plus an LLM fallback, a Redis-backed job queue (Bull) for Bulk/Crawl modes, Postgres for result storage, and caching/rate limiting. Do not copy Diffbot's brand, proprietary assets, or UI. Out of scope: a 10-billion-entity knowledge graph, web-wide crawling at commercial scale, entity resolution/sentiment parity with Diffbot's NLP API, and production-grade uptime. The README must document setup steps, the classification/extraction approach, and known limitations (e.g., JS-heavy pages, login-walled content, accuracy trade-offs).