ScrapeGraphAI

ScrapeGraphAI ("You Only Scrape Once") transforms traditional web scraping into an automated pipeline driven by natural language prompts. Its core uses a directed graph architecture, encapsulating scraping, parsing, and LLM inference as graph nodes executed in sequence, outputting structured JSON data without writing CSS selectors.

Core Features

• Natural Language Driven: Extract data simply by providing a Prompt and a URL/file.
• Preset Graph Pipelines: Offers ready-to-use pipelines like SmartScraperGraph (single page), SearchGraph (search multi-page), SpeechGraph (to audio), and ScriptCreatorGraph (generates reusable scripts).
• Multi-model & Localization: Supports cloud APIs (OpenAI/Groq/Gemini) and local models via Ollama for privacy compliance.
• Anti-detection Rendering: Built-in Playwright and undetected-playwright engines for dynamic page rendering.

Architecture & Workflow Built on LangChain's graph pipeline mechanism, data is passed between nodes via state. The scraping layer uses Playwright, the parsing layer relies on BeautifulSoup and html2text, and text chunking is handled by semchunk. Supports visual tracing via Burr.

Use Cases Widely used for data acquisition preprocessing in AI Agent and RAG systems, no-code rapid data research, batch extraction from search engine results, and large-scale data access via enterprise hosted APIs.

Quick Start After running pip install scrapegraphai and playwright install, instantiate SmartScraperGraph with LLM model configurations and call the run() method to get results. Supports containerized deployment via Docker.

Unconfirmed Information

• Academic paper: citation.cff exists but no formally published paper link found
• v2.0.0 release status: pyproject.toml shows 2.0.0, PyPI latest is still 1.76.0
• Python version compatibility: pyproject.toml requires >=3.12, PyPI states >=3.10, >=3.12 is recommended
• n8n integration link points to localhost, formal integration pending confirmation