Maintained by Zyte with over 500 other contributors

The world’s most-used open source data extraction framework

A collaborative, open source framework for extracting public web data.

Get started

Build Scrapy spiders with AI

Created by Zyte

Two agentic tools that scaffold standard Scrapy projects you fully control.

VS Code Extension

Web Scraping Copilot

Generate, test, and deploy production-ready Scrapy spiders without leaving your IDE — with full developer control.

Generate selectors
Unblock sites
Deploy to Scrapy Cloud

Install Learn more

Agent Skills

Claude Code Plugin

Turn plain-English prompts into a complete Scrapy project — from just a URL and a data type, no manual scaffolding.

Schema discovery
scrapy-poet page objects
pytest fixtures

Install Read the docs

How it works

A project built to scale

Scrapy scaffolds a full project, not a throwaway script — keeping your spiders, settings, items, and pipelines organized as your crawler grows. Run scrapy startproject to lay it out, then scrapy crawl to run.

quotes/                      # project root
├── scrapy.cfg               # deploy configuration
└── quotes/                  # the project's Python module
    ├── __init__.py
    ├── items.py             # define the data you extract
    ├── middlewares.py
    ├── pipelines.py         # clean, validate & store items
    ├── settings.py          # project-wide configuration
    └── spiders/             # your spiders live here
        ├── __init__.py
        └── quotes.py        # one spider per source

Fast & Powerful

An asynchronous engine crawls many pages at once, with retries, throttling, and caching handled for you.

Customizable

Shape every step of the crawl in pure Python with pluggable spiders, middlewares, and pipelines.

Open Source

Free and BSD-licensed, built in the open by 500+ contributors and trusted by millions of developers.

Batteries included

Everything you need, built in

Select a capability to see it in code.

Powerful selectors: Extract with CSS or XPath — mix them freely.
Crawl at scale: Async engine with smart, polite throttling.
Item pipelines: Validate, clean, and store every item.
Export anywhere: Feed exports to JSON, CSV, S3 and more.
Interactive shell: Test selectors live before writing code.
Extensible by design: Hook in middlewares, signals and extensions.

# CSS or XPath — your choice, even mixed
title = response.css("h1::text").get()
price = response.xpath("//p[@class='price_color']/text()").get()

# Get everything, with a sensible default
tags = response.css("div.tags a.tag::text").getall()
image = response.css("img::attr(src)").get(default="")

# Run regex straight off a selector
sku = response.css("p.sku::text").re_first(r"SKU:\s*(\w+)")

Selectors

Scrapy's built-in selectors let you query the DOM with CSS or XPath, extract attributes, set defaults, and even apply regex — all from one chainable API.

Learn more

Join the community

“ If it wasn't for Scrapy, my freelancing career, and then the scraping business would have never taken off. The Scrapy framework, and especially its documentation, simplifies crawling and scraping for anyone with basic Python skills. I don't know, now there is this emotive bond with Scrapy that I've developed over the years.”

Nishant Choudhary

Founder of DataFlirt.com

“ Scrapy: the gift that revolutionised web scraping and inspired me to pay it forward with Scrapoxy!"
"Scrapy is the central engine of all my projects, since its 1st release.”

Fabien Vauchelles

Creator of Scrapoxy

“ Scrapy stands out as an exceptional tool for seamlessly managing open requests, making large-scale data collection efficient and reliable."
"With its remarkable scraping speed, Scrapy transforms complex web extraction tasks into swift, streamlined operations that save time and maximise output.”

Hyder Khan

Flipdish

“ Scrapy is the cornerstone of web scraping with Python. Without it, scraping would be much harder."
"During my career, I've built thousands of scrapers, mostly with Scrapy. It made my life a lot easier.”

Pierluigi Vinciguerra

Co Founder and CTO at Databoutique.com

“ With over 6 years in web scraping, Scrapy has been my go-to framework for building fast, reliable, and scalable data pipelines."
"In my 6+ years of scraping experience, nothing has matched Scrapy’s balance of power, flexibility, and community support.”

Ganesh Satheendran

Turbolab Technologies

“ When Google Search enforced JavaScript, our scraper broke. Struggling to build a working alternative with Puppeteer on a short notice, we fired up Scrapy, Splash, and whipped up a script. Lo and behold – it worked! Our Google benchmark was buzzing again.”

Adam

Proxyway

Join millions of developers using Scrapy.

Fast, free web scraping backed by a thriving community.

Get started Explore docs