SEP-010: REST API

Contents

  1. Introduction
  2. Operations
    1. Get list of available spiders
    2. Get list of closed spiders
    3. Get list of scheduled spiders
    4. Get list of running spiders
    5. Schedule spider
    6. Close spider
    7. Get global stats
    8. Get spider-specific stats
    9. Get engine status

SEP:10
Title:REST API
Author:Pablo Hoffman
Created:2009-11-16
StatusIn progress (being written)

Introduction

This SEP proposes a JSON REST API for controlling Scrapy in server-mode, which is launched with: scrapy-ctl.py start

Operations

Get list of available spiders

GET /spiders/all

Get list of closed spiders

GET /spiders/closed

Get list of scheduled spiders

GET /spiders/scheduled

  • note: contains closed

Get list of running spiders

GET /spiders/opened

  • returns list of dicts containing spider id and domain_name

Schedule spider

POST /spiders

  • args: schedule=example.com

Close spider

POST /spider/1238/close

Get global stats

GET /stats

  • note: spider-specific not included

Get spider-specific stats

GET /spider/1238/stats/

Get engine status

GET /engine/status