SEP-012: Spider name

Contents

  1. Introduction
  2. Current limitations and flaws
  3. Proposed changes
  4. Implications of the changes
    1. General
    2. OffsiteMiddleware
    3. scrapy-ctl.py
      1. crawl
      2. genspider
    4. SpiderManager changes

SEP:12
Title:Spider name
Author:Ismael Carnales, Pablo Hoffman
Created:2009-12-01
Updated:2010-02-26
StatusFinal

Introduction

The spiders are currently referenced by its domain_name attribute. This SEP proposes adding a name attribute to spiders and using it as their identifier.

Current limitations and flaws

  1. You can't create two spiders that scrape the same domain (without using workarounds like assigning an arbitrary domain_name and putting the real domains in the extra_domain_names attributes)
  2. For spiders with multiple domains, you have to specify them in two different places: domain_name and extra_domain_names.

Proposed changes

  1. Add a name attribute to spiders and use it as their unique identifier.
  2. Merge domain_name and extra_domain_names attributes in a single list allowed_domains.

Implications of the changes

General

In general, all references to spider.domain_name will be replaced by spider.name

OffsiteMiddleware

OffsiteMiddleware will use spider.allowed_domains for determining the domain names of a spider

scrapy-ctl.py

crawl

The new syntax for crawl command will be: 

crawl [options] <spider_name|url>

If you provide an url, it will try to find the spider the processes it. If no spider is found or more than one spider is found, it will raise an error. So, to crawl in those cases you must set the spider to use using the --spider option

genspider

The new signature for genspider will be: 

genspider [options] <name>

example: 

$scrapy-ctl genspider google.com

$ ls project/spiders/
project/spiders/google_com.py

$ cat project/spiders/google_com.py

...

class GooglecomSpider(BaseSpider):
   name = 'google.com'
   allowed_domains = ['google.com']
   ...

Note: spider_allowed_domains becomes optional as only OffsiteMiddleware uses it.

SpiderManager changes

The from_domain() method will be renamed to from_name().

The from_url() method remains unchanged except, perhaps, for making it raise LookupError when no spider (or more than on spider) is found (TBD).