Changeset 1849:1d0ac164cf62

Timestamp:

11/14/09 20:28:59 (9 months ago)

Author:

Pablo Hoffman <pablo@…>

Branch:

default

Message:

StatsCollector?: ported methods to receive spider instances (closes #113), removed list_domains() method, added iter_spider_stats() method

Files:

• docs/topics/stats.rst (modified) (7 diffs)
• scrapy/contrib/corestats.py (modified) (2 diffs)
• scrapy/contrib/downloadermiddleware/stats.py (modified) (3 diffs)
• scrapy/contrib/itemsampler.py (modified) (4 diffs)
• scrapy/contrib/pipeline/images.py (modified) (3 diffs)
• scrapy/contrib/spidermiddleware/depth.py (modified) (3 diffs)
• scrapy/contrib/statsmailer.py (modified) (2 diffs)
• scrapy/contrib/webconsole/stats.py (modified) (1 diff)
• scrapy/contrib_exp/spiderprofiler.py (modified) (1 diff)
• scrapy/core/engine.py (modified) (2 diffs)
• scrapy/core/scraper.py (modified) (6 diffs)
• scrapy/stats/collector/__init__.py (modified) (4 diffs)
• scrapy/stats/collector/mysql.py (modified) (2 diffs)
• scrapy/stats/collector/simpledb.py (modified) (1 diff)
• scrapy/stats/signals.py (modified) (1 diff)
• scrapy/tests/test_downloadermiddleware_stats.py (modified) (3 diffs)
• scrapy/tests/test_spidermiddleware_depth.py (modified) (4 diffs)
• scrapy/tests/test_stats.py (modified) (3 diffs)

docs/topics/stats.rst

@@ -9 +9 @@
 
 Scrapy provides a convenient service for collecting stats in the form of
-key/values, both globally and per spider/domain. It's called the Stats
-Collector, and it's a singleton which can be imported and used quickly, as
-illustrated by the examples in the :ref:`topics-stats-usecases` section below.
+key/values, both globally and per spider. It's called the Stats Collector, and
+it's a singleton which can be imported and used quickly, as illustrated by the
+examples in the :ref:`topics-stats-usecases` section below.
 
 The stats collection is enabled by default but can be disabled through the
@@ -27 +27 @@
 enabled) and extremely efficient (almost unnoticeable) when disabled.
 
-The Stats Collector keeps one stats table per open spider/domain and one global
-stats table. You can't set or get stats from a closed domain, but the
-domain-specific stats table is automatically opened when the spider is opened,
-and closed when the spider is closed.
+The Stats Collector keeps one stats table per open spider and one global stats
+table. You can't set or get stats from a closed spider, but the spider-specific
+stats table is automatically opened when the spider is opened, and closed when
+the spider is closed.
 
 .. _topics-stats-usecases:
@@ -62 +62 @@
     8
 
-Get all global stats from a given domain::
+Get all global stats (ie. not particular to any spider)::
 
     >>> stats.get_stats()
     {'hostname': 'localhost', 'spiders_crawled': 8}
 
-Set domain/spider specific stat value (domains must be opened first, but this
+Set spider specific stat value (spider stats must be opened first, but this
 task is handled automatically by the Scrapy engine)::
 
-    stats.set_value('start_time', datetime.now(), domain='example.com')
-
-Increment domain-specific stat value::
-
-    stats.inc_value('pages_crawled', domain='example.com')
-
-Set domain-specific stat value only if greater than previous::
-
-    stats.max_value('max_items_scraped', value, domain='example.com')
-
-Set domain-specific stat value only if lower than previous::
-
-    stats.min_value('min_free_memory_percent', value, domain='example.com')
-
-Get domain-specific stat value::
-
-    >>> stats.get_value('pages_crawled', domain='example.com')
+    stats.set_value('start_time', datetime.now(), spider=some_spider)
+
+Where ``some_spider`` is a :class:`~scrapy.spider.BaseSpider` object.
+
+Increment spider-specific stat value::
+
+    stats.inc_value('pages_crawled', spider=some_spider)
+
+Set spider-specific stat value only if greater than previous::
+
+    stats.max_value('max_items_scraped', value, spider=some_spider)
+
+Set spider-specific stat value only if lower than previous::
+
+    stats.min_value('min_free_memory_percent', value, spider=some_spider)
+
+Get spider-specific stat value::
+
+    >>> stats.get_value('pages_crawled', spider=some_spider)
     1238
 
-Get all stats from a given domain::
-
-    >>> stats.get_stats('pages_crawled', domain='example.com')
+Get all stats from a given spider::
+
+    >>> stats.get_stats('pages_crawled', spider=some_spider)
     {'pages_crawled': 1238, 'start_time': datetime.datetime(2009, 7, 14, 21, 47, 28, 977139)}
 
@@ -109 +111 @@
 .. class:: StatsCollector
     
-    .. method:: get_value(key, default=None, domain=None)
+    .. method:: get_value(key, default=None, spider=None)
  
         Return the value for the given stats key or default if it doesn't exist.
-        If domain is ``None`` the global stats table is consulted, other the
-        domain specific one is. If the domain is not yet opened a ``KeyError``
+        If spider is ``None`` the global stats table is consulted, otherwise the
+        spider specific one is. If the spider is not yet opened a ``KeyError``
         exception is raised.
 
-    .. method:: get_stats(domain=None)
-
-        Get all stats from the given domain/spider (if domain is given) or all
-        global stats otherwise, as a dict. If domain is not opened ``KeyError``
-        is raied.
-
-    .. method:: set_value(key, value, domain=None)
+    .. method:: get_stats(spider=None)
+
+        Get all stats from the given spider (if spider is given) or all global
+        stats otherwise, as a dict. If spider is not opened ``KeyError`` is
+        raied.
+
+    .. method:: set_value(key, value, spider=None)
 
         Set the given value for the given stats key on the global stats (if
-        domain is not given) or the domain-specific stats (if domain is given),
+        spider is not given) or the spider-specific stats (if spider is given),
         which must be opened or a ``KeyError`` will be raised.
 
-    .. method:: set_stats(stats, domain=None)
-
-        Set the given stats (as a dict) for the given domain. If the domain is
+    .. method:: set_stats(stats, spider=None)
+
+        Set the given stats (as a dict) for the given spider. If the spider is
         not opened a ``KeyError`` will be raised.
 
-    .. method:: inc_value(key, count=1, start=0, domain=None)
+    .. method:: inc_value(key, count=1, start=0, spider=None)
 
         Increment the value of the given stats key, by the given count,
-        assuming the start value given (when it's not set). If domain is not
-        given the global stats table is used, otherwise the domain-specific
+        assuming the start value given (when it's not set). If spider is not
+        given the global stats table is used, otherwise the spider-specific
         stats table is used, which must be opened or a ``KeyError`` will be
         raised.
 
-    .. method:: max_value(key, value, domain=None)
+    .. method:: max_value(key, value, spider=None)
 
         Set the given value for the given key only if current value for the
         same key is lower than value. If there is no current value for the
-        given key, the value is always set. If domain is not given the global
-        stats table is used, otherwise the domain-specific stats table is used,
+        given key, the value is always set. If spider is not given the global
+        stats table is used, otherwise the spider-specific stats table is used,
         which must be opened or a KeyError will be raised.
 
-    .. method:: min_value(key, value, domain=None)
+    .. method:: min_value(key, value, spider=None)
 
         Set the given value for the given key only if current value for the
         same key is greater than value. If there is no current value for the
-        given key, the value is always set. If domain is not given the global
-        stats table is used, otherwise the domain-specific stats table is used,
+        given key, the value is always set. If spider is not given the global
+        stats table is used, otherwise the spider-specific stats table is used,
         which must be opened or a KeyError will be raised.
 
-    .. method:: clear_stats(domain=None)
-
-        Clear all global stats (if domain is not given) or all domain-specific
-        stats if domain is given, in which case it must be opened or a
+    .. method:: clear_stats(spider=None)
+
+        Clear all global stats (if spider is not given) or all spider-specific
+        stats if spider is given, in which case it must be opened or a
         ``KeyError`` will be raised.
 
-    .. method:: list_domains()
-
-        Return a list of all opened domains.
-
-    .. method:: open_domain(domain)
-
-        Open the given domain for stats collection. This method must be called
-        prior to working with any stats specific to that domain, but this task
+    .. method:: iter_spider_stats()
+
+        Return a iterator over ``(spider, spider_stats)`` for each open spider
+        currently tracked by the stats collector, where ``spider_stats`` is the
+        dict containing all spider-specific stats.
+
+        Global stats are not included in the iterator. If you want to get
+        those, use :meth:`get_stats` method.
+
+    .. method:: open_spider(spider)
+
+        Open the given spider for stats collection. This method must be called
+        prior to working with any stats specific to that spider, but this task
         is handled automatically by the Scrapy engine.
 
-    .. method:: close_domain(domain)
-
-        Close the given domain. After this is called, no more specific stats
-        for this domain can be accessed. This method is called automatically on
+    .. method:: close_spider(spider)
+
+        Close the given spider. After this is called, no more specific stats
+        for this spider can be accessed. This method is called automatically on
         the :signal:`spider_closed` signal.
 
@@ -197 +204 @@
 
     A simple stats collector that keeps the stats of the last scraping run (for
-    each domain) in memory, which can be accessed through the ``domain_stats``
-    attribute
+    each spider) in memory, after they're closed. The stats can be accessed
+    through the :attr:`domain_stats` attribute, which is a dict keyed by spider
+    domain name.
 
     This is the default Stats Collector used in Scrapy.
@@ -204 +212 @@
     .. attribute:: domain_stats
 
-       A dict of dicts (keyed by domain) containing the stats of the last
-       scraping run for each domain.
+       A dict of dicts (keyed by spider domain name) containing the stats of
+       the last scraping run for each domain.
 
 DummyStatsCollector
@@ -284 +292 @@
    :synopsis: Stats Collector signals
 
-.. signal:: stats_domain_opened
-.. function:: stats_domain_opened(domain)
-
-    Sent right after the stats domain is opened. You can use this signal to add
-    startup stats for domain (example: start time).
-
-    :param domain: the stats domain just opened
-    :type domain: str
-
-.. signal:: stats_domain_closing
-.. function:: stats_domain_closing(domain, reason)
-
-    Sent just before the stats domain is closed. You can use this signal to add
+.. signal:: stats_spider_opened
+.. function:: stats_spider_opened(spider)
+
+    Sent right after the stats spider is opened. You can use this signal to add
+    startup stats for spider (example: start time).
+
+    :param spider: the stats spider just opened
+    :type spider: str
+
+.. signal:: stats_spider_closing
+.. function:: stats_spider_closing(spider, reason)
+
+    Sent just before the stats spider is closed. You can use this signal to add
     some closing stats (example: finish time).
 
-    :param domain: the stats domain about to be closed
-    :type domain: str
-
-    :param reason: the reason why the domain is being closed. See
+    :param spider: the stats spider about to be closed
+    :type spider: str
+
+    :param reason: the reason why the spider is being closed. See
         :signal:`spider_closed` signal for more info.
     :type reason: str
 
-.. signal:: stats_domain_closed
-.. function:: stats_domain_closed(domain, reason, domain_stats)
-
-    Sent right after the stats domain is closed. You can use this signal to
-    collect resources, but not to add any more stats as the stats domain has
-    already been close (use :signal:`stats_domain_closing` for that instead).
-
-    :param domain: the stats domain just closed
-    :type domain: str
-
-    :param reason: the reason why the domain was closed. See
+.. signal:: stats_spider_closed
+.. function:: stats_spider_closed(spider, reason, spider_stats)
+
+    Sent right after the stats spider is closed. You can use this signal to
+    collect resources, but not to add any more stats as the stats spider has
+    already been close (use :signal:`stats_spider_closing` for that instead).
+
+    :param spider: the stats spider just closed
+    :type spider: str
+
+    :param reason: the reason why the spider was closed. See
         :signal:`spider_closed` signal for more info.
     :type reason: str
 
-    :param domain_stats: the stats of the domain just closed.
+    :param spider_stats: the stats of the spider just closed.
     :type reason: dict

scrapy/contrib/corestats.py

@@ -11 +11 @@
 from scrapy.core import signals
 from scrapy.stats import stats
-from scrapy.stats.signals import stats_domain_opened, stats_domain_closing
+from scrapy.stats.signals import stats_spider_opened, stats_spider_closing
 from scrapy.conf import settings
 
@@ -23 +23 @@
         stats.set_value('envinfo/pid', os.getpid())
 
-        dispatcher.connect(self.stats_domain_opened, signal=stats_domain_opened)
-        dispatcher.connect(self.stats_domain_closing, signal=stats_domain_closing)
+        dispatcher.connect(self.stats_spider_opened, signal=stats_spider_opened)
+        dispatcher.connect(self.stats_spider_closing, signal=stats_spider_closing)
         dispatcher.connect(self.item_scraped, signal=signals.item_scraped)
         dispatcher.connect(self.item_passed, signal=signals.item_passed)
         dispatcher.connect(self.item_dropped, signal=signals.item_dropped)
 
-    def stats_domain_opened(self, domain):
-        stats.set_value('start_time', datetime.datetime.utcnow(), domain=domain)
-        stats.set_value('envinfo/host', stats.get_value('envinfo/host'), domain=domain)
-        stats.inc_value('domain_count/opened')
+    def stats_spider_opened(self, spider):
+        stats.set_value('start_time', datetime.datetime.utcnow(), spider=spider)
+        stats.set_value('envinfo/host', stats.get_value('envinfo/host'), spider=spider)
+        stats.inc_value('spider_count/opened')
 
-    def stats_domain_closing(self, domain, reason):
-        stats.set_value('finish_time', datetime.datetime.utcnow(), domain=domain)
-        stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, domain=domain)
-        stats.inc_value('domain_count/%s' % reason, domain=domain)
+    def stats_spider_closing(self, spider, reason):
+        stats.set_value('finish_time', datetime.datetime.utcnow(), spider=spider)
+        stats.set_value('finish_status', 'OK' if reason == 'finished' else reason, spider=spider)
+        stats.inc_value('spider_count/%s' % reason, spider=spider)
 
     def item_scraped(self, item, spider):
-        stats.inc_value('item_scraped_count', domain=spider.domain_name)
+        stats.inc_value('item_scraped_count', spider=spider)
         stats.inc_value('item_scraped_count')
 
     def item_passed(self, item, spider):
-        stats.inc_value('item_passed_count', domain=spider.domain_name)
+        stats.inc_value('item_passed_count', spider=spider)
         stats.inc_value('item_passed_count')
 
     def item_dropped(self, item, spider, exception):
         reason = exception.__class__.__name__
-        stats.inc_value('item_dropped_count', domain=spider.domain_name)
-        stats.inc_value('item_dropped_reasons_count/%s' % reason, domain=spider.domain_name)
+        stats.inc_value('item_dropped_count', spider=spider)
+        stats.inc_value('item_dropped_reasons_count/%s' % reason, spider=spider)
         stats.inc_value('item_dropped_count')

scrapy/contrib/downloadermiddleware/stats.py

@@ -6 +6 @@
 
 class DownloaderStats(object):
-    """DownloaderStats store stats of all requests, responses and
-    exceptions that pass through it.
-
-    To use this middleware you must enable the DOWNLOADER_STATS setting.
-    """
 
     def __init__(self):
@@ -17 +12 @@
 
     def process_request(self, request, spider):
-        domain = spider.domain_name
         stats.inc_value('downloader/request_count')
-        stats.inc_value('downloader/request_count', domain=domain)
-        stats.inc_value('downloader/request_method_count/%s' % request.method, domain=domain)
+        stats.inc_value('downloader/request_count', spider=spider)
+        stats.inc_value('downloader/request_method_count/%s' % request.method, spider=spider)
         reqlen = len(request_httprepr(request))
-        stats.inc_value('downloader/request_bytes', reqlen, domain=domain)
+        stats.inc_value('downloader/request_bytes', reqlen, spider=spider)
         stats.inc_value('downloader/request_bytes', reqlen)
 
     def process_response(self, request, response, spider):
-        domain = spider.domain_name
         stats.inc_value('downloader/response_count')
-        stats.inc_value('downloader/response_count', domain=domain)
-        stats.inc_value('downloader/response_status_count/%s' % response.status, domain=domain)
+        stats.inc_value('downloader/response_count', spider=spider)
+        stats.inc_value('downloader/response_status_count/%s' % response.status, spider=spider)
         reslen = len(response_httprepr(response))
-        stats.inc_value('downloader/response_bytes', reslen, domain=domain)
+        stats.inc_value('downloader/response_bytes', reslen, spider=spider)
         stats.inc_value('downloader/response_bytes', reslen)
         return response
@@ -38 +31 @@
         ex_class = "%s.%s" % (exception.__class__.__module__, exception.__class__.__name__)
         stats.inc_value('downloader/exception_count')
-        stats.inc_value('downloader/exception_count', domain=spider.domain_name)
-        stats.inc_value('downloader/exception_type_count/%s' % ex_class, domain=spider.domain_name)
+        stats.inc_value('downloader/exception_count', spider=spider)
+        stats.inc_value('downloader/exception_type_count/%s' % ex_class, spider=spider)

scrapy/contrib/itemsampler.py

@@ -54 +54 @@
 
     def process_item(self, spider, item):
-        sampled = stats.get_value("items_sampled", 0, domain=spider.domain_name)
+        sampled = stats.get_value("items_sampled", 0, spider=spider)
         if sampled < items_per_spider:
             self.items[item.guid] = item
             sampled += 1
-            stats.set_value("items_sampled", sampled, domain=spider.domain_name)
+            stats.set_value("items_sampled", sampled, spider=spider)
             log.msg("Sampled %s" % item, spider=spider, level=log.INFO)
             if close_spider and sampled == items_per_spider:
@@ -72 +72 @@
 
     def spider_closed(self, spider, reason):
-        if reason == 'finished' and not stats.get_value("items_sampled", domain=spider.domain_name):
+        if reason == 'finished' and not stats.get_value("items_sampled", spider=spider):
             self.empty_domains.add(spider.domain_name)
         self.spiders_count += 1
@@ -88 +88 @@
 
     def process_spider_input(self, response, spider):
-        if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider:
+        if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
             return []
         elif max_response_size and max_response_size > len(response_httprepr(response)):  
@@ -101 +101 @@
                 items.append(r)
 
-        if stats.get_value("items_sampled", domain=spider.domain_name) >= items_per_spider:
+        if stats.get_value("items_sampled", spider=spider) >= items_per_spider:
             return []
         else:

scrapy/contrib/pipeline/images.py

@@ -220 +220 @@
                 (status, request, referer)
         log.msg(msg, level=log.DEBUG, spider=info.spider)
-        self.inc_stats(info.spider.domain_name, status)
+        self.inc_stats(info.spider, status)
 
         try:
@@ -259 +259 @@
             log.msg('Image (uptodate): Downloaded %s from <%s> referred in <%s>' % \
                     (self.MEDIA_NAME, request.url, referer), level=log.DEBUG, spider=info.spider)
-            self.inc_stats(info.spider.domain_name, 'uptodate')
+            self.inc_stats(info.spider, 'uptodate')
 
             checksum = result.get('checksum', None)
@@ -296 +296 @@
             yield thumb_key, thumb_image, thumb_buf
 
-    def inc_stats(self, domain, status):
-        stats.inc_value('image_count', domain=domain)
-        stats.inc_value('image_status_count/%s' % status, domain=domain)
+    def inc_stats(self, spider, status):
+        stats.inc_value('image_count', spider=spider)
+        stats.inc_value('image_status_count/%s' % status, spider=spider)
 
     def convert_image(self, image, size=None):

scrapy/contrib/spidermiddleware/depth.py

@@ -19 +19 @@
 
     def process_spider_output(self, response, result, spider):
-        domain = spider.domain_name
         def _filter(request):
             if isinstance(request, Request):
@@ -29 +28 @@
                     return False
                 elif self.stats:
-                    stats.inc_value('request_depth_count/%s' % depth, domain=domain)
-                    if depth > stats.get_value('request_depth_max', 0, domain=domain):
-                        stats.set_value('request_depth_max', depth, domain=domain)
+                    stats.inc_value('request_depth_count/%s' % depth, spider=spider)
+                    if depth > stats.get_value('request_depth_max', 0, spider=spider):
+                        stats.set_value('request_depth_max', depth, spider=spider)
             return True
 
@@ -37 +36 @@
         if self.stats and 'depth' not in response.request.meta: 
             response.request.meta['depth'] = 0
-            stats.inc_value('request_depth_count/0', domain=domain)
+            stats.inc_value('request_depth_count/0', spider=spider)
 
         return (r for r in result or () if _filter(r))

scrapy/contrib/statsmailer.py

@@ -1 +1 @@
 """
-StatsMailer extension sends an email when a domain finishes scraping.
+StatsMailer extension sends an email when a spider finishes scraping.
 
 Use STATSMAILER_RCPTS setting to enable and give the recipient mail address
@@ -18 +18 @@
         if not self.recipients:
             raise NotConfigured
-        dispatcher.connect(self.stats_domain_closed, signal=signals.stats_domain_closed)
+        dispatcher.connect(self.stats_spider_closed, signal=signals.stats_spider_closed)
         
-    def stats_domain_closed(self, domain, domain_stats):
+    def stats_spider_closed(self, spider, spider_stats):
         mail = MailSender()
         body = "Global stats\n\n"
         body += "\n".join("%-50s : %s" % i for i in stats.get_stats().items())
-        body += "\n\n%s stats\n\n" % domain
-        body += "\n".join("%-50s : %s" % i for i in domain_stats.items())
-        mail.send(self.recipients, "Scrapy stats for: %s" % domain, body)
+        body += "\n\n%s stats\n\n" % spider.domain_name
+        body += "\n".join("%-50s : %s" % i for i in spider_stats.items())
+        mail.send(self.recipients, "Scrapy stats for: %s" % spider.domain_name, body)

scrapy/contrib/webconsole/stats.py

@@ -23 +23 @@
         s += "<h3>Global stats</h3>\n"
         s += stats_html_table(stats.get_stats())
-        for domain in stats.list_domains():
-            s += "<h3>%s</h3>\n" % domain
-            s += stats_html_table(stats.get_stats(domain))
+        for spider, spider_stats in stats.iter_spider_stats():
+            s += "<h3>%s</h3>\n" % spider.domain_name
+            s += stats_html_table(spider_stats)
         s += "</body>\n"
         s += "</html>\n"

scrapy/contrib_exp/spiderprofiler.py

@@ -46 +46 @@
             mafter = self._memusage()
             ct = time() - tbefore
-            domain = spider.domain_name
-            tcc = stats.get_value('profiling/total_callback_time', 0, domain=domain)
-            sct = stats.get_value('profiling/slowest_callback_time', 0, domain=domain)
-            stats.set_value('profiling/total_callback_time' % spider.domain_name, \
-                tcc+ct, domain=domain)
+            tcc = stats.get_value('profiling/total_callback_time', 0, spider=spider)
+            sct = stats.get_value('profiling/slowest_callback_time', 0, spider=spider)
+            stats.set_value('profiling/total_callback_time', tcc+ct, spider=spider)
             if ct > sct:
-                stats.set_value('profiling/slowest_callback_time', ct, domain=domain)
+                stats.set_value('profiling/slowest_callback_time', ct, spider=spider)
                 stats.set_value('profiling/slowest_callback_name', function.__name__, \
-                    domain=domain)
+                    spider=spider)
                 stats.set_value('profiling/slowest_callback_url', args[0].url, \
-                    domain=domain)
+                    spider=spider)
             if self._memusage:
                 stats.inc_value('profiling/total_mem_allocated_in_callbacks', \
-                    count=mafter-mbefore, domain=domain)
+                    count=mafter-mbefore, spider=spider)
             return r
         return new_callback

scrapy/core/engine.py

@@ -245 +245 @@
         self.downloader.open_spider(spider)
         self.scraper.open_spider(spider)
-        stats.open_domain(spider.domain_name)
+        stats.open_spider(spider)
 
         send_catch_log(signals.spider_opened, sender=self.__class__, spider=spider)
@@ -306 +306 @@
         send_catch_log(signal=signals.spider_closed, sender=self.__class__, \
             spider=spider, reason=reason)
-        stats.close_domain(spider.domain_name, reason=reason)
+        stats.close_spider(spider, reason=reason)
         dfd = defer.maybeDeferred(spiders.close_spider, spider)
         dfd.addBoth(log.msg, "Spider closed (%s)" % reason, spider=spider)

scrapy/core/scraper.py

@@ -88 +88 @@
         site = self.sites[spider]
         dfd = site.add_response_request(response, request)
-        # FIXME: this can't be called here because the stats domain may be
+        # FIXME: this can't be called here because the stats spider may be
         # already closed
         #stats.max_value('scraper/max_active_size', site.active_size, \
-        #    domain=spider.domain_name)
+        #    spider=spider)
         def finish_scraping(_):
             site.finish_response(response)
@@ -98 +98 @@
         dfd.addBoth(finish_scraping)
         dfd.addErrback(log.err, 'Scraper bug processing %s' % request, \
-            domain=spider.domain_name)
+            spider=spider)
         self._scrape_next(spider, site)
         return dfd
@@ -139 +139 @@
         log.msg(msg, log.ERROR, spider=spider)
         stats.inc_value("spider_exceptions/%s" % _failure.value.__class__.__name__, \
-            domain=spider.domain_name)
+            spider=spider)
 
     def handle_spider_output(self, result, request, response, spider):
@@ -153 +153 @@
         """
         # TODO: keep closing state internally instead of checking engine
-        domain = spider.domain_name
         if spider in self.engine.closing:
             return
@@ -166 +165 @@
                 item=output, spider=spider, response=response)
             self.sites[spider].itemproc_size += 1
-            # FIXME: this can't be called here because the stats domain may be
+            # FIXME: this can't be called here because the stats spider may be
             # already closed
             #stats.max_value('scraper/max_itemproc_size', \
-            #        self.sites[domain].itemproc_size, domain=domain)
+            #        self.sites[spider].itemproc_size, spider=spider)
             dfd = self.itemproc.process_item(output, spider)
             dfd.addBoth(self._itemproc_finished, output, spider)
@@ -196 +195 @@
         """ItemProcessor finished for the given ``item`` and returned ``output``
         """
-        domain = spider.domain_name
         self.sites[spider].itemproc_size -= 1
         if isinstance(output, Failure):

scrapy/stats/collector/__init__.py

@@ -6 +6 @@
 from scrapy.xlib.pydispatch import dispatcher
 
-from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \
-    stats_domain_closed
+from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
+    stats_spider_closed
 from scrapy.utils.signal import send_catch_log
 from scrapy.core import signals
@@ -20 +20 @@
         dispatcher.connect(self._engine_stopped, signal=signals.engine_stopped)
 
-    def get_value(self, key, default=None, domain=None):
-        return self._stats[domain].get(key, default)
+    def get_value(self, key, default=None, spider=None):
+        return self._stats[spider].get(key, default)
 
-    def get_stats(self, domain=None):
-        return self._stats[domain]
+    def get_stats(self, spider=None):
+        return self._stats[spider]
 
-    def set_value(self, key, value, domain=None):
-        self._stats[domain][key] = value
+    def set_value(self, key, value, spider=None):
+        self._stats[spider][key] = value
 
-    def set_stats(self, stats, domain=None):
-        self._stats[domain] = stats
+    def set_stats(self, stats, spider=None):
+        self._stats[spider] = stats
 
-    def inc_value(self, key, count=1, start=0, domain=None):
-        d = self._stats[domain]
+    def inc_value(self, key, count=1, start=0, spider=None):
+        d = self._stats[spider]
         d[key] = d.setdefault(key, start) + count
 
-    def max_value(self, key, value, domain=None):
-        d = self._stats[domain]
+    def max_value(self, key, value, spider=None):
+        d = self._stats[spider]
         d[key] = max(d.setdefault(key, value), value)
 
-    def min_value(self, key, value, domain=None):
-        d = self._stats[domain]
+    def min_value(self, key, value, spider=None):
+        d = self._stats[spider]
         d[key] = min(d.setdefault(key, value), value)
 
-    def clear_stats(self, domain=None):
-        self._stats[domain].clear()
+    def clear_stats(self, spider=None):
+        self._stats[spider].clear()
 
-    def list_domains(self):
-        return [d for d in self._stats.keys() if d is not None]
+    def iter_spider_stats(self):
+        return [x for x in self._stats.iteritems() if x[0]]
 
-    def open_domain(self, domain):
-        self._stats[domain] = {}
-        send_catch_log(stats_domain_opened, domain=domain)
+    def open_spider(self, spider):
+        self._stats[spider] = {}
+        send_catch_log(stats_spider_opened, spider=spider)
 
-    def close_domain(self, domain, reason):
-        send_catch_log(stats_domain_closing, domain=domain, reason=reason)
-        stats = self._stats.pop(domain)
-        send_catch_log(stats_domain_closed, domain=domain, reason=reason, \
-            domain_stats=stats)
+    def close_spider(self, spider, reason):
+        send_catch_log(stats_spider_closing, spider=spider, reason=reason)
+        stats = self._stats.pop(spider)
+        send_catch_log(stats_spider_closed, spider=spider, reason=reason, \
+            spider_stats=stats)
         if self._dump:
-            log.msg("Dumping domain stats:\n" + pprint.pformat(stats), \
-                domain=domain)
-        self._persist_stats(stats, domain)
+            log.msg("Dumping spider stats:\n" + pprint.pformat(stats), \
+                spider=spider)
+        self._persist_stats(stats, spider)
 
     def _engine_stopped(self):
@@ -68 +68 @@
         if self._dump:
             log.msg("Dumping global stats:\n" + pprint.pformat(stats))
-        self._persist_stats(stats, domain=None)
+        self._persist_stats(stats, spider=None)
 
-    def _persist_stats(self, stats, domain=None):
+    def _persist_stats(self, stats, spider=None):
         pass
 
@@ -78 +78 @@
         super(MemoryStatsCollector, self).__init__()
         self.domain_stats = {}
-        
-    def _persist_stats(self, stats, domain=None):
-        self.domain_stats[domain] = stats
+
+    def _persist_stats(self, stats, spider=None):
+        if spider is not None:
+            self.domain_stats[spider.domain_name] = stats
 
 
 class DummyStatsCollector(StatsCollector):
 
-    def get_value(self, key, default=None, domain=None):
+    def get_value(self, key, default=None, spider=None):
         return default
 
-    def set_value(self, key, value, domain=None):
+    def set_value(self, key, value, spider=None):
         pass
 
-    def set_stats(self, stats, domain=None):
+    def set_stats(self, stats, spider=None):
         pass
 
-    def inc_value(self, key, count=1, start=0, domain=None):
+    def inc_value(self, key, count=1, start=0, spider=None):
         pass
 
-    def max_value(self, key, value, domain=None):
+    def max_value(self, key, value, spider=None):
         pass
 
-    def min_value(self, key, value, domain=None):
+    def min_value(self, key, value, spider=None):
         pass
 

scrapy/stats/collector/mysql.py

@@ -17 +17 @@
         self._mysql_conn = mysql_connect(mysqluri, use_unicode=False) if mysqluri else None
         
-    def _persist_stats(self, stats, domain=None):
-        if domain is None: # only store domain-specific stats
+    def _persist_stats(self, stats, spider=None):
+        if spider is None: # only store spider-specific stats
             return
         if self._mysql_conn is None:
@@ -28 +28 @@
         c = self._mysql_conn.cursor()
         c.execute("INSERT INTO %s (domain,stored,data) VALUES (%%s,%%s,%%s)" % table, \
-            (domain, stored, datas))
+            (spider.domain_name, stored, datas))
         self._mysql_conn.commit()

scrapy/stats/collector/simpledb.py

@@ -23 +23 @@
         connect_sdb().create_domain(self._sdbdomain)
 
-    def _persist_stats(self, stats, domain=None):
-        if domain is None: # only store domain-specific stats
+    def _persist_stats(self, stats, spider=None):
+        if spider is None: # only store spider-specific stats
             return
         if not self._sdbdomain:
             return
         if self._async:
-            dfd = threads.deferToThread(self._persist_to_sdb, domain, stats.copy())
+            dfd = threads.deferToThread(self._persist_to_sdb, spider, stats.copy())
             dfd.addErrback(log.err, 'Error uploading stats to SimpleDB', \
-                domain=domain)
+                spider=spider)
         else:
-            self._persist_to_sdb(domain, stats)
+            self._persist_to_sdb(spider, stats)
 
-    def _persist_to_sdb(self, domain, stats):
-        ts = self._get_timestamp(domain).isoformat()
-        sdb_item_id = "%s_%s" % (domain, ts)
+    def _persist_to_sdb(self, spider, stats):
+        ts = self._get_timestamp(spider).isoformat()
+        sdb_item_id = "%s_%s" % (spider.domain_name, ts)
         sdb_item = dict((k, self._to_sdb_value(v, k)) for k, v in stats.iteritems())
-        sdb_item['domain'] = domain
+        sdb_item['domain'] = spider.domain_name
         sdb_item['timestamp'] = self._to_sdb_value(ts)
         connect_sdb().put_attributes(self._sdbdomain, sdb_item_id, sdb_item)
 
-    def _get_timestamp(self, domain):
+    def _get_timestamp(self, spider):
         return datetime.utcnow()
 

scrapy/stats/signals.py

@@ -1 @@
-stats_domain_opened = object()
-stats_domain_closing = object()
-stats_domain_closed = object()
+stats_spider_opened = object()
+stats_spider_closing = object()
+stats_spider_closed = object()

scrapy/tests/test_downloadermiddleware_stats.py

@@ -1 +1 @@
 from unittest import TestCase
 
-from scrapy.conf import settings
 from scrapy.contrib.downloadermiddleware.stats import DownloaderStats
 from scrapy.http import Request, Response
@@ -11 +10 @@
 
     def setUp(self):
-        self.spider = BaseSpider()
-        self.spider.domain_name = 'scrapytest.org'
+        self.spider = BaseSpider('scrapytest.org')
         self.mw = DownloaderStats()
 
-        stats.open_domain(self.spider.domain_name)
+        stats.open_spider(self.spider)
 
         self.req = Request('scrapytest.org')
@@ -23 +21 @@
         self.mw.process_request(self.req, self.spider)
         self.assertEqual(stats.get_value('downloader/request_count', \
-            domain=self.spider.domain_name), 1)
+            spider=self.spider), 1)
         
     def test_process_response(self):
         self.mw.process_response(self.req, self.res, self.spider)
         self.assertEqual(stats.get_value('downloader/response_count', \
-            domain=self.spider.domain_name), 1)
+            spider=self.spider), 1)
 
     def test_process_exception(self):
         self.mw.process_exception(self.req, Exception(), self.spider)
         self.assertEqual(stats.get_value('downloader/exception_count', \
-            domain=self.spider.domain_name), 1)
+            spider=self.spider), 1)
 
     def tearDown(self):
-        stats.close_domain(self.spider.domain_name, '')
+        stats.close_spider(self.spider, '')
 

scrapy/tests/test_spidermiddleware_depth.py

@@ -15 +15 @@
         settings.overrides['DEPTH_STATS'] = True
 
-        self.spider = BaseSpider()
-        self.spider.domain_name = 'scrapytest.org'
+        self.spider = BaseSpider('scrapytest.org')
 
-        stats.open_domain(self.spider.domain_name)
+        stats.open_spider(self.spider)
 
         self.mw = DepthMiddleware()
@@ -32 +31 @@
         self.assertEquals(out, result)
 
-        rdc = stats.get_value('request_depth_count/1',
-                              domain=self.spider.domain_name)
+        rdc = stats.get_value('request_depth_count/1', spider=self.spider)
         self.assertEquals(rdc, 1)
 
@@ -41 +39 @@
         self.assertEquals(out2, [])
 
-        rdm = stats.get_value('request_depth_max',
-                              domain=self.spider.domain_name)
+        rdm = stats.get_value('request_depth_max', spider=self.spider)
         self.assertEquals(rdm, 1)
  
@@ -50 +47 @@
         settings.disabled = True
 
-        stats.close_domain(self.spider.domain_name, '')
+        stats.close_spider(self.spider, '')
 

scrapy/tests/test_stats.py

@@ -1 +1 @@
 import unittest 
 
+from scrapy.spider import BaseSpider
 from scrapy.xlib.pydispatch import dispatcher
 from scrapy.stats.collector import StatsCollector, DummyStatsCollector
-from scrapy.stats.signals import stats_domain_opened, stats_domain_closing, \
-    stats_domain_closed
+from scrapy.stats.signals import stats_spider_opened, stats_spider_closing, \
+    stats_spider_closed
 
 class StatsCollectorTest(unittest.TestCase):
+
+    def setUp(self):
+        self.spider = BaseSpider()
 
     def test_collector(self):
@@ -44 +48 @@
         stats.max_value('v2', 100)
         stats.min_value('v3', 100)
-        stats.open_domain('a')
-        stats.set_value('test', 'value', domain='a')
+        stats.open_spider('a')
+        stats.set_value('test', 'value', spider=self.spider)
         self.assertEqual(stats.get_stats(), {})
         self.assertEqual(stats.get_stats('a'), {})
@@ -52 +56 @@
         signals_catched = set()
 
-        def domain_opened(domain):
-            assert domain == 'example.com'
-            signals_catched.add(stats_domain_opened)
+        def spider_opened(spider):
+            assert spider is self.spider
+            signals_catched.add(stats_spider_opened)
 
-        def domain_closing(domain, reason):
-            assert domain == 'example.com'
+        def spider_closing(spider, reason):
+            assert spider is self.spider
             assert reason == 'testing'
-            signals_catched.add(stats_domain_closing)
+            signals_catched.add(stats_spider_closing)
 
-        def domain_closed(domain, reason, domain_stats):
-            assert domain == 'example.com'
+        def spider_closed(spider, reason, spider_stats):
+            assert spider is self.spider
             assert reason == 'testing'
-            assert domain_stats == {'test': 1}
-            signals_catched.add(stats_domain_closed)
+            assert spider_stats == {'test': 1}
+            signals_catched.add(stats_spider_closed)
 
-        dispatcher.connect(domain_opened, signal=stats_domain_opened)
-        dispatcher.connect(domain_closing, signal=stats_domain_closing)
-        dispatcher.connect(domain_closed, signal=stats_domain_closed)
+        dispatcher.connect(spider_opened, signal=stats_spider_opened)
+        dispatcher.connect(spider_closing, signal=stats_spider_closing)
+        dispatcher.connect(spider_closed, signal=stats_spider_closed)
 
         stats = StatsCollector()
-        stats.open_domain('example.com')
-        stats.set_value('test', 1, domain='example.com')
-        stats.close_domain('example.com', 'testing')
-        assert stats_domain_opened in signals_catched
-        assert stats_domain_closing in signals_catched
-        assert stats_domain_closed in signals_catched
+        stats.open_spider(self.spider)
+        stats.set_value('test', 1, spider=self.spider)
+        self.assertEqual([(self.spider, {'test': 1})], list(stats.iter_spider_stats()))
+        stats.close_spider(self.spider, 'testing')
+        assert stats_spider_opened in signals_catched
+        assert stats_spider_closing in signals_catched
+        assert stats_spider_closed in signals_catched
 
-        dispatcher.disconnect(domain_opened, signal=stats_domain_opened)
-        dispatcher.disconnect(domain_closing, signal=stats_domain_closing)
-        dispatcher.disconnect(domain_closed, signal=stats_domain_closed)
+        dispatcher.disconnect(spider_opened, signal=stats_spider_opened)
+        dispatcher.disconnect(spider_closing, signal=stats_spider_closing)
+        dispatcher.disconnect(spider_closed, signal=stats_spider_closed)
 
 if __name__ == "__main__":