Ticket #157 (closed defect: fixed)

Opened 4 months ago

Last modified 4 months ago

HttpErrorMiddleware: Notify of dropped responses calling request errback

Reported by: daniel Owned by: daniel
Priority: major Milestone: 0.9
Component: code Version: 0.9
Keywords: Cc: daniel pablo

Description

When a requests is ignored by HttpErrorMiddleware? because response is non-200, spider hasn't a chance to be notified to react before discarting the request.

This is a problem with spiders that returns batches of requests, keeping a track of the pending count of requests to be processed until the next batch is returned. If some of the requests are ignored and spider isn't notified then spider lost the count, and it never schedule the next batch again.

Change History

Changed 4 months ago by daniel

  • status changed from new to closed
  • version changed from 0.8 to 0.9
  • resolution set to fixed
  • milestone set to 0.9

fixed by [29fb46c3ae92]

Changed 4 months ago by daniel

this bugfix changes API for process_spider_input, now it is not possible to return an iterable from process_spider_input.

It only accepts returning None or raising an exception.

Note: See TracTickets for help on using tickets.