Timeline
09/02/10: Yesterday
- 21:34 Ticket #220 (Spider Queue (aka. Execution Queue refactoring)) created by
- Currently, implementing execution queue backends is awkward because it …
- 19:39 Ticket #219 (S3 download handler) created by
- We should add a download handler for Amazon S3. It should handle …
09/01/10:
- 16:59 Ticket #218 (New Scrapy service with egg upload support) created by
- Scrapy 0.10 will include a service runner with support for: * multiple …
- 14:53 Ticket #132 (Images Pipeline's function item_completed has some bug) closed by
- worksforme: Example spider code doesn't extract urls anymore, I tried adapting it to …
08/31/10:
- 21:40 Changeset [2242:e01b7b93b60c] by
-
Removed unused imports
- 16:05 Ticket #217 (Simplify Images pipeline) closed by
- fixed: Done in r2241
- 16:03 Changeset [2241:8d96749025db] by
-
Simplified images pipeline by allowing it to be used without having to ...
- 16:01 Ticket #217 (Simplify Images pipeline) created by
- Images pipeline should work out of the box without the user having to …
08/28/10:
- 18:06 Changeset [2240:be275c298235] by
-
Yet another scrapy.cmdline code refactoring by removing --settings and ...
- 14:47 Changeset [2239:681f56cf4aa3] by
-
Fixed typo
- 14:46 Ticket #216 (Support passing spider arguments in crawl command) closed by
- fixed: Done in r2238
- 14:44 Ticket #216 (Support passing spider arguments in crawl command) created by
- scrapy crawl command should support passing spider arguments with a …
- 14:43 Changeset [2238:ed1404c3b025] by
-
Support passing spider arguments in crawl command with -a option. Closes ...
- 14:07 Changeset [2237:439f497c4164] by
-
Minor change to --pidfile argument
08/27/10:
- 18:36 Changeset [2236:7d07adfd0238] by
-
call Spider.closed() method (if it exists) on SpiderManager?.close_spider()
- 17:21 Changeset [2235:3e86c540d53d] by
-
Fixed typo
- 16:19 Changeset [2234:61dae73012ef] by
-
Restored SpiderManager?.close_spider() method but using signals instead of ...
- 15:50 Changeset [2233:6423cf994096] by
-
Moved tests to reflect new module location
- 13:45 Changeset [2232:bb992796df0d] by
-
Check that arguments and keyword arguments are not passed simultaneously ...
- 13:45 Changeset [2231:6b49bcfb718b] by
-
Support passing all keyword arguments to ExecutionQueue? append_spider_name ...
- 11:15 Changeset [2230:98d9d21282f3] by
-
Added Scrapy to scrapy --version
- 01:05 Changeset [2229:5e91270f6d53] by
-
Updated some missing references to scrapy-ws script
- 00:53 Changeset [2228:a9cf219219a5] by
-
Print Scrapy on first log line
- 00:33 Changeset [2227:c1b3c0a5d616] by
-
Moved scrapy-ws script to extras/ and fixed broken methods due to changes ...
08/26/10:
- 23:23 Changeset [2226:4b1245c0c37a] by
-
Fixed log formatter tests
- 23:19 Changeset [2225:3210b5319ec4] by
-
Added pluggable log formatter
- 22:20 Changeset [2224:44b60238bc2d] by
-
Simplified engine by removing the configure() and kill() methods. Also ...
- 21:15 Changeset [2223:5af273545eb4] by
-
Moved module: scrapy.core.queue to scrapy.queue
- 20:32 Changeset [2222:8063c344abb3] by
-
Improved Twisted version detection (wasn't working for Twisted 10.0.0)
08/25/10:
- 21:23 Scrapy010Changes edited by
- (diff)
- 21:06 Changeset [2221:85012fc54f57] by
-
Fixed typo
- 21:04 Changeset [2220:7b7c86cd53c2] by
-
Added docstring to test_engine.py
- 20:41 Ticket #77 (Library API) closed by
- fixed: This ticket can be considered resolved now that Scrapy can be embedded …
- 20:37 Ticket #215 (Refactor engine tests) closed by
- fixed: Done in r2216.
- 20:36 Ticket #214 (Support embedding Scrapy crawler in Twisted applications) closed by
- fixed: Done in r2213.
- 19:59 Changeset [2219:5ba3f9f51791] by
-
Fixed crawler reference
- 19:59 Changeset [2218:8abd27ce1792] by
-
Moved scrapy.cfg auto-discovery to scrapy.conf.EnvironmentSettings? class
- 19:31 Changeset [2217:a70265ea52cd] by
-
Replaced old manager references with crawler
- 19:24 Changeset [2216:2d340ec4e027] by
-
Refactoring of Crawler, Commands, Execution Queue and Spider ...
- 16:58 Ticket #215 (Refactor engine tests) created by
- The current Scrapy engine tests (@scrapy.tests.test_engine@) are very bad. …
- 06:41 Changeset [2215:ab23fb44cfae] by
-
Removed obsolete setting
- 05:33 Changeset [2214:faef562f0fc9] by
-
Instantiate SpiderManager? in Crawler constructor
- 05:24 Changeset [2213:ff1db5cd50a8] by
-
Added CrawlerProcess? class, isolating all (Twisted) reactor-controlling ...
- 05:20 Ticket #214 (Support embedding Scrapy crawler in Twisted applications) created by
- Scrapy should support embedding its crawler from other Twisted …
- 05:06 Changeset [2212:613600e61a0d] by
-
Removed unneeded line
08/23/10:
- 22:04 Changeset [2211:477cc844b9fa] by
-
Removed scrapy-sqs script, as it has been superseded by the new scrapy ...
- 21:40 AptRepos edited by
- (diff)
- 21:31 Ticket #211 (Setup Ubuntu repos for Scrapy 0.10) closed by
- fixed: Finished setting up Buildbot for Scrapy 0.10 and added documentation about …
- 21:28 Changeset [2210:e59cd31893ab] by
-
Added documentation for Ubuntu packages. Refs #211
- 13:35 Ticket #213 (Fix Github mirror) closed by
- fixed: Fixed using Ping Yin suggestion from …
- 12:39 Ticket #213 (Fix Github mirror) created by
- Github mirror is not working. See …
- 10:26 AptRepos edited by
- (diff)
- 00:25 Changeset [2209:b2df01cebbf0] by
-
Moved spidermanager tests module according to policies
- 00:23 Changeset [2208:abf2bb166538] by
-
Some minor fixes to contribution Contributing documentation
08/22/10:
- 22:50 Ticket #212 (Deployment documentation) created by
- We need to add a Deployment section to the documentation. It …
- 22:42 Changeset [2207:e49bef0037c2] by
-
removed (somewhat hacky) MAIL_DEBUG setting
- 22:20 Scrapy010Changes edited by
- (diff)
- 22:20 Scrapy010Changes edited by
- (diff)
- 22:11 Ticket #174 (Values that evaluate to False are not loaded in ItemLoader) closed by
- fixed: Patch applied in r2206, thanks Anibal.
- 22:07 Changeset [2206:5c0c5b999868] by
-
Fixed Item Loader bug that was preventing values that evaluate to False ...
- 22:02 Ticket #109 (setup.py bdist_wininst bug packaging data files when run from Linux) closed by
- invalid: I'm closing this ticket as invalid, as this is something that needs to be …
- 21:57 Scrapy010Changes edited by
- (diff)
- 21:53 Changeset [2205:e0fbf63245ba] by
-
Moved module: scrapy.contrib.spidermanager to scrapy.spidermanager
- 20:10 Changeset [2204:57b0ce0612ba] by
-
Minor improvement to bash autocompletion
- 19:37 Changeset [2203:71a334954b3f] by
-
Fixed tests on Windows
- 19:08 Changeset [2202:24a5686c08db] by
-
better skip test message
- 19:08 Changeset [2201:de5fc1ccc9ee] by
-
minor fixes to FAQ
- 06:16 Ticket #211 (Setup Ubuntu repos for Scrapy 0.10) created by
- We need to setup Ubuntu repos for Scrapy 0.10 and: * add a …
- 05:59 Changeset [2200:e152ccf30212] by
-
Added FAQ entry about feed exports
- 05:48 Changeset [2199:6aea46de2dec] by
-
Renamed webservice ManagerResource? to CrawlerResource?
- 05:38 Changeset [2198:3ac71e5c1cf8] by
-
Removed webservice Spiders and Extensions resources since they can now be ...
- 05:33 Changeset [2197:4034b3452fdb] by
-
Made ExtensionManager? a subclass of MiddlewareManager?
- 05:10 Ticket #106 (Merge parse and crawl commands) closed by
- wontfix: We won't be merging crawl and parse for now. After the parse command …
- 05:08 Ticket #173 (Parse command issues (yield vs. return, _values key in items)) closed by
- fixed: The parse command refactoring introduced in r2196 fixes this issue.
- 05:04 Changeset [2196:e3d774da79fe] by
-
"parse" command refactoring. This fixes #173 and renders #106 invalid.
- 02:49 Scrapy010Changes edited by
- (diff)
- 02:38 Ticket #189 (Move spider/extension manager singletons to scrapy.project) closed by
- fixed: Done in r2193, r2194, r2195 - we went with option 1.
- 02:34 Scrapy010Changes edited by
- (diff)
- 02:30 Ticket #204 (Split stats collection facility (singleton) from stats collector classes) closed by
- fixed: Done in r2192.
- 02:15 Changeset [2195:3ab0768618fd] by
-
Moved scrapy.extension.extensions singleton to a "extensions" attribute of ...
- 02:15 Changeset [2194:8edb5ff17422] by
-
Moved scrapy.spider.spiders singleton to a "spiders" attribute of the ...
- 02:10 Changeset [2193:4d6a00c51e1b] by
-
Moved scrapymanager singleton to scrapy.project module. Refs #189
Detail ...
- 01:24 Changeset [2192:520318a18030] by
-
Splitted stats collector classes from stats collection facility (#204)
* ...
08/21/10:
- 05:10 Changeset [2191:3f0c4d104561] by
-
Added settings to Scrapy shell variables
- 05:03 Changeset [2190:a61ec3ab4c13] by
-
Improved some commands descriptions
- 04:56 Changeset [2189:d8ae6b3bbe07] by
-
example projects: added scrapy.cfg and removed scrapy-ctl.py
- 04:46 Changeset [2188:678086e7dea4] by
-
genspider command refactoring. Also updated tests and doc
- 03:37 Changeset [2187:3c8286fe018e] by
-
Made command-line too output more concise
- 03:26 Scrapy010Changes edited by
- (diff)
- 03:26 Ticket #210 (Add bash completion to Scrapy command-line tool) closed by
- fixed: Done in r2186.
- 03:23 Changeset [2186:7539ee343834] by
-
Added bash completion for the Scrapy command-line tool. Closes #210
- 03:14 Ticket #210 (Add bash completion to Scrapy command-line tool) created by
- We can start with something simple like completing commands and spider …
- 01:47 Scrapy010Changes edited by
- (diff)
- 01:46 Ticket #209 (Rename command "start" to "runserver") closed by
- fixed: Done in r2184.
- 01:44 Changeset [2185:ac37d7345391] by
-
Fixed missing reference to old 'start' command. Refs #209
- 01:42 Changeset [2184:081cc3237b59] by
-
Renamed command "start" to "runserver". Closes #209
- 01:35 Ticket #209 (Rename command "start" to "runserver") created by
- start is a bit ambiguous as there is already a startproject command …
- 01:32 Ticket #207 (Several documentation fixes) closed by
- fixed: I've applied the patch in r2183, thanks for contribution these fixes …
- 01:26 Changeset [2183:ef01112cccc9] by
-
Applied documentation patch provided by Lucian Ursu (closes #207)
- 01:24 Changeset [2182:1295fd28f02c] by
-
Removed obsolete files
08/20/10:
- 11:59 Ticket #208 (Multi-project support in command-line tool) created by
- Add support for passing the project name to the scrapy command-line …
- 11:26 Changeset [2181:43e1540044cc] by
-
Scrapy shell refactoring
- 10:29 Ticket #207 (Several documentation fixes) created by
- Hi, guys, I've finally managed to solve the problem with the line …
- 01:33 Changeset [2180:44c5df64ca4c] by
-
Scrapy shell: moved python console starting code to scrapy.utils.console ...
08/19/10:
- 21:19 Scrapy010Changes edited by
- (diff)
- 21:17 Ticket #206 (Don't hide Scrapy log on Scrapy shell) closed by
- fixed: Done in r2178.
- 21:17 Ticket #206 (Don't hide Scrapy log on Scrapy shell) created by
- Scrapy shell currently filters log messages with level under WARNING, we …
- 21:14 Changeset [2179:9a4ca005afa4] by
-
Minor change to log message
- 21:14 Ticket #205 (Scrapy shell hangs if request fails to download) closed by
- fixed: Fixed in r2178.
- 21:11 Changeset [2178:0b33436abf80] by
-
Fixed bug in Scrapy shell which hanged if requests failed to download ...
- 19:02 Ticket #205 (Scrapy shell hangs if request fails to download) created by
- If request fails to download for some reason the Scrapy shell hangs and …
- 18:09 Ticket #204 (Split stats collection facility (singleton) from stats collector classes) created by
- Right now both stats collector classes and stats collection facility …
- 17:59 Changeset [2177:1d8aa9c998dd] by
-
updated FAQ entry to recommend using higher download delays
- 17:57 Changeset [2176:85b02f8684a9] by
-
Improved support for scrapy-ctl -> scrapy migration by generating the ...
- 16:51 Changeset [2175:ce958359dba0] by
-
Added FAQ entry about response code 999
- 03:07 Scrapy010Changes edited by
- (diff)
- 03:01 Ticket #203 (Persistent spider contexts) created by
- We need a extension that allows spiders to keep a persistent context …
- 02:59 Ticket #198 (Persistent execution queue with scrapy command to control it) closed by
- fixed: Done in r2174.
- 02:55 Changeset [2174:3d0afb02075e] by
-
Added persistent execution queue (based on SQLite), and a new 'queue' ...
- 02:30 Changeset [2173:99c5c9de085c] by
-
Added tests for runspider command
- 01:58 Changeset [2172:ec83fc4fc860] by
-
Fixed bug with runspider command that appeared after the introduction of ...
- 00:30 Ticket #202 (Document Scrapy command-line tool) closed by
- fixed: Documentation improved in r2170.
- 00:30 Ticket #202 (Document Scrapy command-line tool) created by
- The Scrapy command-line tool documentation in 0.9 is very poor, we need a …
- 00:07 Changeset [2171:c8c01156edf0] by
-
removed obsolete setting
- 00:04 Changeset [2170:f619f8f5a649] by
-
Improved documentation of Scrapy command-line tool
08/18/10:
- 20:00 Scrapy010Changes edited by
- (diff)
- 19:52 Ticket #199 (Replace "scrapy-ctl.py" tool with simpler "scrapy" tool with project ...) closed by
- fixed: Completed in r2169: * scrapy-ctl is deprecated in favour of using …
- 19:48 Changeset [2169:dbfa0177be10] by
-
Deprecated scrapy-ctl.py command in favour of simpler "scrapy" command. ...
- 13:05 Changeset [2168:b6451dd3fbe9] by
-
pipeline process_item methods decorated with inlineCallbacks fails because ...
08/17/10:
- 20:32 Scrapy010Changes edited by
- (diff)
- 18:34 Scrapy010Changes edited by
- (diff)
- 18:33 Ticket #201 (Default settings per command should be specified in the command class) closed by
- fixed: Done in r2166.
- 18:31 Changeset [2167:e1d7ced651a5] by
-
removed hacky command_executed signal
- 18:30 Ticket #201 (Default settings per command should be specified in the command class) created by
- This would be simpler and more elegant than having a separate group of …
- 18:30 Changeset [2166:6173106d771d] by
-
Default per-command settings are now specified in the default_settings ...
- 14:48 Changeset [2165:c973dbbb83ad] by
-
minor setting fix
- 14:42 Scrapy010Changes edited by
- (diff)
- 14:41 Ticket #197 (Feed exporter with pluggable backends) closed by
- fixed: Feed export extension added in r2163 with some storage tests included. …
- 14:37 Changeset [2164:66dea6f26663] by
-
fixed minor formatting issue with new feed exports doc
- 14:32 Ticket #200 (Add tests for Feed export extension) created by
- Some feed storages are tested, but we need to tests the main FeedExport …
- 14:27 Changeset [2163:173bf8926ca7] by
-
Added new Feed exports extension with documentation and storage tests. ...
- 01:02 Ticket #140 (scrapy-ctl.py vs scrapy-ctl.py) closed by
- duplicate: This will be resolved by #199
- 00:59 Changeset [2162:8065ea308804] by
-
Added "scrapy" command with project settings auto-discovery. Refs #199
- 00:46 Ticket #199 (Replace "scrapy-ctl.py" tool with simpler "scrapy" tool with project ...) created by
- A common confusion among Scrapy users is whether they should use the …
08/16/10:
- 12:25 Changeset [2161:2ea67b191309] by
-
fixed bug in url_is_from_spider() when no allowed_domains class attribute ...
- 11:09 Ticket #198 (Persistent execution queue with scrapy command to control it) created by
- We need to provide a persistent execution queue that works out of the box …
- 10:10 Changeset [2160:874d7832b2ac] by
-
Simplified BaseSpider? code by removing backwards compatibility code
08/14/10:
- 21:24 Scrapy010Changes edited by
- (diff)
- 21:23 Ticket #193 (Deferred signals) closed by
- fixed: r2159 adds support for returning deferreds from handlers of these …
- 21:10 Changeset [2159:83331220addd] by
-
Added support for returning deferreds from (some) signal handlers. Closes ...
- 16:25 Ticket #197 (Feed exporter with pluggable backends) created by
- One of the most useful (and probably required) features for implementing …
08/13/10:
- 01:50 Ticket #196 (Spider errors logged as "Unhandled errors") closed by
- fixed: Fixed in r2158.
- 01:45 Changeset [2158:1b1b48e3bda2] by
-
Improve spider errors logging which were previously logged as confusing ...
- 01:20 Ticket #196 (Spider errors logged as "Unhandled errors") created by
- Most spider errors are being logged as unhandled errors because the …
08/12/10:
- 20:45 Changeset [2157:0c784ef294fe] by
-
updated old documentation references
- 20:37 Ticket #156 (Documentation patch) closed by
- invalid: Closed due to lack of feedback.
- 20:33 Ticket #112 (Unhandled error on engine.crawl()) closed by
- fixed: This was actually a harmless error, which was silenced in r2139 and then …
- 20:30 Ticket #184 (A closing quote is missing in doc file) closed by
- wontfix: Thanks for reporting, but scheduler middleware doc was removed from trunk …
- 10:57 Scrapy010Changes edited by
- (diff)
- 10:54 Ticket #195 (New item pipeline open/close_spider methods with deferred support) closed by
- fixed: Done in r2156.
- 10:48 Changeset [2156:697d3fac305b] by
-
Some improvements to Item Pipeline (closes #195):
* Made Item Pipeline ...
- 10:26 Changeset [2155:2a5bdf23f12b] by
-
added prepend_level argument to log._adapt_eventdict()
- 03:04 Changeset [2154:02b5ccacc05a] by
-
made MiddlewareManager? an abstract class, and minor change to log message
- 02:57 Ticket #195 (New item pipeline open/close_spider methods with deferred support) created by
- We need to add open_spider/close_spider methods to item pipeline and …
- 00:33 Changeset [2153:a096e41083a4] by
-
Error logging improvements
08/10/10:
- 18:27 Changeset [2152:2521587d913e] by
-
removed reference to old middleware
- 18:26 Changeset [2151:a29d1e4a591f] by
-
removed unused import
- 18:22 Changeset [2150:b6d533ab3506] by
-
removed unused import
- 18:15 Changeset [2149:e5fb67776d9b] by
-
remove old unsupported item sampler middleware
- 18:03 Changeset [2148:c6820ff4fc06] by
-
added new MiddlewareManager? class that will be used as base class for ...
- 17:49 Scrapy010Changes edited by
- (diff)
- 17:49 Scrapy010Changes edited by
- (diff)
- 17:47 Changeset [2147:aecaa77a094e] by
-
updated missing doc reference from previous commit
- 17:42 Changeset [2146:7b1b98cf9946] by
-
added more information to deprecation notice
- 17:40 Changeset [2145:54f082658424] by
-
moved scrapy.core.signals to scrapy.signals, keeping backwards ...
- 17:36 Changeset [2144:c1b116a5bf32] by
-
moved scrapy.core.exceptions to scrapy.exceptions, keeping backwards ...
- 17:23 Changeset [2143:505b3419de4b] by
-
removed old unsupported SpiderProfiler? extension
- 16:59 Changeset [2142:99ea226a2712] by
-
removed scheduler middleware doc, as scheduler middleware will be removed ...
08/09/10:
- 14:41 Changeset [2141:e253de82535d] by
-
Removed 'sender' argument when sending signals, as we're not sending it ...
- 13:32 Changeset [2140:48d879892b8b] by
-
add signal handler name when logging errors
- 13:24 Changeset [2139:b3d1bb782d49] by
-
silence irrelevant (and confusing) errors generated in tests by signals ...
- 13:22 Changeset [2138:36519525de0f] by
-
fixed utils.signal tests broken in previous commit
- 12:06 Ticket #194 (Log full traceback on signal handler errors) closed by
- fixed: Done in r2137.
- 12:05 Changeset [2137:d95386ebb137] by
-
Log full traceback of signal handler errors in send_catch_log() - closes ...
- 11:57 Ticket #194 (Log full traceback on signal handler errors) created by
- Error logging of signal handler error is pretty poot, it only prints the …
- 11:09 Changeset [2136:d0740febb213] by
-
moved scrapy log observer logic into a separate function
- 11:07 Changeset [2135:1f1a1c71d594] by
-
avoid noisy KeyError? in enqueue_scrape, when closing spiders manually
08/08/10:
- 07:30 Changeset [2134:d1a4dd03c3fb] by
-
Moved scrapy/command/init.py to scrapy/command.py
- 05:25 CompaniesUsingScrapy edited by
- remove spam (diff)
- 05:09 ScrapyRecipes edited by
- more spam (diff)
- 05:08 ScrapyRecipes edited by
- remove spam links (diff)
- 03:08 Ticket #193 (Deferred signals) created by
- We need to support returning deferreds in signal handlers. This is …
- 00:37 SEP-009 edited by
- (diff)
08/07/10:
- 16:05 Scrapy010Changes edited by
- (diff)
- 15:54 Ticket #192 (JSON item exporter) closed by
- fixed: Done in r2133.
- 15:52 Changeset [2133:d5285c2b5204] by
-
Added JSON item exporter with doc and unittests (closes #192), and ...
- 14:14 Ticket #192 (JSON item exporter) created by
- So for we have avoided adding a true JSON item exporter because it doesn't …
08/06/10:
- 15:06 Scrapy010Changes edited by
- (diff)
- 15:05 Changeset [2132:b62ca38e5cd9] by
-
changed variable names for clarity
- 15:04 Ticket #191 (Spiders should be able to tell which requests they can handle) closed by
- fixed: Done in r191.
- 14:59 Changeset [2131:5b57b280167b] by
-
Added handles_request() class method to BaseSpider? - closes #191
- 14:57 Ticket #191 (Spiders should be able to tell which requests they can handle) created by
- Right now, the default spider manager looks in the name and …
08/05/10:
- 20:46 Changeset [2130:b287a823b964] by
-
Added support for logging twisted errors generated outside of Scrapy - ...
- 13:31 Changeset [2129:5984a19ccd64] by
-
make runtests.sh more virtualenv-friendly
08/04/10:
- 14:20 Changeset [2128:660d832637b9] by
-
Fixed bug with non-keepalive execution queues (closes #190)
- 14:20 Ticket #190 (Problem with non-keepalive deferred execution queues) closed by
- fixed: Fixed in r2128.
- 14:18 Ticket #190 (Problem with non-keepalive deferred execution queues) created by
- Non-keepalive execution queues don't work properly now. Instead of …
