Merge parse and crawl commands

[pablo]

There is a somewhat hacky parse command to fetch specific pages, parse them and show the results.

The problem with this parse command is that it runs outside the crawl loop, and thus it could return different results. For example, if a spider returns a particular requests, it won't follow then (when using the parse command).

We need to "integrate" better the parse command inside the engine crawling loop to provide results more similar to those that would be obtained by crawling with the crawl command.

[pablo]

This will be probably done with the refactoring of the spider middleware that is planned for 0.10.

[anibal]

see also #173

[pablo]

We won't be merging crawl and parse for now. After the parse command refactoring introduced in r2196, it should be easier to maintain it.