Patch for def canonicalize_url

[sdogi]

Current behavior of Scrapy when finding links like:
/fclick.php?variable

is to canonicalize them to:
/fclick.php?variable=

This however makes Scrapy follow an incorrect link and cause an error page to load. This is really fault of web script programmers really who use variables without value. But for the sake of robustness Scrapy should follow the correct links.

I made a small patch for this. All it does really is that when it faces variables with 0 length value it crops out the =.

[sdogi]

Patched url.py

[sdogi]

Diff:

154a155,157

for pair in keyvals:
if len(pair[1]) == 0:
query = query.replace(pair[0] + "=", pair[0])

[sdogi]

Just in case there are more url.py 's in the package. The one I'm talking about is:
scrapy/utils/url.py