Changeset 1936:a32f3fe4fe6d for scrapy/http/response/text.py

Timestamp:

02/24/10 14:01:29 (7 months ago)

Author:

Pablo Hoffman <pablo@…>

Branch:

default

Message:

Fixed encoding issue (reported in #135) when the encoding declared in the HTTP header is unknown. This is the patch proposed by Rolando, with an update to the Request/Response documentation.

Files:

• scrapy/http/response/text.py (modified) (2 diffs)

scrapy/http/response/text.py

@@ -6 +6 @@
 """
 
+import codecs
 import re
 
@@ -65 +66 @@
             encoding = self._ENCODING_RE.search(content_type)
             if encoding:
-                return encoding.group(1)
+                enc = encoding.group(1)
+                try:
+                    codecs.lookup(enc) # check if the encoding is valid
+                    return enc
+                except LookupError:
+                    pass
 
     @memoizemethod_noargs