爬虫踩坑系列——etree.HTML解析异常

在爬虫的过程中，难免会遇到各种各样的问题。在这里，为大家分享一个关于etree.HTML解析异常的问题。
1.问题描述：
爬虫过程中，一般会使用requests.get()方法获取一个网页上的HTML内容，然后通过lxml库中的etree.HTML来解析这个网页的结构，最后通过xpath获取自己所需的内容。

本人爬虫的具体代码可简单抽象如下：

	res = requests.get(url)
	html = etree.HTML(res.text)
	contents = html.xpaht('//div/xxxx')

然后遇到了如下的错误信息：

	Traceback (most recent call last):
	  File "xxxxxxxx.py", line 157, in <module>
	    get_website_title_content(url)
	  File "xxxxxxxx.py", line 141, in get_website_title_content
	    html = etree.HTML(html_text)
	  File "src\lxml\etree.pyx", line 3170, in lxml.etree.HTML
	  File "src\lxml\parser.pxi", line 1872, in lxml.etree._parseMemoryDocument
	ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

关键错误就是 ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.

2.解决方法
通过查阅相关资料，造成这个错误的原因其实是requests返回的 res.text 和 res.content 两者区别的问题。查阅requests源代码中是text和content定义（如下所示）可知：res.text返回的是Unicode类型的数据，而res.content返回的是bytes类型的数据。

	@property
    def content(self):
        """Content of the response, in bytes."""

        if self._content is False:
            # Read the contents.
            if self._content_consumed:
                raise RuntimeError(
                    'The content for this response was already consumed')

            if self.status_code == 0 or self.raw is None:
                self._content = None
            else:
                self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''

        self._content_consumed = True
        # don't need to release the connection; that's been handled by urllib3
        # since we exhausted the data.
        return self._content

	@property
    def text(self):
        """Content of the response, in unicode.

        If Response.encoding is None, encoding will be guessed using
        ``chardet``.

        The encoding of the response content is determined based solely on HTTP
        headers, following RFC 2616 to the letter. If you can take advantage of
        non-HTTP knowledge to make a better guess at the encoding, you should
        set ``r.encoding`` appropriately before accessing this property.
        """

        # Try charset from content-type
        content = None
        encoding = self.encoding

        if not self.content:
            return str('')

        # Fallback to auto-detected encoding.
        if self.encoding is None:
            encoding = self.apparent_encoding

        # Decode unicode from given encoding.
        try:
            content = str(self.content, encoding, errors='replace')
        except (LookupError, TypeError):
            # A LookupError is raised if the encoding was not found which could
            # indicate a misspelling or similar mistake.
            #
            # A TypeError can be raised if encoding is None
            #
            # So we try blindly encoding.
            content = str(self.content, errors='replace')

        return content

导致该错误的原因是etree解析是不支持编码声明的Unicode字符串的。
因此解决方法很简单，第一种就是直接使用 res.content，如下：

	res = requests.get(url)
	html = etree.HTML(res.content )
	contents = html.xpath('//div/xxxx')

第二种方法则是将Unicode字符串转换为bytes数组，如下：

	res = requests.get(url)
	html_text = bytes(bytearray(res.text, encoding='utf-8'))
	html = etree.HTML(html_text)
	contents = html.xpath('//div/xxxx')