'Selector' object has no attribute '_default_type'

Question

'Selector' object has no attribute '_default_type'

Opened this issue 3 months ago · 0 comments

目标网站地址

css selector / xpath 内容

from scrapy.selector import Selector

# 示例代码段
html = "<html><body><p>Hello World</p></body></html>"
selector = Selector(text=html)
print(selector.xpath("//p/text()").get())

实际发生了什么？

2024-06-07 22:57:22,720 [ThreadPoolExecutor-0_0:140009119401728] [scheduler:monitor:195] [ERROR]- Traceback (most recent call last):
File "/home/ubuntu/work/WebMonitor/task/utils/scheduler.py", line 138, in monitor
content = get_content(url, is_chrome, selector_type, selector,
File "/home/ubuntu/work/WebMonitor/task/utils/extract_info.py", line 62, in get_content
content_dict = selector_handler.get_by_xpath(url, selector_dict,
File "/home/ubuntu/work/WebMonitor/task/utils/selector/request_selector.py", line 30, in get_by_xpath
result[key] = self.xpath_parse(html, xpath_ext)
File "/home/ubuntu/work/WebMonitor/task/utils/selector/selector.py", line 22, in xpath_parse
res = Selector(
File "/home/ubuntu/work/WebMonitor/lib/python3.8/site-packages/scrapy/selector/unified.py", line 86, in __init__
st = _st(response, type or self._default_type)
AttributeError: 'Selector' object has no attribute '_default_type'

部署相关信息

手动

环境信息

(WebMonitor) ubuntu@ip:~/work/WebMonitor$ pip list | grep parse
parsel               1.9.1

原因

Scrapy 和 Parsel 之间的版本冲突可能会导致这种错误。将 Parsel 降级到 1.7.0 解决了问题

解决办法

修改 requirements.txt 文件，添加下面内容

parsel==1.7.0