Gerapy/Gerapy

scrapy运行无日志,gerapy请求则会报错

Spider-sun opened this issue · 1 comments

2023-04-08T15:21:06+0000 [twisted.web.server.Request#critical]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/twisted/web/server.py", line 292, in render
body = resrc.render(self)
File "/usr/local/lib/python3.11/site-packages/twisted/web/_template_util.py", line 194, in render
self.d.addCallback(self._cbChild, request).addErrback(self._ebChild, request)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 532, in addCallback
return self.addCallbacks(callback, callbackArgs=args, callbackKeywords=kwargs)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 512, in addCallbacks
self._runCallbacks()
--- ---
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/usr/local/lib/python3.11/site-packages/twisted/web/_template_util.py", line 200, in _cbChild
request.render(resource.getChildForRequest(child, request))
File "/usr/local/lib/python3.11/site-packages/twisted/web/server.py", line 292, in render
body = resrc.render(self)
File "/usr/local/lib/python3.11/site-packages/twisted/web/_auth/wrapper.py", line 198, in render
return super().render(request)
File "/usr/local/lib/python3.11/site-packages/twisted/web/resource.py", line 268, in render
return m(request)
File "/usr/local/lib/python3.11/site-packages/twisted/web/static.py", line 657, in render_GET
producer.start()
File "/usr/local/lib/python3.11/site-packages/twisted/web/static.py", line 770, in start
self.fileObject.seek(self.offset)
builtins.OSError: [Errno 22] Invalid argument

2023-04-08T15:21:06+0000 [twisted.python.log#info] "172.17.0.1" - - [08/Apr/2023:15:21:05 +0000] "GET /logs/VulSpiders/cnnvdSpider/f7a99998d62011ed94c70242ac11000a.log HTTP/1.1" 500 95 "-" "python-requests/2.28.1"

这个问题是views.py中的job_log这个函数在发起请求时的请求头参数headers={'Range': 'bytes=-1000'}引起的,当日志信息为空时,发起请求获取最后1000字节的数据,会使scrapyd发生异常,这应该是scrapyd的问题