jerrymakesjelly/autoremove-torrents

程序卡住

jerrymakesjelly opened this issue · 12 comments

jerrymakesjelly commented

具体配置文件如下：

task1:
  client: qbittorrent
  host: http://127.0.0.1:2010
  username: aaaa
  password: aaaaaa
  strategies:
    strategy_mteaaa:
      trackers: mteaaa.me
      ratio: 3.0
      seed_size:
        limit: 5200
        action: remove-old-seeds
    strategy_hdskyaaa:
      trackers: hdskyaaa.me
      seed_size:
        limit: 1000
        action: remove-old-seeds
  delete_data: true

trackers是一个列表，所以应该写成

trackers:
  - mteaaa.me

和

trackers:
  - hdskyaaa.me

多谢解答，另外种子数量多的时候，经常卡死，能看下如何提高遍历效率吗

卡死的时候程序运行到哪个阶段？是获取种子阶段还是运行策略阶段？

主要是两种情况卡死，一种是login时卡着不动，一个是login成功后，获取种子阶段
我之前设置的是10分钟运行一次，没想到过了一段时间系统都卡的不行，autoremove的进程大概有四五百个，只是偶尔能成功完成一次执行。与之对比的是同期flexget 能够正常login 添加下载任务。

Originally posted by @monokoo in #13 (comment)

jerrymakesjelly commented

#13 (comment)

@monokoo 程序通过WebAPI向客户端发起请求，不过我没有设置请求的超时时间。因此，在登录或获取种子阶段卡死很有可能是qBittorrent本身的问题，建议同时查看该时间点的qBittorrent和FlexGet的日志，或许可以更快地找到原因。

我翻了一下日志，我的盒子也出现了程序卡住不动的情况，不过这是由于机器负荷过重引起的，当qBittorrent返回数据后也没有驻留的进程。下一个版本我会添加超时机制，先防止无限等待的事情发生。

wxlg1117 commented

当我的qb跑的种子数目多起来后（大概500），测试模式时就发生了卡住：
root@nodeovh1:~# /usr/local/bin/autoremove-torrents --view --conf=/data/autoremove-torrents/config.yml
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.main INFO: Auto Remove Torrents 1.2.5
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.main INFO: Loading configurations...
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.main INFO: Found 1 task(s) in the file.
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.task INFO: Running task 'task'...
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.task INFO: Logging in...
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.task INFO: Login successfully. The client is qBittorrent v4.1.5.
Mon, 08 Apr 2019 12:27:40 autoremovetorrents.task INFO: Getting all the torrents...

qb日志显示：
(N) 2019-04-08T12:27:40 - 网页API登录成功。IP：::ffff:127.0.0.1

因为我刷的opencd，所以服务器基本没负载load在0.08左右，另外flexget通过api添加种子到qb正常

大婶们，怎么配置让显示更详细的日志啊？

  • 👀1
jerrymakesjelly commented

@wxlg1117 卡住的时候，用浏览器能顺利打开WebUI吗？

wxlg1117 commented

@jerrymakesjelly,qb webui操作完全正常啊，art卡住的期间flexget定时任务还正常向qb添加种子呢

wxlg1117 commented

卡住后通过ctrl+c中断：
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.main INFO: Auto Remove Torrents 1.2.5
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.main INFO: Loading configurations...
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.main INFO: Found 1 task(s) in the file.
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.task INFO: Running task 'task'...
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.task INFO: Logging in...
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.task INFO: Login successfully. The client is qBittorrent v4.1.5.
Tue, 09 Apr 2019 18:12:52 autoremovetorrents.task INFO: Getting all the torrents...
^CTraceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/autoremove-torrents", line 11, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/main.py", line 63, in main
pre_processor(sys.argv[1:])
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/main.py", line 50, in pre_processor
Task(task_name, result[task_name], not view_mode).execute()
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/task.py", line 119, in execute
self._get_torrents()
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/task.py", line 88, in _get_torrents
self._torrents.append(self._client.torrent_properties(hash_value))
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/client/qbittorrent.py", line 93, in torrent_properties
trackers = self._torrent_trackers(torrent_hash)
File "/usr/local/lib/python3.6/dist-packages/autoremovetorrents/client/qbittorrent.py", line 69, in _torrent_trackers
cookies=self._cookies).json()
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/local/lib/python3.6/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
File "/usr/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.6/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt

  • 👀1
jerrymakesjelly commented

我往qbittorrent v4.1.5添加了500个种子后成功复现了这个问题，感谢您的反馈 @wxlg1117

不过比较有意思的是，当art与qb运行在同一台机器时就会发生获取种子时卡住的问题，但若qb运行在远程机器而在本地使用art访问它就不会出现这个问题，所以看起来问题的原因可能是访问速度过快？

我会进一步查明原因 😃

wxlg1117 commented

@jerrymakesjelly,问题解决了没呢？

jerrymakesjelly commented

@wxlg1117 已经改了 目前在500种子的机器上定时运行观察 没问题了发布

wxlg1117 commented

@jerrymakesjelly dev分支提交的么？

wxlg1117 commented

root@nodeovh1:~# /usr/local/bin/autoremove-torrents --view --conf=/data/autoremove-torrents/config.yml [1566/1566]
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.main INFO: Auto Remove Torrents 1.2.5
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.main INFO: Loading configurations...
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.main INFO: Found 1 task(s) in the file.
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.task INFO: Running task 'task'...
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.task INFO: Logging in...
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.task INFO: Login successfully. The client is qBittorrent v4.1.5.
Mon, 15 Apr 2019 16:17:47 autoremovetorrents.task INFO: Getting all the torrents...
Mon, 15 Apr 2019 16:17:48 autoremovetorrents.task INFO: Please wait...We have found 49 seed(s).
Mon, 15 Apr 2019 16:17:49 autoremovetorrents.task INFO: Please wait...We have found 106 seed(s).
Mon, 15 Apr 2019 16:17:50 autoremovetorrents.task INFO: Please wait...We have found 164 seed(s).
Mon, 15 Apr 2019 16:17:51 autoremovetorrents.task INFO: Please wait...We have found 222 seed(s).
Mon, 15 Apr 2019 16:17:52 autoremovetorrents.task INFO: Please wait...We have found 279 seed(s).
Mon, 15 Apr 2019 16:17:53 autoremovetorrents.task INFO: Please wait...We have found 335 seed(s).
Mon, 15 Apr 2019 16:17:54 autoremovetorrents.task INFO: Please wait...We have found 391 seed(s).
Mon, 15 Apr 2019 16:17:55 autoremovetorrents.task INFO: Please wait...We have found 447 seed(s).
Mon, 15 Apr 2019 16:17:56 autoremovetorrents.task INFO: Please wait...We have found 503 seed(s).
Mon, 15 Apr 2019 16:17:57 autoremovetorrents.task INFO: Please wait...We have found 560 seed(s).
Mon, 15 Apr 2019 16:17:58 autoremovetorrents.task INFO: Found 596 seed(s) in the client.
Mon, 15 Apr 2019 16:17:58 autoremovetorrents.strategy INFO: Running strategy open.cd...
Mon, 15 Apr 2019 16:17:58 autoremovetorrents.strategy INFO: Total: 408 seed(s). 341 seed(s) can be removed.
Mon, 15 Apr 2019 16:17:58 autoremovetorrents.strategy INFO: To be deleted:
Mon, 15 Apr 2019 16:17:58 autoremovetorrents.strategy INFO: 群星 热门抖音60首 4CD 2018 - WAV 整轨
Size:2.41GiB Ratio:2.879 Total Uploaded:6.93GiB Seeding Time:26d 21:44:27 Category:OPENCD
Create Time:2019-03-19 17:28:03
......

感谢大佬修复及时，初步试了几次暂时没什么问题;
PS:
INFO: Getting all the torrents...
INFO: Please wait...We have found 49 seed(s).
INFO: Found 596 seed(s) in the client.
这里的seed(s) 推荐换用 torrent 比较好吧？

  • 👀1
jerrymakesjelly commented

好的，可以改

初步认为导致art卡住的原因是qbittorrent的WebAPI在请求速度过快的时候会卡住，目前已经向qbittorrent项目提交了这个问题
qbittorrent/qBittorrent#10487 (comment)

art目前限制了所有客户端的WebAPI的请求速度，最大速度为100个种子/秒

wxlg1117 commented

Wed, 17 Apr 2019 09:27:44 main.py[line:44] INFO Auto Remove Torrents 1.2.5
Wed, 17 Apr 2019 09:27:44 main.py[line:46] INFO Loading configurations...
Wed, 17 Apr 2019 09:27:44 main.py[line:49] INFO Found 1 task(s) in the file.
Wed, 17 Apr 2019 09:27:44 task.py[line:116] INFO Running task 'task'...
Wed, 17 Apr 2019 09:27:44 task.py[line:74] INFO Logging in...
Wed, 17 Apr 2019 09:27:44 task.py[line:79] INFO Login successfully. The client is qBittorrent v4.1.5.
Wed, 17 Apr 2019 09:27:44 task.py[line:83] INFO Getting all the torrents...
Wed, 17 Apr 2019 09:27:45 task.py[line:91] INFO Please wait...We have found 205 seed(s).
Wed, 17 Apr 2019 09:27:46 task.py[line:91] INFO Please wait...We have found 439 seed(s).
Wed, 17 Apr 2019 09:27:46 task.py[line:93] INFO Found 604 seed(s) in the client.
Wed, 17 Apr 2019 09:27:46 task.py[line:99] DEBUG Strategy open.cd was specified to process 604 torrent(s).
Wed, 17 Apr 2019 09:27:46 strategy.py[line:96] INFO Running strategy open.cd...
Wed, 17 Apr 2019 09:27:46 strategy.py[line:54] DEBUG Filter Category is working to process 604 torrent(s): Accept All: No, Accept: 1, Reject: Not Specified.
Wed, 17 Apr 2019 09:27:46 strategy.py[line:54] DEBUG Filter Tracker is working to process 416 torrent(s): Accept All: Yes, Accept: Not Specified, Reject: Not Specified.
Wed, 17 Apr 2019 09:27:46 strategy.py[line:54] DEBUG Filter Status is working to process 416 torrent(s): Accept All: Yes, Accept: Not Specified, Reject: Not Specified.
Wed, 17 Apr 2019 09:27:46 strategy.py[line:79] DEBUG Remove condition seeding_time was specified to process 416 torrent(s).
...

@jerrymakesjelly ,监控到了代码更新，然后我重新安装了下并执行，加载种子列表速度快多了...不过那个seed还是没改...

  • 👍1