Podcastindex-org/podping-hivewatcher

simple-watcher.py error: Request Timeout Retry RPC Call on node: https://api.hive.blog

Opened this issue · 3 comments

$ python simple-watcher.py
Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (1/5)

Retrying in 0 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (2/5)

Retrying in 2 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (3/5)

Retrying in 3 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (4/5)

Retrying in 5 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (5/5)

Retrying in 6 seconds

output: None of identifier 81906994
Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (1/5)

Retrying in 0 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (2/5)

Retrying in 2 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (3/5)

Retrying in 3 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (4/5)

Retrying in 5 seconds

Error: Request Timeout
Retry RPC Call on node: https://api.hive.blog (5/5)

Retrying in 6 seconds

Traceback (most recent call last):
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 89, in <module>
    main()
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 56, in main
    start_block = block_num_back_in_minutes(blockchain, 15)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 45, in block_num_back_in_minutes
    block_num = blockchain.get_estimated_block_num(back_time)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchain.py", line 325, in get_estimated_block_num
    last_block = self.get_current_block()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchain.py", line 300, in get_current_block
    return Block(
           ^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/block.py", line 70, in __init__
    super(Block, self).__init__(
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchainobject.py", line 137, in __init__
    self.refresh()
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/block.py", line 163, in refresh
    raise BlockDoesNotExistsException("output: %s of identifier %s" % (str(block), str(self.identifier)))
beem.exceptions.BlockDoesNotExistsException: output: None of identifier 81906994

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 92, in <module>
    main()
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 56, in main
    start_block = block_num_back_in_minutes(blockchain, 15)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/build/podping-hivewatcher/simple-watcher.py", line 45, in block_num_back_in_minutes
    block_num = blockchain.get_estimated_block_num(back_time)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchain.py", line 325, in get_estimated_block_num
    last_block = self.get_current_block()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchain.py", line 300, in get_current_block
    return Block(
           ^^^^^^
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/block.py", line 70, in __init__
    super(Block, self).__init__(
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/blockchainobject.py", line 137, in __init__
    self.refresh()
  File "/home/ryan/.local/lib/python3.11/site-packages/beem/block.py", line 163, in refresh
    raise BlockDoesNotExistsException("output: %s of identifier %s" % (str(block), str(self.identifier)))
beem.exceptions.BlockDoesNotExistsException: output: None of identifier 81907007

@agates Is the simple watcher deprecated at this point? I think it basically is right?

FYI, the same node also fails in hive-watcher.py among others:

2024-01-16T02:09:14+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.6s (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16T02:09:15+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.4s (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16T02:09:16+1100 | ERROR backoff MainThread : |  Giving up _req(...) after 3 tries (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16 02:09:16,325 lighthive    ERROR    521 Server Error:  for url: https://rpc.mahdiyari.info/
2024-01-16T02:09:16+1100 | ERROR lighthive MainThread : |  521 Server Error:  for url: https://rpc.mahdiyari.info/
2024-01-16 02:09:16,325 lighthive    WARNING  Ignoring node https://rpc.mahdiyari.info for 3600 seconds: ('get_dynamic_global_properties',), {}
2024-01-16T02:09:16+1100 | WARNING lighthive MainThread : |  Ignoring node https://rpc.mahdiyari.info for 3600 seconds: ('get_dynamic_global_properties',), {}
2024-01-16T02:09:22+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.5s (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16T02:09:23+1100 | INFO backoff MainThread : |  Backing off _req(...) for 1.0s (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16T02:09:24+1100 | ERROR backoff MainThread : |  Giving up _req(...) after 3 tries (requests.exceptions.HTTPError: 521 Server Error:  for url: https://rpc.mahdiyari.info/)
2024-01-16 02:09:24,725 lighthive    ERROR    521 Server Error:  for url: https://rpc.mahdiyari.info/
2024-01-16 02:09:24,725 lighthive    ERROR    521 Server Error:  for url: https://rpc.mahdiyari.info/
2024-01-16T02:09:24+1100 | ERROR lighthive MainThread : |  521 Server Error:  for url: https://rpc.mahdiyari.info/
2024-01-16 02:09:24,726 lighthive    WARNING  Ignoring node https://rpc.mahdiyari.info for 3600 seconds: ('get_block', {'block_num': 81941038}), {}
2024-01-16 02:09:24,726 lighthive    WARNING  Ignoring node https://rpc.mahdiyari.info for 3600 seconds: ('get_block', {'block_num': 81941038}), {}
2024-01-16T02:09:24+1100 | WARNING lighthive MainThread : |  Ignoring node https://rpc.mahdiyari.info for 3600 seconds: ('get_block', {'block_num': 81941038}), {}
2024-01-16T02:09:32+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.6s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3))
2024-01-16T02:09:37+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.4s (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3))
2024-01-16T02:09:41+1100 | ERROR backoff MainThread : |  Giving up _req(...) after 3 tries (requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3))
2024-01-16 02:09:41,779 lighthive    ERROR    HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3)
2024-01-16 02:09:41,779 lighthive    ERROR    HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3)
2024-01-16T02:09:41+1100 | ERROR lighthive MainThread : |  HTTPSConnectionPool(host='api.hive.blog', port=443): Read timed out. (read timeout=3)
2024-01-16 02:09:41,780 lighthive    WARNING  Ignoring node https://api.hive.blog for 3600 seconds: ('get_block', {'block_num': 81941039}), {}
2024-01-16 02:09:41,780 lighthive    WARNING  Ignoring node https://api.hive.blog for 3600 seconds: ('get_block', {'block_num': 81941039}), {}
2024-01-16T02:09:41+1100 | WARNING lighthive MainThread : |  Ignoring node https://api.hive.blog for 3600 seconds: ('get_block', {'block_num': 81941039}), {}
2024-01-16T02:09:42+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.7s (requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://rpc.ecency.com/)
2024-01-16T02:09:44+1100 | INFO backoff MainThread : |  Backing off _req(...) for 0.7s (requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://rpc.ecency.com/)
2024-01-16T02:09:46+1100 | ERROR backoff MainThread : |  Giving up _req(...) after 3 tries (requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://rpc.ecency.com/)
2024-01-16 02:09:46,440 lighthive    ERROR    502 Server Error: Bad Gateway for url: https://rpc.ecency.com/
2024-01-16 02:09:46,440 lighthive    ERROR    502 Server Error: Bad Gateway for url: https://rpc.ecency.com/
2024-01-16T02:09:46+1100 | ERROR lighthive MainThread : |  502 Server Error: Bad Gateway for url: https://rpc.ecency.com/
2024-01-16 02:09:46,440 lighthive    WARNING  Ignoring node https://rpc.ecency.com for 3600 seconds: ('get_block', {'block_num': 81941039}), {'num_retries': 2}
2024-01-16 02:09:46,440 lighthive    WARNING  Ignoring node https://rpc.ecency.com for 3600 seconds: ('get_block', {'block_num': 81941039}), {'num_retries': 2}
2024-01-16T02:09:46+1100 | WARNING lighthive MainThread : |  Ignoring node https://rpc.ecency.com for 3600 seconds: ('get_block', {'block_num': 81941039}), {'num_retries': 2}
agates commented

I've never used simple-watcher for what it's worth... but it might want to remove the failing server from the accepted node list if it continues.

They come and go, so I'm not surprised. That's why I added the backoff logic to lighthive.