filecoin-project/boost

New incoming offline deals are blocked when boost receives 200+ online deals.

cryptowhizzard opened this issue · 3 comments

cryptowhizzard commented

Checklist

  • This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
  • This is not a new feature request. If it is, please file a feature request instead.
  • This is not an enhancement request. If it is, please file a improvement suggestion instead.
  • I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
  • I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
  • I did not make any code changes to boost.

Boost component

  • boost daemon - storage providers
  • boost client
  • boost UI
  • boost data-transfer
  • boost index-provider
  • Other

Boost Version

Latest master october 2 2023

Describe the Bug

When boost receives a lot of online deals ( 200+ ) the channel for making new offline deals seems to get blocked.

Both client and SP receive error:
Error: send proposal rpc: failed to read response: stream reset

I verified the numbers for storage in the config. They are all above 1000 ( lotus-miner as well as boost ).

Logging Information

Error: send proposal rpc: failed to read response: stream reset

Repo Steps

  1. Run '...'
  2. Do '...'
  3. See error '...'
    ...
LexLuthr commented

We need to move the deal acceptance flow to goroutines instead of handling them all in a single thread.

cryptowhizzard commented

This PR 1746 seems to improve a bit but the problem resides in provider_dealfilter.go.

Boost needs to output a lot of things about sectors and statuses to the external filter ( if configured ) and this is where the delay originates. Disabling everything except the deal parameters like client / provider / cid seems to resolve the problem in combination with this PR.

In my situation i am running a 30 PB+ machine and boost needs to collect a lot of info to send to that filter causing massive delays.

Please investigate per part what 's the exact cause of the delay in provider_dealfilter.go and let's see if we can find a way to work around that. I will migrate this now to my testing setup in next days so our customer's won't be affected anymore.

LexLuthr commented

@cryptowhizzard I would love for you to weigh in on #1747 (comment)
We can use the new issue to figure out what design would be most efficient.