a proposal for warming up strategy for newly joined node

[xujihui1985]

Feature request:

I'd like to propose three enhancements to dfdaemon in Dragonfly:

Proposal:
InitialDelayDuration: This feature would introduce a delay before dfdaemon announces itself to the scheduler, helping to manage the load during startup phases.

MinimalPeers: This setting would allow a daemon to download from peers only if the number of peers possessing the required pieces meets a predefined minimum. This ensures more efficient and reliable downloads.

Random Peer Selection: Implement a mechanism to randomly select peers from the list of candidates. until all candidates exhaust. This approach aims to prevent any single peer from becoming a bottleneck or hotspot due to excessive requests.

Use case:

Environment
3 master node
3 scheduler
0 peer seed
1000 dfdaemon
diskio capacity for dfdaemon roughly 150MB/s BPS

We've observed that when a new batch of machines is added to the cluster, the latency for image downloads increases significantly, accompanied by a spike in the cancellation rate of dfdaemon client requests.

"msg": "receive reminding piece tasks request error: rpc error: code = Canceled desc = context canceled"

Additionally, disk utilization on these nodes approaches 100%, and the I/O wait times are extremely high. This leads me to suspect that the timeouts are due to the new nodes joining the cluster and simultaneously pulling large amounts of image data for daemonsets, while applications on these nodes are also starting up, resulting in high I/O utilization.

This situation is further exacerbated if these nodes also start uploading pieces as peers, worsening the I/O utilization.

These proposed features aim to reduce the load and improve the efficiency of the data distribution in large-scale deployments, specifically during periods of high activity such as adding new nodes to a cluster.