thoughtworks/metrik

Optimize the efficiency of Bamboo synchronization

Closed this issue · 0 comments

Is your feature request related to a problem? Please describe.

The Bamboo synchronization is using the following steps:

  1. Call a Bamboo API to get the maximum build number
  2. For each build, call a Bamboo API to get build details and save it into DB

For step 2, currently we're using parallelStream and RestTemplate to send request parallelly, whose performance is limited by the number of CPU cores.

Therefore, when sync large amount builds (1000+), it may take several minutes to finish, which caused time out.

Describe the solution you'd like

  • Instead of using RestTemplate, WebClient could be considered, since it send requests asynchronous.
  • Kotlin coroutines also could be considered

Additional context

The Current solution is setting the keepalive_timeout in Nginx to 10 minutes, which is not an ideal long-term solution.