Optimize the efficiency of Bamboo synchronization
Closed this issue · 0 comments
hyrepo commented
Is your feature request related to a problem? Please describe.
The Bamboo synchronization is using the following steps:
- Call a Bamboo API to get the maximum build number
- For each build, call a Bamboo API to get build details and save it into DB
For step 2, currently we're using parallelStream
and RestTemplate
to send request parallelly, whose performance is limited by the number of CPU cores.
Therefore, when sync large amount builds (1000+), it may take several minutes to finish, which caused time out.
Describe the solution you'd like
- Instead of using
RestTemplate
,WebClient
could be considered, since it send requests asynchronous. - Kotlin coroutines also could be considered
Additional context
The Current solution is setting the keepalive_timeout
in Nginx to 10 minutes, which is not an ideal long-term solution.