Split upload of large pipelines into multiple requests so that we don't hit Buildkite API limits
ivy opened this issue · 2 comments
ivy commented
Buildkite imposes a default limit of 500 jobs in a single buildkite-agent pipeline upload
invocation. We should automatically break up the pipeline for large batches.
technicalpickles commented
We chatted about this one the other day. The 500 job limit is what Buildkite defaults to, but we requested they increase it to 600.
I think if we wanted to increase parallelism of rspec past 600, I don't know if Knapsack Pro is able to handle jobs split across pipelines. We could at least split rspec and graphql into their own pipelines though.
ngan commented
This is done with detached groups.