Gusto/buildkite-builder

Split upload of large pipelines into multiple requests so that we don't hit Buildkite API limits

ivy opened this issue · 2 comments

ivy commented

Buildkite imposes a default limit of 500 jobs in a single buildkite-agent pipeline upload invocation. We should automatically break up the pipeline for large batches.

We chatted about this one the other day. The 500 job limit is what Buildkite defaults to, but we requested they increase it to 600.

I think if we wanted to increase parallelism of rspec past 600, I don't know if Knapsack Pro is able to handle jobs split across pipelines. We could at least split rspec and graphql into their own pipelines though.

ngan commented

This is done with detached groups.