Sharded GCP resources
Closed this issue · 2 comments
Given feedback from recent work such as #592, create a first-class concept of a "GCE worker shard" which accepts these variables:
- project (string)
- region (string)
- name (string)
- zones (list)
- shard_count (number)
- worker_count_per_shard (number)
- (any gcloud-cleanup and travis-worker config injection)
e.g.:
module "gce_worker_shards_paid" {
# ...
project = "${var.project}"
region = "us-central1"
name = "${var.env}-${var.index}-paid"
zones = ["a", "b", "f"]
shard_count = 6
worker_count_per_shard = 5
# ...
}
This should work similarly to the current gce_worker_group
+ gce_worker
modules, but without having knowledge of three distinct instance groups (com, com-free, org). The GCE worker shard module should be responsible for generating service accounts and keys for both workers and a shard-specific gcloud-cleanup deployment. The capacity of the shard should not exceed 300 travis-worker processes and a single gcloud-cleanup process. In the above example, the worker count per shard of 5
multiplied by a pool size (not shown) of 50
multiplied by 6
shards would make a total pool size of 1500
.
This generally looks good to me, one open question here would be the NATs. Also, I'd say this is dependent on whether we can get our quotas lifted. If we can't, I'd be inclined to going back to sharding by project and adding at least one more project to our setup.
FTR I'm not planning to do any work toward this, nor do I encourage anyone else to do work toward this, at least until we get some more/better feedback from friends at Google about API quotas.