Cloud Controller should install new buildpacks in a more deterministic order

[tcdowney]

Issue

Operators specify an array of buildpacks for Cloud Controller to install in capi-release. Cloud Controller groups these by name and then enqueues jobs to either update existing buildpacks or install new ones. These jobs are picked up by the local workers in the order in which they get to them. Since multiple workers are handling these jobs this can cause the buildpacks to be installed in non-deterministic orders.

Context

Buildpack order matters since it affects buildpack auto-detection. Many operators assume that the order they specify for their buildpacks in their CF deployment manifest will result in them using that order for auto-detection. The fact that the installation order/position for newly installed buildpacks is non-deterministic is surprising to operators.

Steps to Reproduce

This is difficult to reproduce due to the randomness, but after multiple deploys installing a fresh set of buildpacks (not updating existing ones) you should expect to see the buildpacks installed in a different order each time.

Expected result

I would expect the buildpacks to be installed in a consistent order.

Current result

The buildpacks are usually installed in the specified order (grouped by buildpack name in cases where multiple versions of buildpacks exist for different stacks). However occasionally the buildpacks will be installed in completely different orders which can cause issues with auto-detection.

Examples from two different installations:

➜ cf buildpacks | egrep "nginx|web_servers"
4          nginx_buildpack          cflinuxfs3    true      false    READY   nginx_buildpack-cached-cflinuxfs3-v1.2.14.zip
14         web_servers_cnb_beta     cflinuxfs4    true      false    READY   web-servers-cnb-buildpack-cflinuxfs4-v0.16.0.zip
22         nginx_buildpack          cflinuxfs4    true      false    READY   nginx_buildpack-cached-cflinuxfs4-v1.2.14.zip

➜ cf buildpacks | egrep "nginx|web_servers"
7          nginx_buildpack          cflinuxfs3   true      false    READY   nginx_buildpack-cached-cflinuxfs3-v1.2.14.zip
8          nginx_buildpack          cflinuxfs4   true      false    READY   nginx_buildpack-cached-cflinuxfs4-v1.2.14.zip
24         web_servers_cnb_beta     cflinuxfs4   true      false    READY   web-servers-cnb-buildpack-cflinuxfs4-v0.16.0.zip

Possible Fix

Operators can reorder the buildpacks to their liking using cf update-buildpack BUILDPACK_NAME --position <position>. This updates the position column in the data base. We could consider having our install buildpacks task specify a position based on the install_buildpacks order when installing new buildpacks instead of leaving it up to chance.

mysql> select name,position from buildpacks;
+-----------------------+----------+
| name                  | position |
+-----------------------+----------+
| staticfile_buildpack  |        1 |
| java_buildpack        |        2 |
| ruby_buildpack        |        3 |
| dotnet_core_buildpack |        4 |
| nodejs_buildpack      |        5 |
| go_buildpack          |        6 |
| python_buildpack      |        7 |
| nginx_buildpack       |        8 |
| php_buildpack         |        9 |
| r_buildpack           |       10 |
| binary_buildpack      |       11 |
+-----------------------+----------+
11 rows in set (0.04 sec)

Notes

• It is important that the position of existing buildpacks is not changed by any fix for this issue since an operator may have explicitly ordered them using CLI commands. It should only affect newly installed buildpacks.