Cloud Controller should install new buildpacks in a more deterministic order
tcdowney opened this issue · 0 comments
Issue
Operators specify an array of buildpacks for Cloud Controller to install in capi-release. Cloud Controller groups these by name and then enqueues jobs to either update existing buildpacks or install new ones. These jobs are picked up by the local workers in the order in which they get to them. Since multiple workers are handling these jobs this can cause the buildpacks to be installed in non-deterministic orders.
Context
Buildpack order matters since it affects buildpack auto-detection. Many operators assume that the order they specify for their buildpacks in their CF deployment manifest will result in them using that order for auto-detection. The fact that the installation order/position for newly installed buildpacks is non-deterministic is surprising to operators.
Steps to Reproduce
This is difficult to reproduce due to the randomness, but after multiple deploys installing a fresh set of buildpacks (not updating existing ones) you should expect to see the buildpacks installed in a different order each time.
Expected result
I would expect the buildpacks to be installed in a consistent order.
Current result
The buildpacks are usually installed in the specified order (grouped by buildpack name in cases where multiple versions of buildpacks exist for different stacks). However occasionally the buildpacks will be installed in completely different orders which can cause issues with auto-detection.
Examples from two different installations:
➜ cf buildpacks | egrep "nginx|web_servers"
4 nginx_buildpack cflinuxfs3 true false READY nginx_buildpack-cached-cflinuxfs3-v1.2.14.zip
14 web_servers_cnb_beta cflinuxfs4 true false READY web-servers-cnb-buildpack-cflinuxfs4-v0.16.0.zip
22 nginx_buildpack cflinuxfs4 true false READY nginx_buildpack-cached-cflinuxfs4-v1.2.14.zip
➜ cf buildpacks | egrep "nginx|web_servers"
7 nginx_buildpack cflinuxfs3 true false READY nginx_buildpack-cached-cflinuxfs3-v1.2.14.zip
8 nginx_buildpack cflinuxfs4 true false READY nginx_buildpack-cached-cflinuxfs4-v1.2.14.zip
24 web_servers_cnb_beta cflinuxfs4 true false READY web-servers-cnb-buildpack-cflinuxfs4-v0.16.0.zip
Possible Fix
Operators can reorder the buildpacks to their liking using cf update-buildpack BUILDPACK_NAME --position <position>
. This updates the position column in the data base. We could consider having our install buildpacks task specify a position
based on the install_buildpacks
order when installing new buildpacks instead of leaving it up to chance.
mysql> select name,position from buildpacks;
+-----------------------+----------+
| name | position |
+-----------------------+----------+
| staticfile_buildpack | 1 |
| java_buildpack | 2 |
| ruby_buildpack | 3 |
| dotnet_core_buildpack | 4 |
| nodejs_buildpack | 5 |
| go_buildpack | 6 |
| python_buildpack | 7 |
| nginx_buildpack | 8 |
| php_buildpack | 9 |
| r_buildpack | 10 |
| binary_buildpack | 11 |
+-----------------------+----------+
11 rows in set (0.04 sec)
Notes
- It is important that the position of existing buildpacks is not changed by any fix for this issue since an operator may have explicitly ordered them using CLI commands. It should only affect newly installed buildpacks.