Clarification on Build and Release Pipelines
vslepakov opened this issue · 3 comments
Hi,
I don't quite understand the pipeline design.
In the build you basically just build modules without pushing them to a container registry.
Then in the Release phase you use the code again and do basically exactly the same things: replacing tokens and building modules, now pushing them to a directory. As I see it in this set up you don't really need the Build pipeline, you could just clone the repo as a source.
Wouldn't it make more sense to replace tokens, build module, push modules to a container registry as part of the Build pipeline and then you could just create a deployment in the release pipeline using those pushed images (tagged with build number for example)?
I would also go for unified pipelines (build and release in one yaml). I'd be happy to contribute on that.
Best Regards
Vitaliy
I asked myself similar questions when I was juggling creating something that could both teach and serve as an example of best practices, so there may be some give/take in trying to accommodate both. Also, DevOps in general is rarely one-size-fits-all. We could definitely go in the direction that you are suggesting (using a unified pipeline), but this seems like it has potential to overproduce deployments that may be unnecessary. Another smaller instructional benefit to keeping the pipelines separate is that it allows for the lab content to demonstrate using a build-definition in code (yaml) in addition to a visual pipeline (the Release build).
Going deeper, my line of thinking regarding separating the build and release pipelines is to distinguish between what should be produced during Continuous Integration versus Deployment. For Continuous Integration, I follow the principle that any code that is checked should build successfully, if it does not build in CI then it should not be accepted into the current branch. In CI, we are only concerned with whether our code is good enough to be checked-in.
Given this goal, there is no need to produce a Deployment for every minor code change and as such, there is no need to push artifacts to a container registry during every CI build. This is more apparent when multiple developers are working on separate features which require completion before a targeted Deployment should be made to QA or Prod. If a developer needs to test something that has only passed CI, they would do best to pull source and test images locally instead of cutting a Deployment to QA or Prod. After all, CI enforces that building and testing locally should always be possible with whatever is currently checked-in. Deploying to QA should be seen as a deliberate decision as the testing in that phase is more rigorous and often handled by a separate QA team. In my opinion, separating these processes allows for increased developer velocity and controlled promotion to QA, which is especially important on larger projects and solutions involving a large number of IoT Edge devices.
At the end of the day, the most important artifact created during the development of any IoT Edge solution is the deployment.template.json which defines how dependent modules should be retrieved and created. This is the only thing that ultimately drives eventual deployment to an IoT Edge device. As such, it seems reasonable to ensure at a minimum that dependent modules are published to a container registry at the time that the associated deployment.template.json is created.
As-is, the Release pipeline can be based on any commit that has passed CI allowing for a little bit of versatility instead of always deploying based on last check-in and allows for controlled deployment. Within the Release pipeline it performs the publish to registry and create deployment steps consecutively and then smoke tests that the generated deployment is sound by deploying to an ephemeral containerized IoT Edge device. This test ultimately gives us everything needed to confirm that a deployment is ready to be applied to a real device in QA or prod.
We could produce a container for each CI check-in and eventually use those containers in a release pipeline later, but what if we need to apply different settings for any dependent services (for example AppInsights in Dev vs Prod?) and what is the benefit of keeping those containers around when they can just as easily be built locally from what has currently passed CI? For this reason, my CI artifact is the unaltered deployment.template.json at the time of check-in (so that dependent services and other criteria can be replaced as needed to accommodate QA / Prod) and I only publish to a container registry when an associated deployment.template.json is created to go along with it.
@toolboc ,
thanks for your explanation, I really appreciate it, though I do not agree completely.
I do absolutely agree that there is need for a fast feedback loop, but from what I saw in the field, customers want to have higher degree of confidence regarding the quality of code on the master branch. So they for example might open a Pull Request fairly early and run PR-Builds, just building code and performing static code analysis, unit tests, credentials scanning, dependency (licence) checking (no containers yet) enabling fast feedback.
Then at some point the PR gets approved and merged into master which triggers the actual Integration-Build. There things like security scanning of containers, maybe signing etc. happen. The artifact created is, as you also mentioned, deployment.template.json file pointing to security scanned and approved containers. They also look at Git-Flow or Microsoft Release Flow as the basis for development. In fact I am missing this concepts in this repo. I assume this is just to not over-complicate and to let folks out there figure out what works best for them. Still this thought has some implications on the design of the pipeline itself.
Another thought that I am having is, that often there are different roles - devs who build stuff and folks responsible for the release. If the build happens as part of the Release pipeline those non-dev folks would be forced to either figure out on their own why a build (inside release pipeline) fails (security scanning issues, static code analysis) or they would need to get one of the devs to look at it. This would cause way too much overhead in my opinion.
Also performing those security and static analysis steps can take quite some time and could make a release take a while, which is why I personally prefer to do the heavy lifting upfront to also encounter issues earlier in the process and not first when I am about to release.
Also accounting for different configuration/settings depending on the stage/environment should not be a problem. This can still happen as part of the release phase using e.g. variables. I do also believe that stuff that changes depending on the environment should not be part of the code but rather configuration, so you would end up replacing tokens in e.g. deployment manifest even if containers already exist and pass configuration via environment variables. So I don't see a problem here producing containers in the build phase.
As for unified pipelines, I think that is the future. They so naturally let us version the complete pipeline definition alongside our code base (export and import of the visual pipeline is somewhat clumsy compared to YAML imho) and completely remove the risk, that folks end up not being able to build/release an older version of the code because the pipeline definition and code diverged over time.
This is just my two cents, I really appreciate the work you are doing here. Especially that azure-iot-edge-device-container is really awesome
Unless you want to add something to this conversation please consider this issue clarified and feel free to close it.
Thanks
Vitaliy
I really like the idea of incorporating vulnerability scanning. That is something that could be easily added in either pipeline as-is but might not be the best path forward. You bring up some good points, ideally I'd like to see an extremely fast CI dev loop on PR branches, with an additional pipeline that is triggered when a merge is made to Master that performs a bit more heavy lifting (vulnerability scans, possibly pushing containers etc.), and an additional controlled Deployment pipeline. Since we have the CI and Deployment pieces already, we could perhaps look at incorporating a new pipeline as you are suggesting in between them. I'd be happy to accept a PR here, as it should not require much change to the documentation and existing pipelines.