werf/werf

Userspace builds without docker server (buildah)

distorhead opened this issue · 11 comments

distorhead commented

As a werf user I want to have an ability to run werf builds in Kubernetes itself. To run werf builds in Kubernetes werf should have a mode to work without docker server (as https://github.com/GoogleContainerTools/kaniko does).

donbeave commented

Maybe by adding buildpacks support?

https://buildpacks.io/

  • 👍2
tumauwt commented

Is this on the radar for future support?

distorhead commented

Currently implementing first version of werf with buildah included to build only dockerfiles in this PR: #3735

TODO check list:

  • Check werf binaries ldd dependencies;
    • Try to build werf with buildah and cgo enabled as statical binary
  • Try to run werf-build in native-rootless buildah mode in Kubernetes.
  • buildah-login should work for docker-with-fuse buildah mode.
  • Investigate ci-env + buildah-mode login issue.
  • Fix native-rootless buildah mode initialization procedure for werf.
  • Debug native-rootless buildah mode in werf-build.
  • Performance testing.
  • E2E testing.
  • Temporarily forbid usage of :local stages storage (werf-build command without --repo arg) with buildah container runtime.
  • 👍1
ilya-lesikov commented

Current performance testing results for building sample Dockerfile are:

  • Linux:
    • old Docker builder: 516 sec
    • new Buildah native builder: 481 sec
    • new Buildah non-native builder: 515 sec
  • macOS:
    • old Docker builder: 242 sec
    • new Buildah non-native builder: 347 sec
  • Windows
    • old Docker builder: 385 sec
    • new Buildah non-native builder: 434 sec

We are continuing to work on it.

develar commented

Will be Stapel covered or it is another issue? (ability to werf build without running Docker).

ilya-lesikov commented

@develar Yes, it will be, but after implementing Dockerfile builder with Buildah.

distorhead commented

Status update

Werf currently supports building of Dockerfile-based images using buildah container runtime by setting WERF_CONTAINER_RUNTIME_BUILDAH=auto|native-rootless|docker-with-fuse.

Next main steps:

  1. Run werf+buildah in Kubernetes (for Dockerfiles only): #3933
  2. Add buildah support for stapel images builder: #3936

Upd.

Corrected initial error: not WERF_BUILDAH_CONTAINER_RUNTIME, but WERF_CONTAINER_RUNTIME_BUILDAH environment variable.

ilya-lesikov commented

Current progress

Now werf can be run in Kubernetes or in standalone containers. We provide preconfigured images for this. Only available for Dockerfile builds for now, next step is to allow stapel builds with Buildah, thus allowing to run stapel builds in containers/Kubernetes too.

What's done and lands soon in all release channels up to stable:

  • Now WERF_BUILDAH_MODE env var used instead of WERF_CONTAINER_RUNTIME_BUILDAN to set Buidah mode/isolation. WERF_BUILDAH_MODE="auto" is enough to enable builds with Buildah, without Docker.
  • native-chroot Buildah mode is the only mode available in containers.
  • Only Linux: native-rootless (oci) mode now used by the default if not in a container (previously native-chroot). For macOS/Windows the same docker-with-fuse mode is used.
  • VFS storage driver available in addition to OverlayFS. Any OS, any mode, containerized or not. To enable: WERF_BUILDAH_STORAGE_DRIVER=vfs
  • Preconfigured images with werf provided to run werf in containers/Kubernetes. Based on Alpine, Ubuntu and CentOS. ghcr.io/werf/werf:1.2-(alpha|beta|ea|stable)-(alpine|ubuntu|centos)
  • If native kernel OverlayFS support is available it is used automatically. If not hen fuse-overlayfs will be used.
  • Now you can run werf in unprivileged containers/Pods.
  • Documentation added to explain how to run werf in Kubernetes/containers and how to integrate it with your CI.

Next

  • 👍1
shurup commented

#4146 sheds some light on why buildah was chosen for werf instead of some other alternatives available (e.g. kaniko).

distorhead commented

Update. In this update we will refresh plans of implementing buildah.

Full support of buildah will be introduced eventually, and includes:

  • Buildah is the main container backend both for CI/CD usage and local development.
  • Buildah backend is crossplatform, on Windows and MacOS platforms it could work through virtualization layer.
  • werf commands support:
    • Delivery commands (main and secondary):
      • werf build
      • werf converge
      • werf bundle publish
      • werf render
      • werf export
      • werf kube-run
    • Cleaning commands:
      • werf cleanup
      • werf purge
      • werf host cleanup
    • Other commands:
      • werf ci-env
      • werf compose
  • Not supported commands:
    • werf run
  • Local development without --repo param is supported.

Current buildah support:

  • Building of Dockerfiles with buildah.
  • Only repo stages storage supported (it means --repo CONTAINER_REGISTRY_REPO param is mandatory to use buildah).
  • Buildah working through docker-server on MacOS and Windows platforms.

Next stage of buildah support will include:

  • Support of building Dockerfiles and stapel images with buildah.
  • Only repo stages storage supported (it means --repo CONTAINER_REGISTRY_REPO param is mandatory to use buildah).
  • Buildah working through docker-server on MacOS and Windows platforms.
  • All main werf commands work with buildah backend.

See progress of this stage in the issue: #3936 (comment).

alexey-igrychev commented

Solved. The user can build images using Stapel and Dockerfile syntax with Docker or Buildah backends 🎉