Apache Flink Docker Images

This repo contains Dockerfiles for building Docker images for Apache Flink, and are used to build the "official" flink images hosted on Docker Hub (reviewed and build by Docker), as well as the images published on apache/flink DockerHub (maintained by Flink committers).

These Dockerfiles are maintained by the Apache Flink community, but the Docker community is responsible for building and hosting the images on Docker Hub.

Flink Docker image lifecycle

For more information about how changes in this repo are reflected on Docker Hub, see the "An image's source changed in Git, now what?" FAQ entry
For outstanding changes to the Apache Flink images on Docker Hub, see PRs with the "library/flink" label on the official-images repository
For the "source of truth" for which Dockerfile and revision is reflected in the Apache Flink images on Docker Hub, see the library/flink file in the official-images repository.

Development workflow

The master branch of this repository serves as a pure publishing area for releases.

Development happens on the various dev-X branches.

Pull requests for a specific version should be opened against the respective dev-<version> branch. Pull requests for all versions, or for the next major Flink release, should be opened against the dev-master branch.

For new major Flink releases, once the release branch is created in the apache/flink repo, the corresponding dev-x.y branch must be created in apache/flink-docker:

Create the branch based on dev-master
Update .github/workflows/ci.yml in the corresponding dev-* branches: Make sure that the correct snapshot version is stated; for dev-x.y it should point to x.y-SNAPSHOT, while for dev-master it should point to the most recent snapshot version (i.e. x.(y+1)-SNAPSHOT).
Update .github/workflows/snapshot.yml in master mapping each (supported) version to the right branch.

CI

The dev-master branch is tested against nightly Flink snapshots for the next major Flink version. This allows us to develop features in tandem with Flink.

The dev-1.x branches are tested against the latest corresponding minor Flink release, to ensure any changes we make are compatible with the currently used Flink version.

Workflow for new Flink releases

Notes for new Flink major (x.y.0) releases

There are additional steps required when a new Flink major version (x.y.0) is released.

Since only the current and previous major versions of Flink are supported, the Dockerfiles for older versions must be removed when adding the new version to this repo

Release workflow

When a new release of Flink is available, the Dockerfiles in the master branch should be updated and a new manifest sent to the Docker Library official-images repo.

The Dockerfiles are generated on the respective dev-<version> branches, and copied over to the master branch for publishing.

Updating the Dockerfiles involves the following steps:

Generate the Dockerfiles
- Checkout the dev-x.y(minor release)/dev-master(major release) branch of the respective release, e.g., dev-1.9
- Update add-version.sh with the GPG key ID of the key used to sign the new release
  - Be sure to use the full fingerprint of the GPG key (as shown in the example below), as the official images require this.
  - Commit this change with message Add GPG key for x.y.z release ^[example]
- Create a pull request against the dev-x.y/dev-master branch containing this commit.
- Run add-version.sh with the appropriate arguments (-r flink-major-version -f flink-full-version)
  - e.g. ./add-version.sh -r 1.2 -f 1.2.1
Update Dockerfiles on the the master branch
- Remove any existing Dockerfiles from the same major version
  - e.g. rm -r 1.2, if the new Flink version is 1.2.1
- Remove any not-supported versions (i.e. versions that are not meant to be released anymore)
- Copy the generated Dockerfiles from the dev-x.y/dev-master branch to master
- Commit the changes with message Update Dockerfiles for x.y.z release ^[example]
- Create a pull request against the master branch containing this commit.

Once the pull request has been merged, we can release the new docker images:

For publishing to DockerHub: apache/flink , you need to perform the following steps:

Make sure that you are authenticated with your Docker ID, and that your Docker ID has access to apache/flink: docker login -u <username>
1. If you do not have access, you should seek help via the mailing list. We have a limited number of seats which are full, see INFRA-23623 for more information. See also INFRA-21276.
Generate and upload the new images: ./publish-to-dockerhub.sh. (Do not execute on the arm platform machine, such as Apple Silicon)

For publishing as an official image, a new manifest should be generated and a pull request opened on the Docker Library official-images repo.

Run ./generate-stackbrew-library.sh to output the new manifest (see note below regarding usage of this script)
In a clone of the official-images repo, overwrite the file library/flink with the new manifest
Commit this change with message Update to Flink x.y.z ^[example]

A pull request can then be opened on the official-images repo with the new manifest. ^[example]

Once the pull request has been merged (often within 1 business day), the new images will be available shortly thereafter.

Release checklist

Checklist for the dev branch:

The GPG key ID of the key used to sign the release has been added to add-version.sh and committed with the message Add GPG key for x.y.z release
./add-version.sh -r x.y -f x.y.z has been run on the respective dev branch

Checklist for the master branch:

.github/workflows/snapshot.yml in master has each supported version being mapped to the right branch. Deprecated versions should have been removed.
(new major releases only) any unsupported Flink major version Dockerfiles have been removed (only two x.y/ directories should be present)
(new minor releases only) any existing generated files for the same major version have been removed
The updated Dockerfiles have been committed with the message Update Dockerfiles for x.y.z release
A pull request with the above changes has been opened on this repo and merged
The new library manifest has been generated with generate-stackbrew-library.sh and a pull request opened on the official-images repo with commit message Update to Flink x.y.z

Stackbrew Manifest

generate-stackbrew-library.sh is used to generate the library manifest file required for official Docker Hub images.

When the Dockerfiles in this repo are updated, the output of this script should replace the contents of library/flink in the Docker official-images repo via a pull request.

Note: Since this script requires the bashbrew binary and a compatible version of Bash, the script generate-stackbrew-library-docker.sh can be executed on developer's local machine and it will invoke the script in a Docker container, which will be pulled from the internet automatically, with the necessary dependencies.

Example(official-images repo has been cloned locally):

./generate-stackbrew-library-docker.sh > /path/to/official-images/library/flink

License

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

Apache Flink, Flink®, Apache®, the squirrel logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation.

morazow/flink-docker