Proposal: first-class support for developing on Docker

[aiven-anton]

I've opened a few draft PRs related to improving the Docker container, and wanted to share where I'm going with this. We decided to put those on hold until we have at least a rudimentary smoke test in place for the container image.

The purpose of this issue to serve as an umbrella for those and further changes, that will facilitate easier set up of development environments, especially for non-linux work stations. The end goal is to be able to easily run the test suite in a docker container, with local changes mounted for quick iterations. The draft PRs that are already lined-up also help in facilitating this by making sure local container rebuilds are quick.

In making these changes, I am also modernizing some of the tooling that we're using where appropriate, like using Docker's Github actions to build container metadata and versioning, using a more recent Python version, etc.

It's important to note that all of these changes are intended to only have minor effects on the resulting production container image.

Items to address

• Introduce smoke-test of container image. Should start up container in both modes (rest/registry) with a Kafka back-end, and verify communications work through some relatively simple operations.
• Build ARM image (fixes #521).
• Optimize container build time.
• #580 Narrow docker context, this assures changes to unrelated files in the repository don't trigger cache eviction of Docker layers.
• Introduce a dev build target, separate from the production build target, that installs full test requirements in the container.
• Make sure there's a docker compose file that uses the new dev target, should enable running tests with something like docker compose run --rm karapace pytest.
• Introduce proxy targets in Makefile, so that we can easily run common make targets within a container. make docker:unit-tests should result in a call to make unit-tests within the docker-compose container.

Why?

• Current setup leaves lingering processes on developer machines.
• No need to install, and keep versions in sync, of dependency binaries on developer machines (java, protoc, snappy, kafka).
• Alignment of developer environments, possibly easier to reason about and recreate issues.
• A streamlined dev env setup hopefully means it's easier to attract contributors.