icelake-io/icelake

Refactor integration tests to use docker compose.

Closed this issue · 0 comments

In our integration tests, we need to setup several docker images for minio, spark, etc to verify result. I previously introduced testcontainers-rs to setup them by code with isolation for different tests. However I realized that there are several problems in it:

  1. Difficult to maintain for complex setup. It's difficult to get the dependencies from codes.
  2. Difficult to debug. It's difficult to control the lifecycle of containers manually, which is painful when there are problems in container settings.

The only advantage is that it provides isolation for different tests, but I found that docker compose also provides isolation so that we can run several projects in parallel, just follow the rules below:

  1. Give each run a project name by -p.
  2. Don't specify container name explicitly. Docker compose give each container name like this: <project name>-<service name>-1.
  3. Don't specify port mapping, but expose ports only.
  4. Don't specify network explicitly. Docker compose will add a network for each run.

An example can be found at https://github.com/icelake-io/icelake/blob/main/icelake/tests/rest_catalog_tests.rs

The only left tests is https://github.com/icelake-io/icelake/blob/6035b7bdc41e04b676bc9017c0e457c536936afd/icelake/tests/insert_tests.rs

We should refactor to use docker compose and remove testcontainers.