delta-io/connectors

Flink Delta Connector end-to-end tests

rdziadosz opened this issue · 2 comments

rdziadosz commented

Connector end-to-end tests are developed in order to identify potential bugs automatically at the early development stage.

Implementation plan

In order to make code review easier, the implementation of end-to-end tests should be divided into multiple pull requests. We plan to do it in the following steps (some of them can be done in parallel):

PR-01: Project skeleton.

  • Shows how the project structure looks like.
  • Add sbt modules (tests + fatjar).
  • Add a smoke test (does not schedule any flink job actually).
  • Add terraform skeleton (almost empty).
  • Add run-end-to-end-tests.sh script.

PR-02-A: Terraform.

  • Add terraform creating all necessary infrastructure in AWS.

PR-02-B: Flink REST Client.

  • Java implementation of a client for job scheduling.

PR-03-A: Add bounded reader test scenarios.

  • Add corresponding Flink job.
  • Implement job scenarios. Create multiple pull requests if there are lots of test scenarios.

PR-03-B: Add bounded writer test scenarios.

  • Add corresponding Flink job.
  • Implement job scenarios. Create multiple pull requests if there are lots of test scenarios.

PR-04-A: Add unbounded reader test scenarios.

  • Add corresponding Flink job.
  • Implement job scenarios. Create multiple pull requests if there are lots of test scenarios.

PR-04-B: Add unbounded writer test scenarios.

  • Add corresponding Flink job.
  • Implement job scenarios. Create multiple pull requests if there are lots of test scenarios.

PR-05: Enable to run tests in parallel.

rdziadosz commented

PR-02-A: #445

In order to provide fine-grained permissions, IAM roles for service accounts feature should be used: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
Currently S3A connector is configured to use EnvironmentVariableCredentialsProvider, because WebIdentityTokenCredentialsProvider is supported by aws-java-sdk as of version 1.11.704
Currently com.amazonaws:aws-java-sdk-bundle:1.11.271 is used (hadoop-aws:3.1.0 dependency), which does not support WebIdentityTokenCredentialsProvider.
After upgrading hadoop-aws to a newer version, it is recommended to change to IAM roles for service accounts feature.

vkorukanti commented

This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See #556 for details.