[Flink connector][test][enhancement] - Create TableInfo class for Delta Source and Sing integration tests.
kristoffSC opened this issue · 2 comments
Integration tests like DeltaSinkBatchExecutionITCase, DeltaSinkExecutionITCaseBase, DeltaSinkStreamingExecutionITCase, DeltaSourceBoundedExecutionITCaseTest and DeltaSourceContinuousExecutionITCaseTest are using predefined delta tables placed under test/resources/test-data.
Information about table path, schema, data values and number of rows is scattered between all test classes and helper classec which is problematic to track which leads to code duplication.
The goal of this enhancement would be to create a TableInfo class that contained the path, column names, column types, making all information coupled together. This class could be used for all IT tests where we use predefined.
We could create TableInfo class for every test table. This class will contain all information for this table.
Extra thing:
change partition columns from "col1", "col2" etc. to "part1", "part2" etc. since this is a convention commonly used in delta lake on spark.
Partially implemented by implemented by #547.
What left is:
change partition columns from "col1", "col2" etc. to "part1", "part2" etc. since this is a convention commonly used in delta lake on spark.
This repo has been deprecated and the code is moved under connectors module in https://github.com/delta-io/delta repository. Please create the issue in repository https://github.com/delta-io/delta. See #556 for details.