Add varchar length metadata to the data returned by the built-in JDBC source

[yruslan]

Background

Relational databases often store text in varchar(n) data type. Spark when reading a JDBC source converts such columns to strings without maximum length. Keeping the original maximum length of fields can be helpful, especially for exposing tables via Hive if the consumer is also a relational database that requires meximum lengths for strings.

Feature

Add varchar length metadata to the data returned by the built-in JDBC source.

Proposed Solution

When a dataframe is requested, the JDBC source can connect to the same DB using java.sql and fetch varchar types.

The configuration key enable.schema.metadata is more generic than just varchar-related for the potential reuse for other metadata.

This adds a delay to the job so the feature should be configurable:

{
    name = "source1_name"
    factory.class = "za.co.absa.pramen.core.source.JdbcSource"

    #...

    enable.schema.metadata = true
}

Caveat

The standard of metadata provisioning is DB provider related and can differ between DBs. For instance, column name is sometimes metadata.getColumnName(i), but sometimes metadata.getColumnLabel(i)