AbsaOSS/atum

Enable user to specify measurement controlType

Closed this issue · 2 comments

Enable user to chose which aggregate (AggregatedTotal, AbsAggregatedTotal or HashCrc32) is Atum supposed to use for a given input column, this is for method "withAggregateColumns".

Use case: In a scenario where input for aggregate column generates controlType.HashCrc32 whereas requirement to match with metrics received from control file from source , user needs to choose which aggregate Atum is supposed use for the aggregate column.

Hi @MatloaItumeleng, thanks for submitting this idea. I understand the current behavior of the ControlMeasureBuilder does not suit you (in essence, now: for numeric types controlType.absAggregatedTotal;for non-numeric controlType.HashCrc32 is chosen.)

So what granularity of the setup do you need? I understand that you want to choose the controlType, but is it sufficient to specify a common controlType for all columns in aggregateColumns to create the control counts? (Here, there could be a slight problem if numeric aggregation would be set for non-numeric columns, too.)

Or do you imagine that there will need to be an option to specify controlType set for every column in aggregateColumns?

// Edit: perhaps check the outline in commit f526eec - just what it might look like on the API, its just an API change/suggestion, but without implementation. Or do we want to go deeper and design a more convoluted class for the Specific case (where each aggregateColumn has it's defined controlType?)

Hi @dk1844, for this use case one column is currently being passed in the aggrigateColumns where I'd like to select one controlType (for this instance AggregateTotal ) which will be used when I pass it in ControlMeasureBuilder.forDF(df).withAggregateColumns