[FEATURE/BUG] Enable definition of traning and validation set for privacy-preserving machine learning

[prasser]

Is your feature request related to a problem? Please describe.

The privacy-preserving machine learning framwork in ARX uses k-fold cross-validation to quantify the performance of privacy-preserving models. This can lead to misleading estimates as training and validation sets both influence the optimization process performed during anonymization.

Describe the solution you'd like
As an alternative it would be good to enable users to specify a training and a validation set in such a way that only the training set influences the anonymization process. This can easily be done in ARX by using the "research subset" feature, which allows selecting a subset of the records in a dataset that are then anonymized. What would needed to be added is a feature that allows to specify that machine learning performance is determined based on the set of records that is not included in the research subset.

[prasser]

@srcds or @idhamari might want to take a look at this.

[idhamari]

@prasser @srcds sounds interesting, I will have a look at this next week.

[idhamari]

@prasser

When a user select random records by clicking on "Select Randomly", a view output does not reflect that e.g. by selecting 0.80:

    System.out.println("this.model.getOutput().getNumRows()                   : " + this.model.getOutput().getNumRows());
    System.out.println("this.model.getOutput().getView().getNumRows()         : " + this.model.getOutput().getView().getNumRows());
    System.out.println("this.model.getInputConfig().getResearchSubset.size()  : " + this.model.getInputConfig().getResearchSubset().size());  

I get the output:

         this.model.getOutput().getNumRows()                          : 30162
         this.model.getOutput().getView().getNumRows()          : 30162
         this.model.getInputConfig().getResearchSubset.size()  : 24070

I am using the view output to get the training and the testing subset which makes a problem.

[prasser]

This is OK and the expected behaviour.

[prasser]

An implementation of this is provided in the following branch: feature-training-test

Might still need a little bit of polishing, though.

[jenno-verdonck]

I was wondering if this feature can already be used it it's current state?

[prasser]

Yes, should work. We would be happy to receive feedback.

[prasser]

To be clear. The feature lives here in this branch:

https://github.com/arx-deidentifier/arx/tree/feature-training-test