/CrossValidationMultiLabel

A code to execute and save cross-validation in multilabel classification

Primary LanguageRGNU General Public License v3.0GPL-3.0

X-Folds Cross Validation MultiLabel

A code to execute and save cross-validation in multilabel classification. This code is part of my doctoral research.

How to cite

@misc{Gatto2022, author = {Gatto, E. C.}, title = {Cross-Validation for MultiLabel Classification}, year = {2022}, publisher = {GitHub}, journal = {GitHub repository}, howpublished = {\url{https://github.com/cissagatto/CrossValidationMultiLabel}}}

Scripts

This source code consists of an R project for R Studio and the following R scripts:

  1. libraries.R
  2. utils.R
  3. CrossValidationMultiLabel.R
  4. main.R
  5. cvm.R

Multi-Label Datasets

You can download the multi-label datasets at this link: https://cometa.ujaen.es/datasets/

Preparing your experiment

Step-1

Confirms if the folder utils contains the following files: Clus.jar, R_csv_2_arff.jar, and weka.jar, and also the folder lib with commons-math-1.0.jar, jgap.jar, weka.jar and Clus.jar. Without these jars, the code not runs.

Step-2

Copy this code and place it where you want. The folder configurations is "~/CrossValidationMultiLabel"

Step-3

A file called datasets-original.csv must be in the root project folder. This file is used to read information about the datasets and they are used in the code. We have 90 multilabel datasets in this .csv file. If you want to use another dataset, please, add the following information about the dataset in the file:

Parameter Status Description
Id mandatory Integer number to identify the dataset
Name mandatory Dataset name (please follow the benchmark)
Domain optional Dataset domain
Instances mandatory Total number of dataset instances
Attributes mandatory Total number of dataset attributes
Labels mandatory Total number of labels in the label space
Inputs mandatory Total number of dataset input attributes
Cardinality optional
Density optional
Labelsets optional
Single optional
Max.freq optional
Mean.IR optional
Scumble optional
TCS optional
AttStart mandatory Column number where the attribute space begins*
AttEnd mandatory Column number where the attribute space ends
LabelStart mandatory Column number where the label space begins
LabelEnd mandatory Column number where the label space ends
Distinct optional
xn mandatory Value for Dimension X of the Kohonen map
yn mandatory Value for Dimension Y of the Kohonen map
gridn mandatory X times Y value. Kohonen's map must be square
max.neigbors mandatory The maximum number of neighbors is given by LABELS -1
  • Because it is the first column the number is always 1.

STEP 4

You need to have installed all the R packages required to execute this code on your machine. Check out which are needed in the file libraries.R. This code does not provide any type of automatic package installation! You can use the Conda environment that I created to perform this experiment. Below are the links to download the files.

| download txt | download yml | download yaml |

Run

To run, first enter the folder ~/CrossValidationMultiLabel/R in a terminal and the type:

Rscript cvm.R [number_dataset] [number_cores] [number_folds] [validation] [folder]

Where:

number_dataset is the dataset number in the datasets.csv file

number_cores is the number of cores that you wanto to use in paralel

number_folds is the number of folds you want for cross-validation

validation 0 if you dont want the validation set and 1 if you want

folder temporary folder like SHM or SCRATCH to speed up the process

Example:

  1. With Validation
Rscript cvm.R 2 10 10 1 "/dev/shm/results"

This code will generate a 10 folds cross-valdation, using 10 cores, for the birds (2) dataset with train, test and validation sets. If you send number_folds = 1 the code will break. For now, this code is specifcally to build a X-Fold Cross-Validation file. Then, you need to pass a value greater than 1 to number_folds parametrer. If you want to use holdout, please consulte the UTIML tutorial https://github.com/rivolli/utiml.

  1. Whithout Validation
Rscript cvm.R 2 1 10 0 "/dev/shm/cv"

This code will generate a 10 folds cross-valdation, using 1 core, for the birds (2) dataset with train and test sets.

Folder Structure

DOWNLOAD RESULTS

Click here

Acknowledgment

  • This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
  • This study was financed in part by the Conselho Nacional de Desenvolvimento Científico e Tecnológico - Brasil (CNPQ) - Process number 200371/2022-3.
  • The authors also thank the Brazilian research agencies FAPESP financial support.

Contact

elainececiliagatto@gmail.com

Links

| Site | Post-Graduate Program in Computer Science | Computer Department | Biomal | CNPQ | Ku Leuven | Embarcados | Read Prensa | Linkedin Company | Linkedin Profile | Instagram | Facebook | Twitter | Twitch | Youtube |

Thanks