[BENCHMARK] Compare results to `dl-4-tsc` on the MTS dataset
tallamjr opened this issue · 1 comments
dl-4-tsc by Fawaz et al compares deep learning classifiers on 128 univariate datasets and 12 multivariate datasets.
It would be good in the first instance to be able to compare results produced here to the multivariate case to gauge the performances on simplified multivariate data.
Show below is a snapshot of the results shown in the repository as of hfawaz/dl-4-tsc@3ee62e1
The following table contains the averaged accuracy over 10 runs of each implemented model on the MTS archive, with the
standard deviation between parentheses.
| Datasets | MLP | FCN | ResNet | Encoder | MCNN | t-LeNet | MCDCNN | Time-CNN | TWIESN |
|---|---|---|---|---|---|---|---|---|---|
| AUSLAN | 93.3(0.5) | 97.5(0.4) | 97.4(0.3) | 93.8(0.5) | 1.1(0.0) | 1.1(0.0) | 85.4(2.7) | 72.6(3.5) | 72.4(1.6) |
| ArabicDigits | 96.9(0.2) | 99.4(0.1) | 99.6(0.1) | 98.1(0.1) | 10.0(0.0) | 10.0(0.0) | 95.9(0.2) | 95.8(0.3) | 85.3(1.4) |
| CMUsubject16 | 60.0(16.9) | 100.0(0.0) | 99.7(1.1) | 98.3(2.4) | 53.1(4.4) | 51.0(5.3) | 51.4(5.0) | 97.6(1.7) | 89.3(6.8) |
| CharacterTrajectories | 96.9(0.2) | 99.0(0.1) | 99.0(0.2) | 97.1(0.2) | 5.4(0.8) | 6.7(0.0) | 93.8(1.7) | 96.0(0.8) | 92.0(1.3) |
| ECG | 74.8(16.2) | 87.2(1.2) | 86.7(1.3) | 87.2(0.8) | 67.0(0.0) | 67.0(0.0) | 50.0(17.9) | 84.1(1.7) | 73.7(2.3) |
| JapaneseVowels | 97.6(0.2) | 99.3(0.2) | 99.2(0.3) | 97.6(0.6) | 9.2(2.5) | 23.8(0.0) | 94.4(1.4) | 95.6(1.0) | 96.5(0.7) |
| KickvsPunch | 61.0(12.9) | 54.0(13.5) | 51.0(8.8) | 61.0(9.9) | 54.0(9.7) | 50.0(10.5) | 56.0(8.4) | 62.0(6.3) | 67.0(14.2) |
| Libras | 78.0(1.0) | 96.4(0.7) | 95.4(1.1) | 78.3(0.9) | 6.7(0.0) | 6.7(0.0) | 65.1(3.9) | 63.7(3.3) | 79.4(1.3) |
| NetFlow | 55.0(26.1) | 89.1(0.4) | 62.7(23.4) | 77.7(0.5) | 77.9(0.0) | 72.3(17.6) | 63.0(18.2) | 89.0(0.9) | 94.5(0.4) |
| UWave | 90.1(0.3) | 93.4(0.3) | 92.6(0.4) | 90.8(0.4) | 12.5(0.0) | 12.5(0.0) | 84.5(1.6) | 85.9(0.7) | 75.4(6.3) |
| Wafer | 89.4(0.0) | 98.2(0.5) | 98.9(0.4) | 98.6(0.2) | 89.4(0.0) | 89.4(0.0) | 65.8(38.1) | 94.8(2.1) | 94.9(0.6) |
| WalkvsRun | 70.0(15.8) | 100.0(0.0) | 100.0(0.0) | 100.0(0.0) | 75.0(0.0) | 60.0(24.2) | 45.0(25.8) | 100.0(0.0) | 94.4(9.1) |
| Average_Rank | 5.208333 | 2.000000 | 2.875000 | 3.041667 | 7.583333 | 8.000000 | 6.833333 | 4.625000 | 4.833333 |
| Wins | 0 | 5 | 3 | 0 | 0 | 0 | 0 | 0 | 2 |
The MTS data has been downloaded from: http://www.mustafabaydogan.com/files/viewcategory/20-data-sets.html and the processed using dl-4-tsc/utils/utils.py with this change:
diff --git a/utils/utils.py b/utils/utils.py
index 0ef692b..c0ae7ab 100755
--- a/utils/utils.py
+++ b/utils/utils.py
@@ -219,8 +219,8 @@ def transform_to_same_length(x, n_var, max_length):
def transform_mts_to_ucr_format():
- mts_root_dir = '/mnt/Other/mtsdata/'
- mts_out_dir = '/mnt/nfs/casimir/archives/mts_archive/'
+ mts_root_dir = '/Users/tallamjr/github/tallamjr/origin/astronet/data/mtsdata/'
+ mts_out_dir = '/Users/tallamjr/github/tallamjr/origin/astronet/data/transformed-mtsdata/'
for dataset_name in MTS_DATASET_NAMES:
# print('dataset_name',dataset_name)
Then by running: $ python main.py transform_mts_to_ucr_format
N.B Empty folders with the names of the datasets found in mtsdata needed to be created first in transformerd-mtsdata before running the above command.
With this inclusion, it may be desirable to refactor how one loads data into astronet.t2.train.py and astronet.t2.opt.hypertrain.py as there will be many more to list in an if/else block now, this may be better served in astronet.t2.utils.py perhaps