Author:
Toni Heittola, Tampere University Email, Homepage, GitHub
- Clone repository from Github.
- Install requirements with command:
pip install -r requirements.txt. - Run the subtask specific applications with default settings:
- Subtask A:
python task1a.pyor./task1a.py - Subtask B:
python task1b.pyor./task1b.py
- Subtask A:
To setup Anaconda environment for the system use following:
conda create --name tf-dcase
conda activate tf-dcase
conda install ipython
conda install numpy
conda install tensorflow-gpu==1.14.0
conda install -c anaconda keras-gpu=2.2.4
conda install -c anaconda cudatoolkit
conda install -c anaconda cudnn
conda install librosa
pip install absl-py==0.9.0
pip install dcase_util==0.2.11
pip install sed_eval
This is the baseline system for the Acoustic scene classification task (Task 1) in Detection and Classification of Acoustic Scenes and Events 2020 (DCASE2020) challenge. The system is intended to provide a simple entry-level state-of-the-art approach that gives reasonable results in the subtasks of Task 1. The baseline system is built on dcase_util toolbox (>=version 0.2.11). In order to use system with evaluation dataset, make sure you are using dcase_util version >= 0.2.12.
Participants can build their own systems by extending the provided baseline system. The system has all needed functionality for the dataset handling, acoustic feature storing and accessing, acoustic model training and storing, and evaluation. The modular structure of the system enables participants to modify the system to their needs. The baseline system is a good starting point especially for the entry level researchers to familiarize themselves with the acoustic scene classification problem.
If participants plan to publish their code to the DCASE community after the challenge, building their approach on the baseline system could potentially make their code more accessible to the community. DCASE organizers strongly encourage participants to share their code in any form after the challenge.
TAU Urban Acoustic Scenes 2020 Mobile Development dataset is used as development dataset for this task.
This subtask is concerned with the basic problem of acoustic scene classification, in which it is required to classify a test audio recording into one of ten known acoustic scene classes. This task targets generalization properties of systems across a number of different devices, and will use audio data recorded and simulated with a variety of devices. Recordings in the dataset were made with three devices (A, B and C) that captured audio simultaneously and 6 simulated devices (S1-S6). Each acoustic scene has 1440 segments (240 minutes of audio) recorded with device A (main device) and 108 segments of parallel audio (18 minutes) each recorded with devices B,C, and S1-S6. The dataset contains in total 64 hours of audio. For a more detailed description see DCASE Challenge task description.
The subtask specific baseline system is implemented in file task1a.py.
The system is a modification of the baselines from previous DCASE challenge editions of acoustic scene classification, however, it replaces use of log mel energies with use of OpenL3 embeddings and replaces the CNN network architecture with two fully-connected feed-forward layers (size 512 and 128) as in the original OpenL3 publication:
J. Cramer, H-.H. Wu, J. Salamon, and J. P. Bello. Look, listen and learn more: design choices for deep audio embeddings. In IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 3852–3856. Brighton, UK, May 2019.
- OpenL3 embedding
- Analysis window size: 1 second, Hop size: 100 ms
- Embeddings parameters:
- Input representation: mel256
- Content type: music
- Embedding size: 512
- Architecture: Two fully connected layers (512 and 128 hidden units)
- Learning: 200 epochs (batch size 64), data shuffling between epochs
- Optimizer: Adam (learning rate 0.001)
- Model selection:
- Approximately 30% of the original training data is assigned to validation set, split done such that training and validation sets do not have segments from the same location and both sets have data from each city
- Model performance after each epoch is evaluated on the validation set, and best performing model is selected
The cross-validation setup provided with the TAU Urban Acoustic Scenes 2020 Mobile Development dataset is used to evaluate the performance of the baseline system. Results are calculated using TensorFlow in GPU mode (using Nvidia Titan XP GPU card). Because results produced with GPU card are generally non-deterministic, the system was trained and tested 10 times, and mean and standard deviation of the performance from these 10 independent trials are shown in the results tables.
| Scene class | Accuracy | Log loss |
|---|---|---|
| Airport | 43.6 % | 1.693 |
| Bus | 61.5 % | 1.019 |
| Metro | 54.6 % | 1.263 |
| Metro station | 54.3 % | 1.227 |
| Park | 71.0 % | 1.026 |
| Public square | 43.0 % | 1.658 |
| Shopping mall | 56.9 % | 1.203 |
| Street, pedestrian | 28.8 % | 2.335 |
| Street, traffic | 78.6 % | 0.809 |
| Tram | 53.6 % | 1.326 |
| Average | 54.6 % (+/- 1.5) | 1.356 (+/- 0.055) |
Note: The reported system performance is not exactly reproducible due to varying setups. However, you should be able obtain very similar results.
TAU Urban Acoustic Scenes 2020 3Class Development dataset is used as development dataset for this task.
This subtask is concerned with classification of audio into three major classes (indoor, outdoor and transportation). The task targets low complexity solutions for the classification problem in term of model size, and uses audio recorded with a single device (device A, 48 kHz / 24bit / stereo). The data for the dataset was recorded in 10 acoustic scenes which were later grouped into three major classes used in this subtask. The dataset contains in total 40 hours of audio. For a more detailed description see DCASE Challenge task description.
Classifier complexity for this subtask is limited to 500KB size for the non-zero parameters. This translates into 128K parameters when using float32 (32-bit float) which is often the default data type (128000 parameter values * 32 bits per parameter / 8 bits per byte= 512000 bytes = 500KB). See detailed description how to calculate model size from DCASE Challenge task description. Model calculation for Keras models is implemented in model_size_calculation.py
The subtask specific baseline system is implemented in file task1b.py.
The system implements a convolutional neural network (CNN) based approach, where log mel-band energies are first extracted for each 10-second signal, and a network consisting of two CNN layers and one fully connected layer is trained to assign scene labels to the audio signals. Model size of the baseline is 450.1 KB.
- Analysis frame 40 ms (50% hop size)
- Log mel-band energies (40 bands)
- Input shape: 40 * 500 (10 seconds)
- Architecture:
- CNN layer #1
- 2D Convolutional layer (filters: 32, kernel size: 7) + Batch normalization + ReLu activation
- 2D max pooling (pool size: (5, 5)) + Dropout (rate: 30%)
- CNN layer #2
- 2D Convolutional layer (filters: 64, kernel size: 7) + Batch normalization + ReLu activation
- 2D max pooling (pool size: (4, 100)) + Dropout (rate: 30%)
- Flatten
- Dense layer #1
- Dense layer (units: 100, activation: ReLu )
- Dropout (rate: 30%)
- Output layer (activation: softmax/sigmoid)
- CNN layer #1
- Learning (epochs: 200, batch size: 16, data shuffling between epochs)
- Optimizer: Adam (learning rate: 0.001)
- Model selection:
- Approximately 30% of the original training data is assigned to validation set, split done so that training and validation sets do not have segments from same location and so that both sets have similar amount of data per city
- Model performance after each epoch is evaluated on the validation set, and best performing model is selected
Network summary
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d_7 (Conv2D) (None, 40, 500, 32) 1600
_________________________________________________________________
batch_normalization_7 (Batch (None, 40, 500, 32) 160
_________________________________________________________________
activation_7 (Activation) (None, 40, 500, 32) 0
_________________________________________________________________
max_pooling2d_7 (MaxPooling2 (None, 8, 100, 32) 0
_________________________________________________________________
dropout_10 (Dropout) (None, 8, 100, 32) 0
_________________________________________________________________
conv2d_8 (Conv2D) (None, 8, 100, 64) 100416
_________________________________________________________________
batch_normalization_8 (Batch (None, 8, 100, 64) 32
_________________________________________________________________
activation_8 (Activation) (None, 8, 100, 64) 0
_________________________________________________________________
max_pooling2d_8 (MaxPooling2 (None, 2, 1, 64) 0
_________________________________________________________________
dropout_11 (Dropout) (None, 2, 1, 64) 0
_________________________________________________________________
flatten_4 (Flatten) (None, 128) 0
_________________________________________________________________
dense_7 (Dense) (None, 100) 12900
_________________________________________________________________
dropout_12 (Dropout) (None, 100) 0
_________________________________________________________________
dense_8 (Dense) (None, 10) 1010
=================================================================
Total params: 116,118
Trainable params: 116,022
Non-trainable params: 96
_________________________________________________________________
Input shape : (None, 40, 500, 1)
Output shape : (None, 10)
The cross-validation setup provided with the TAU Urban Acoustic Scenes 2020 3Class Development dataset is used to evaluate the performance of the baseline system. Results are calculated the same way as for subtask A, with mean and standard deviation of the performance from 10 independent trials show in the results table.
| Scene class | Accuracy | Log loss |
|---|---|---|
| Indoor | 82.0 % | 0.680 |
| Outdoor | 88.5 % | 0.365 |
| Transportation | 91.5 % | 0.282 |
| Average | 87.3 % (+/- 0.7) | 0.437 (+/- 0.045) |
Note: The reported system performance is not exactly reproducible due to varying setups. However, you should be able obtain very similar results.
For each subtask there is a separate application (.py file):
task1a.py, DCASE2020 baseline for Task 1A, Acoustic Scene Classification with Multiple Devicestask1b.py, DCASE2020 baseline for Task 1B, Low-Complexity Acoustic Scene Classification
All the usage arguments are shown by python task1a.py -h.
| Argument | Description | |
|---|---|---|
-h |
--help |
Application help. |
-v |
--version |
Show application version. |
-m {dev,eval} |
--mode {dev,eval} |
Selector for application operation mode |
-s PARAMETER_SET |
--parameter_set PARAMETER_SET |
Parameter set id. Can be also comma separated list e.g. `-s set1,set2,set3``. In this case, each set is run separately. |
-p FILE |
--param_file FILE |
Parameter file (YAML) to overwrite the default parameters |
-o OUTPUT_FILE |
--output OUTPUT_FILE |
Output file |
--overwrite |
Force overwrite mode. | |
--download_dataset DATASET_PATH |
Download dataset to given path and exit | |
--show_parameters |
Show active application parameter set | |
--show_sets |
List of available parameter sets | |
--show_results |
Show results of the evaluated system setups |
The system can be used in three different operation modes.
Development mode - dev
In development mode, the development dataset is used with the provided cross-validation setup: training set is used for learning, and testing set is used for evaluating the performance of the system. This is the default operation mode.
Usage example: python task1a.py or python task1a.py -m dev
Challenge mode - eval
Note: This operation mode does not work yet as the evaluation dataset has not been published.
In challenge mode, the full development dataset (including training and test subsets) is used for learning, and a second dataset, evaluation dataset, is used for testing. The system system outputs are generated based on the evaluation dataset. If ground truth is available for the evaluation dataset, the output is also evaluated. This mode is designed to be used for generating the DCASE challenge submission, running the system on the evaluation dataset for generating the system outputs for the submission file.
Usage example: python task1a.py -m eval and python task1b.py -m eval
To save system output to a file: python task1a.py -m eval -o output.csv
The baseline system supports multi-level parameter overwriting, to enable flexible switching between different system setups. Parameter changes are tracked with hashes calculated from parameter sections. These parameter hashes are used in the storage file paths when saving data (features/embeddings, model, or results). By using this approach, the system will compute features/embeddings, models and results only once for the specific parameter set, and after that it will reuse this precomputed data.
Parameters are stored in YAML-formatted files, which are handled internally in the system as Dict like objects (dcase_util.containers.DCASEAppParameterContainer). Default parameters is the set of all possible parameters recognized by the system. Parameter set is a smaller set of parameters used to overwrite values of the default parameters. This can be used to select methods for processing, or tune parameters.
Parameters files are YAML-formatted files, containing the following three blocks:
active_set, default parameter set idsets, list of dictionariesdefaults, dictionary containing default parameters which are overwritten by thesets[active_set]
At the top level of the parameter dictionary there are parameter sections; depending on the name of the section, the parameters inside it are processed sometimes differently. Usually there is a main section (feature_extractor, and method parameter section (feature_extractor_method_parameters) which contains parameters for each possible method. When parameters are processed, the correct method parameters are copied from method parameter section to the main section under parameters. This allows having many methods ready parametrized and easily accessible.
Parameter hashes are MD5 hashes calculated for each parameter section. In order to make these hashes more robust, some pre-processing is applied before hash calculation:
- If section contains field
enablewith valueFalse, all fields inside this section are excluded from the parameter hash calculation. This will avoid recalculating the hash if the section is not used but some of these unused parameters are changed. - If section contains fields with value
False, these fields are excluded from the parameter hash calculation. This will enable to add new flag parameters without changing the hash. Define the new flag such that the previous behaviour is happening when this field is set to false. - All
non_hashable_fieldsfields are excluded from the parameter hash calculation. These fields are set whendcase_util.containers.AppParameterContaineris constructed, and they usually are fields used to print various values to the console. These fields do not change the system output to be saved onto disk, and hence they are excluded from hash.
Easiest way to extend the baseline system is by modifying system parameters. To do so one needs to create a parameter file with a custom parameter set, and run system with this parameter file.
Example 1
In this example, one creates MLP based system. Data processing chain is replaced with a chain which calculated mean over 500 feature vectors. Learner is replaced with a new model definition. Parameter file extra.yaml:
active_set: minimal-mlp
sets:
- set_id: minimal-mlp
description: Minimal MLP system
data_processing_chain:
method: mean_aggregation_chain
data_processing_chain_method_parameters:
mean_aggregation_chain:
chain:
- processor_name: dcase_util.processors.FeatureReadingProcessor
- processor_name: dcase_util.processors.NormalizationProcessor
init_parameters:
enable: true
- processor_name: dcase_util.processors.AggregationProcessor
init_parameters:
aggregation_recipe:
- mean
win_length_frames: 500
hop_length_frames: 500
- processor_name: dcase_util.processors.DataShapingProcessor
init_parameters:
axis_list:
- time_axis
- data_axis
learner:
method: mlp_mini
learner_method_parameters:
mlp_mini:
random_seed: 0
keras_profile: deterministic
backend: tensorflow
validation_set:
validation_amount: 0.20
balancing_mode: identifier_two_level_hierarchy
seed: 0
data:
data_format: channels_last
target_format: same
model:
config:
- class_name: Dense
config:
units: 50
kernel_initializer: uniform
activation: relu
input_shape:
- FEATURE_VECTOR_LENGTH
- class_name: Dropout
config:
rate: 0.2
- class_name: Dense
config:
units: 50
kernel_initializer: uniform
activation: relu
- class_name: Dropout
config:
rate: 0.2
- class_name: Dense
config:
units: CLASS_COUNT
kernel_initializer: uniform
activation: softmax
compile:
loss: categorical_crossentropy
metrics:
- categorical_accuracy
optimizer:
class_name: Adam
fit:
epochs: 50
batch_size: 64
shuffle: true
callbacks:
StasherCallback:
monitor: val_categorical_accuracy
initial_delay: 25
Command to run the system:
python task1a.py -p extra.yaml
Example 2
In this example, one slightly modifies the baseline for subtask B system to have smaller network. Learner is replaced with modified model definition. Since cnn learner method is overloaded, only a subset of the parameters needs to be defined. However, the model config (network definition) has to be redefined fully as list parameters cannot be overloaded partly. Parameter file extra.yaml:
active_set: baseline-minified
sets:
- set_id: baseline-minified
description: Minified DCASE2020 baseline subtask B minified
learner_method_parameters:
cnn:
model:
constants:
CONVOLUTION_KERNEL_SIZE: 3
config:
- class_name: Conv2D
config:
input_shape:
- FEATURE_VECTOR_LENGTH # data_axis
- INPUT_SEQUENCE_LENGTH # time_axis
- 1 # sequence_axis
filters: 8
kernel_size: CONVOLUTION_KERNEL_SIZE
padding: CONVOLUTION_BORDER_MODE
kernel_initializer: CONVOLUTION_INIT
data_format: DATA_FORMAT
- class_name: Activation
config:
activation: CONVOLUTION_ACTIVATION
- class_name: MaxPooling2D
config:
pool_size:
- 5
- 5
data_format: DATA_FORMAT
- class_name: Conv2D
config:
filters: 16
kernel_size: CONVOLUTION_KERNEL_SIZE
padding: CONVOLUTION_BORDER_MODE
kernel_initializer: CONVOLUTION_INIT
data_format: DATA_FORMAT
- class_name: Activation
config:
activation: CONVOLUTION_ACTIVATION
- class_name: MaxPooling2D
config:
pool_size:
- 4
- 100
data_format: DATA_FORMAT
- class_name: Flatten
- class_name: Dense
config:
units: 100
kernel_initializer: uniform
activation: relu
- class_name: Dense
config:
units: CLASS_COUNT
kernel_initializer: uniform
activation: softmax
fit:
epochs: 100
Command to run the system:
python task1b.py -p extra.yaml
Example 3
In this example, multiple different setups are run in a sequence. Parameter file extra.yaml:
active_set: baseline-kernel3
sets:
- set_id: baseline-kernel3
description: DCASE2020 baseline for subtask B with kernel 3
learner_method_parameters:
cnn:
model:
constants:
CONVOLUTION_KERNEL_SIZE: 3
fit:
epochs: 100
- set_id: baseline-kernel5
description: DCASE2020 baseline for subtask B with kernel 5
learner_method_parameters:
cnn:
model:
constants:
CONVOLUTION_KERNEL_SIZE: 5
fit:
epochs: 100
Command to run the system:
python task1b.py -p extra.yaml -s baseline-kernel3,baseline-kernel5
To see results:
python task1b.py --show_results
The code is built on dcase_util toolbox, see manual for tutorials. The machine learning part of the code in built on Keras (v2.2.2), using TensorFlow (v1.14.0) as backend.
.
├── task1a.py # Baseline system for subtask A
├── task1a.yaml # Configuration file for task1a.py
├── task1b.py # Baseline system for subtask B
├── task1b.yaml # Configuration file for task1b.py
|
├── extra.yaml # Example file to show how to extend the system through configuration
├── extra.yaml # Example file to show how to run multiple configurations
|
├── model_size_calculation.py # Utility function for calculating model size for subtask B
├── utils.py # Common functions shared between tasks
|
├── README.md # This file
└── requirements.txt # External module dependencies
- First public release
This software is released under the terms of the MIT License.