ctlearn-project/ctlearn

Stuck running multiple configuration file

Closed this issue · 3 comments

Jsevillamol commented

Anybody know what causes this error in the current master commit?

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/jsevilla/miniconda3/envs/ctlearn/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/jsevilla/miniconda3/envs/ctlearn/lib/python3.6/site-packages/ctlearn/run_model.py", line 141, in run_model
    **data_processing_settings)
TypeError: __init__() got an unexpected keyword argument 'sort_images_by'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "scripts/run_multiple_configurations.py", line 213, in <module>
    log_to_file=args.log_to_file))
  File "/home/jsevilla/miniconda3/envs/ctlearn/lib/python3.6/multiprocessing/pool.py", line 259, in apply
    return self.apply_async(func, args, kwds).get()
  File "/home/jsevilla/miniconda3/envs/ctlearn/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
TypeError: __init__() got an unexpected keyword argument 'sort_images_by'

I was using this multiple configuration file:

Logging:
    model_directory: '/home/jsevilla/output/logs'

Data:
    format: 'HDF5'
    file_list: '/data2/deeplearning/ctlearn/tests/prototype_files_class_balanced.txt'
    Loading:
        min_num_tels: 1
        validation_split: 0.1
        seed: 1234
        use_peak_times: True
    Input:
        prefetch: true
        prefetch_buffer_size: 1
        map: true
        num_parallel_calls: 2
        shuffle: true
        shuffle_buffer_size: 10000
Image Mapping:
    hex_conversion_algorithm: 'oversampling'
    padding:
        'LST': 2
        'MSTF': 1
        'MSTN': 2
        'MSTS': 4
        'SST1': 1
        'SSTA': 0
        'SSTC': 0
        'VTS': 1
Model:
    Model Parameters:
        single_tel:
            # Required dict with keys 'module' and 'function' & string values.
            # Module and function for image classification network.
            # Valid options:
            #   - {module: 'basic', function: 'conv_block'}
            network: {module: 'basic', function: 'conv_block'}

            # Required string or null.
            # Path to a checkpoint file or model directory from which to load
            # pretrained network weights. If null, don't load any weights.
            pretrained_weights: null

        cnn_rnn:
            # Required dict with keys 'module' and 'function' & string values.
            # Module and function for single telescope CNN.
            # Valid options:
            #   - {module: 'basic', function: 'conv_block'}
            cnn_block: {module: 'basic', function: 'conv_block'}
            
            # Required string or null.
            # Path to a checkpoint file or model directory from which to load
            # pretrained CNN block weights. If null, don't load any weights.
            pretrained_weights: null
            
            # Optional float. Default: 0.5
            # Dropout rate of dropout layers. 
            dropout_rate: 0.5

        basic:
            conv_block:
                
                # Required dictionary with keys 'size' and 'strides' and
                # integer values, or null.
                # Max pool size and strides. If null, don't perform any pooling.
                max_pool: {size: 2, strides: 2}
                
                # Required integer or null.
                # Number of output filters of a final 1x1 convolutional layer.
                # If null, don't include this bottleneck layer.
                bottleneck: null
                
                # Optional Boolean. Default: false
                # Whether to include a batch normalization layer after each
                # convolutional layer. Exercise caution when using with
                # array-level models.
                batchnorm: false

Training:
    num_validations: 15
    num_training_steps_per_validation: 2500
    Hyperparameters:
        optimizer: 'Adam'
        adam_epsilon: 1.0e-8
        scale_learning_rate: false
        apply_class_weights: true
        variables_to_train: null

Prediction:
    true_labels_given: true
    export_as_file: true
    prediction_file_path: '/home/jsevilla/output/predictions/predictions.csv'

TensorFlow:
    # Optional Boolean. Default: false
    # Whether to run TensorFlow debugger.
    run_TFDBG: false

Multiple Configurations Settings:
    # Required string.
    # Path to file to save configuration combination for each run for reference.
    run_combinations_path: '/home/jsevilla/output/run_combinations.yml'
    num_grouped_range_values: 15

Multiple Configurations Values:
    example_type:
        config: ['Data', 'Loading', 'example_type']
        value_type: 'grouped'
        values:
            single_tel: 'single_tel'
            cnn_rnn: 'array'
    sorting:
        config: ['Data', 'Processing', 'sort_images_by']
        value_type: 'grouped'
        values:
            single_tel: null
            cnn_rnn: 'size'
    model:
        config: ['Model', 'model']
        value_type: 'grouped'
        values:
            single_tel:
                module: 'single_tel'
                function: 'single_tel_model'
            cnn_rnn:
                module: 'cnn_rnn'
                function: 'cnn_rnn_model'
    learning_rate:
        config: ['Training', 'Hyperparameters', 'base_learning_rate']
        value_type: 'grouped'
        values:
            single_tel: 0.00005
            cnn_rnn: 0.0001
    batch_size:
        config: ['Data', 'Input', 'batch_size']
        value_type: 'grouped'
        values:
            single_tel: 64
            cnn_rnn: 16
    tel_type:
        config: ['Data', 'Loading', 'selected_tel_type']
        value_type: 'grouped'
        values:
            LST: 'LST'
            MSTF: 'MSTF'
            MSTN: 'MSTN'
            MSTS: 'MSTS'
            SST1: 'SST1'
            SSTA: 'SSTA'
            SSTC: 'SSTC'
    layers:
        config: ['Model', 'Model Parameters', 'basic', 'conv_block', 'layers']
        value_type: 'grouped'
        values:
            LST:
                - {filters: 32, kernel_size: 3}
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            MSTF:
                - {filters: 32, kernel_size: 3}
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            MSTN:
                - {filters: 32, kernel_size: 3}
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            MSTS:
                - {filters: 32, kernel_size: 3}
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            SST1:
                - {filters: 32, kernel_size: 3}
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            SSTA:
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
            SSTC:
                - {filters: 32, kernel_size: 3}
                - {filters: 64, kernel_size: 3}
                - {filters: 128, kernel_size: 3}
bryankim96 commented
  • 👍1
aribrill commented

Yes, sort_images_by was renamed to sorting among other changes in the Data:Loading section to handle multiple telescopes. See the latest version of the example config file.

  • 👍1
Jsevillamol commented

The multiple configuration section of the example_config file is outdated, I have opened #80 to reflect that.

Also the cnn_rnn model does not work, I have opened #81 to keep track of that.