How to get the most inportant data?

[Steviey]

Ubuntu LTS 16.4
R 4.x

---

How to obtain the most important data from get_best_models()?

All I get is a horrible nested list- presumably without any usable data.

There is nothing about it in the documantation, except ploting.

[issue-label-bot]

Issue-Label Bot is automatically applying the label question ❓ to this issue, with a confidence of 0.92. Please mark this comment with 👍 or 👎 to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[turgut090]

Hi, could you explain what the most important data means? get_best_models() just provides you with a list of Keras models after search.

[Steviey]

I'm searching for the best hyperparameters. How can I get them in a human friendly way?

[turgut090]

In fact, you could get them manually like this:

library(magrittr)
x_data <- matrix(data = runif(500,0,1),nrow = 50,ncol = 5)
y_data <-  ifelse(runif(50,0,1) > 0.6, 1L,0L) %>% as.matrix()

x_data2 <- matrix(data = runif(500,0,1),nrow = 50,ncol = 5)
y_data2 <-  ifelse(runif(50,0,1) > 0.6, 1L,0L) %>% as.matrix()


library(keras)
library(tensorflow)
library(kerastuneR)

build_model = function(hp) {
  
  model = keras_model_sequential()
  model %>% layer_dense(units = hp$Int('units',
                                       min_value = 32,
                                       max_value = 512,
                                       step=  32),input_shape = ncol(x_data),
                        activation =  'relu') %>%
    layer_dense(units = 1, activation = 'softmax') %>%
    compile(
      optimizer = tf$keras$optimizers$Adam(
        hp$Choice('learning_rate',
                  values=c(1e-2, 1e-3, 1e-4))),
      loss = 'binary_crossentropy',
      metrics = 'accuracy')
  return(model)
}

reticulate::py_config()
tuner = RandomSearch(
  build_model,
  objective = 'val_accuracy',
  max_trials = 2,
  executions_per_trial = 2,
  directory = 'my_dir',
  project_name = 'helloworld')


tuner %>% fit_tuner(x_data,y_data,
                    epochs = 2, 
                    validation_data = list(x_data2,y_data2))

hyp = tuner$get_best_hyperparameters(1L)[[1]]
hyp$get('units')
hyp$get('learning_rate')

However, we have a function plot_tuner(). This will bring back the visualized tuner and a data frame of the results.
try this, please: https://github.com/henry090/kerastuneR#plot-results

p = tuner %>% plot_tuner()
p[[2]]

[Steviey]

Actually I don't want to plot anything. As I understand with

results_summary(tuner = tuner, num_trials=5)

I get the best params too. Am I right? And what does "score" mean in this scenario?

[Results summary]
 |-Results in /home/rstud/helloworld
 |-Showing 5 best trials
 |-Objective(name='val_loss', direction='min')
[Trial summary]
 |-Trial ID: 2d09807abd96e3d3a57968fab1521015
 |-Score: 1.580265924334526
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.01
[Trial summary]
 |-Trial ID: b48d804c350c3a649e80a9c2f31e9708
 |-Score: 16.740461111068726
 |-Best step: 0
 > Hyperparameters:
 |-learning_rate: 0.0001
NULL

[turgut090]

You get the results but you cannot save in a tidy form (table). The score shows the metrics results which one defines during model compiling (based on my example).

[Steviey]

Aha ok, yes I see. To be 100% clear... In my regression-example I'm using the following:

build_model = function(hp) {
  
                        model<- keras_model_sequential() %>% 
                            layer_simple_rnn(units=parameters$unitsOne, input_shape = c(seqLen,myFeatures), activation=parameters$layer_simple_rnn_activation) %>% 
                            
                            layer_batch_normalization() %>% 
                            layer_dropout(rate=parameters$dropOutRateOne) %>% #gut gegen overfitting
                            layer_dense(units = parameters$unitsTwo, activation=parameters$layer_dense_activation,bias_regularizer=regularizer_l2(parameters$l2reg_2)) %>%
                            
                            layer_batch_normalization() %>% 
                            layer_dropout(rate=parameters$dropOutRateTwo) %>% #gut gegen overfitting
                            layer_dense(units = parameters$unitsThree, activation=parameters$layer_dense_activation,bias_regularizer=regularizer_l2(parameters$l2reg_3)) %>%
                            
                            layer_batch_normalization() %>% 
                            layer_dropout(rate=parameters$dropOutRateThree) %>% #gut gegen overfitting
                            layer_dense(units = 1, activation = parameters$layer_dense_activation)

                            model %>% compile(loss = 'mse'
                            #optimizer = 'adam',
                            #,optimizer = optimizer_adam(lr=parameters$adamLearningRate)
                            ,optimizer = tf$keras$optimizers$Adam(hp$Choice('learning_rate',values=c(1e-2, 1e-3, 1e-4)))
                            ,metrics = list("mean_absolute_error"))

tuner = RandomSearch(build_model
                            #objective = 'val_accuracy',
                            #,objective = 'val_accuracy'
                            ,objective = 'val_loss'
                            ,max_trials = 5
                            #,executions_per_trial = 3
                            ,executions_per_trial = 5
                            ,directory = getwd()
                            ,project_name = 'helloworld'
                        )

Since I'm using
1.
,objective = 'val_loss'

metrics = list("mean_absolute_error")

Therefore:

Score: 1.580265924334526

wolud mean: is better then

Score: 16.740461111068726 ???