View models?

[GrayGriffin]

Is there a way to convert the structure of the generated models to a readable textfile? Currently when I try to do so the output is a bunch of symbols.

[vaenyr]

Hi! What search space are using? And is that for predicting or measuring?

[GrayGriffin]

I'm currently using the default nasbench201 part. I'm trying to write a new metric and need a text description of the model structure for it.

[vaenyr]

I see, do you need a full model or just a single cell? You can get a graph representation of a single cell by using this function. A full model is constructed by stacking multiple cells with different number of channels and input image size as shown in Figure 1 of NAS-Bench-201 paper. We don't currently have a function that would create a graph/visualize a full model.

If that's not helpful, I think I would be able to help more if you could explain what you mean by "text description". E.g., do you mean arch description as used in NB2 repo like |nor_conv_3x3~0|+|nor_conv_3x3~0|avg_pool_3x3~1|+|skip_connect~0|nor_conv_3x3~1|skip_connect~2|?
If yes, you can convert our identifiers to the textual identifiers used within NB2 codebase by using this function.

[GrayGriffin]

That kind of text description is what I'm looking for. however, when I try to directly save the model as a string, it tends to output a collection of numbers and symbols like in the attached file:

[vaenyr]

Well, this seems to be protobuf's text representation of a tensor's content (see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/tensor.proto#L38) - this representation uses octal escapes to encode bytes beween 0-32 and 127+.

I'm guessing you're using one of those functions to save a model?
eagle.models.nasbench201.tf_model.build_and_convert_to_graph_def, eagle.models.nasbench201.tf_model.build_graph and/or eagle.models.tf_utils.convert_to_graph_def

If yes, then you can see quite a few of those inlined tensors due to this line in convert_to_graph_def:

graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), output)

which freezes all parameters and encodes their current values directly in a model definition. You can get a representation of your model without those if you comment this line (use sess.graph.as_graph_def() instead) but then you'll get extra nodes representing those parameters explicitly and relevant Read operations (and some other nodes as well), resulting in an overall more complicated graph. Also, even without conversion to constants, the resulting graph would still have some constants that would be encoded similarly, so you won't get rid of them completely - specifically, e.g. shapes used by initializers are encoded as "Const" ops as well.

E.g., without freezing variables:

node {
  name: "nasbench201_dnn/dense/bias"
  op: "VarHandleOp" <------ HERE
  // no tensor_content defined for Variables
  (...)
}
(...)
node {
  name: "nasbench201_dnn/dense/BiasAdd/ReadVariableOp" <-- explicit "Read" operation
  op: "ReadVariableOp"
  input: "nasbench201_dnn/dense/bias" <-- this variable is going to be read
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
}
node {
  name: "nasbench201_dnn/dense/BiasAdd"
  op: "BiasAdd"
  input: "nasbench201_dnn/dense/MatMul"
  input: "nasbench201_dnn/dense/BiasAdd/ReadVariableOp" <-- operand is result of "Read"
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "data_format"
    value {
      s: "NHWC"
    }
  }
}

with freezing:

node {
  name: "nasbench201_dnn/dense/bias"
  op: "Const" <--- bias is now constant
  attr {
    key: "dtype"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "value"
    value {
      tensor {
        dtype: DT_FLOAT
        tensor_shape {
          dim {
            size: 10
          }
        } // Bias value is inlined in the model definition and encoded with octal escapes (here, zeros only)
        tensor_content: "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
      }
    }
  }
}
node {
  name: "nasbench201_dnn/dense/BiasAdd"
  op: "BiasAdd"
  input: "nasbench201_dnn/dense/MatMul"
  input: "nasbench201_dnn/dense/bias" <-- operand is now a constant node itself
  attr {
    key: "T"
    value {
      type: DT_FLOAT
    }
  }
  attr {
    key: "data_format"
    value {
      s: "NHWC"
    }
  }
}

In either way, I'm not sure why the presence of inlined constants would be a problem, the text file you send seems like a correct protobuf of a graph. What is the problem here?

[vaenyr]

Closing due to lack of activity, Feel free to reopen if needed.

[GrayGriffin]

Reopening this because I've sort of figured out how this works and was working on it alone. However, I've sort of figured out how the conversion to text format works, but it seems to require the "gs" data that is only gained through the Torch DataLoader function. Is there a way to get this data but for an individual model?

[vaenyr]

To be honest, I really struggle to understand your questions, so maybe let me just explain roughly how different parts of our project interact:

• each search space (nasbench201, nasbench101, etc., in general, subfolders under eagle/models) defines a function that returns an iterator over identifiers of all models within in. E.g., for nb2 identifiers are vectors and look roughly like this: [1,0,0,3,4,5].
• those identifiers can be used to either: 1) create a model that can be used for benchmarking (e.g. eagle/models/nasbench201/tf_model.py); or 2) get a graph representation of a model (graph representation == adjacency matrix + a list of node labels) that can be used as input to a predictor. In general, 1 and 2 are completely independent use cases and do not depend on each other. Example: eagle.models.nasbench201.get_matrix_and_ops([1,0,0,3,4,5]) should return something like:

>>> import eagle.models.nasbench201 as nb2
>>> m = [1,0,0,3,4,5]
>>> nb2.get_matrix_and_ops(m)
([[0, 1, 1, 0], [0, 0, 0, 1], [0, 0, 0, 1], [0, 0, 0, 0]], ['input', '3', '4', 'output'])

where the first element is adjacency matrix and the second is a list of node labels, so the graph representation for this model looks like this:

       input
     /       \
    3         4
     \       /
       output

• the graph representation from the previous point can be further transform by passing it to get_adjacency_and_features function from the same module - this function performs some extra steps that are useful when training graph neural networks (so our predictor), you should not use them if you only want to see how a graph looks like
• the two functions mentioned above (get_matrix_and_ops and get_adjacency_and_features) are used in the code that deals with training and testing predictors, being given a list of models identifiers our code needs to convert those ids to graphs using both functions and combine them into a single tensor that holds "batched" graphs - this is performed here: https://github.com/SamsungLabs/eagle/blob/main/eagle/predictors/infer.py#L55
• this prepare_tensors method is used directly in the training code, so there's not much to talk about - maybe the only thing to note is that the actual Pytorch Dataloader outputs a list of model ids and relevant labels (e.g., accuracy of each model, if we are predicting accuracy) - the graph are constructed on-the-fly for each batch, similarly to how augmentation is done.

Hope that clarifies everything.