[EdgeTPU Backend] How to distinguish regular tflite model from edgeTPU-compiled tflite model?

[Hyeon-Uk]

Here is our project introduction

Project Introduction : #1621

Background : outputs after compiling tflite model with EdgeTPU

File before compiled with EdgeTPU and file after compiled with EdgeTPU are the same type, .tflite.

The image below illustrates the procedure of converting a .tflite model into an Edge TPU Model using the Edge TPU compiler. The red box located at the bottom represents the .tflite file intended for compilation, while the blue box symbolizes the resulting output of the compilation process.

It means that baseModel and Product are all tflite file. However, tflite file compiled with EdgeTPU doesn’t need to convert to circle file because edgetpu-custom-op cannot be inferenced in ONE-vscode. So this special tflite file is not a Base Model, it is a Product.

As a result, ONE explorer has to show the tflite file in Product type instead of Base Model type after detecting tflite file and checks whether the tflite is a regular file or is from EdgeTPU Compiler.

The way to distinguish Base Model from Product in ONE-vscode

1. Find the Base Model with filename extension such as .pb, .tflite.
2. Read the Base Model’s .cfg file and finds the results that corresponding with Output Path.
3. Display the results.

Of course, it assumes that the user performs all works in ONE-vscode interface.

Problem

As I mentioned, because the file type of the EdgeTPU Compiler’s result is .tflite, ONE-vscode reads this result two times, while finding Base Model and finding Product with Output Path.

We considered about the way to distinguish those two types(regular tflite and tflite from EdgeTPU Compiler) of tflite files.

Process we discussed

1. After generating Base Models, set Products of each Models corresponding to Output Path in cfg files of each Models.
2. At this point, EdgeTPU Compiler’s outputs saved seperately
3. Exclude those files from Base Model and generate the rest files as the Base Model.

However, we believe that this straightforward method might incur a high time complexity. So we keep thinking about the better method.

If you have a better idea about it, please give us an innovative idea.

[dayo09]

Process we discussed

1. After generating Base Models, set Products of each Models corresponding to Output Path in cfg files of each Models.
2. At this point, EdgeTPU Compiler’s outputs saved seperately
3. Exclude those files from Base Model and generate the rest files as the Base Model.

Could you explain more about this?

I agree that excluding edgetpu-compiled tflite files from BaseModel (data structure) is necessary in our view.

I suppose that we could define edgetpu-compiled files to have a certain postfix as *_edgetpu.tflite, like edgetpu-compiler do as default.

[Hyeon-Uk]

1. Before creating the TreeView component, we first utilize the getCfgList() function from OneStorage to fetch a list of cfg files.
2. Following this, we collect and save the output path generated by the EdgeTPU compiler into a variable.
3. Then, within the _buildChildren() code, we implement a conditional check to determine if the file is a .tflite file representing the baseModel.
4. In this check, we verify whether the model exists within the previously collected variables.
5. If we find the model in the collection, we've made the decision not to categorize it as a 'baseModel' because it originates from the EdgeTPU Compiler.

The method described above is what we have considered. We opted for this approach because we believed that if we differentiated them solely by postfix, it might limit our ability to customize the output path, and we would always need to include the postfix.

Initially, we will use postfix to differentiate them, and if we identify an additional method for distinguishing them, we will incorporate it as well.

[imsw0529]

We found a method

When view .tflite file compiled with edge tpu compiler using netron, we notice that description : Exported from Subgraph. We observed that this text appears in .tflite file compiled with edge tpu compiler, regardless of the original description.

Therefore, we concluded that if .tflite file has description : Exported from Subgraph, it is a edgetpu-compiled tflite file.

We attempted to open tflite file using flatbuffers library. We compiled schema.fbs file on Tensorflow's github using flatbuffers-compiler. And we success to read tflite file's description with schema_generated.ts that is result of flatbuffers-compiler.

Regarding this

1. We used Tensorflow's schema.fbs file that licensed under the Apache License V2. Is it ok?
2. When we open 6~7mb tflie file for the first time, it spend 20ms. After second times, it spend 3~4ms. Would this have significant impact on performace?

[dayo09]

@Hyeon-Uk

3. Then, within the _buildChildren() code, we implement a conditional check to determine if the file is a .tflite file representing the baseModel.

How do you 'check to determine'? 🤔
Do you mean that you see whether the file is written as 'Product' in the cfgs?

(I am still curious about this approach even though you have suggested another flatbuffer way below. We may opt this one in case of latency problem arises)

The method described above is what we have considered. We opted for this approach because we believed that if we differentiated them solely by postfix, it might limit our ability to customize the output path, and we would always need to include the postfix.

I totally agree with the point.

[dayo09]

@imsw0529

When view .tflite file compiled with edge tpu compiler using netron, we notice that description : Exported from Subgraph. We observed that this text appears in .tflite file compiled with edge tpu compiler, regardless of the original description.

That's cool :-D

1. We used Tensorflow's schema.fbs file that licensed under the Apache License V2. Is it ok?

Yes, Apache License is okay. :-D
It's the license we can use. (Apache 2.0)
Also the MIT License is fine to use.

2. When we open 6~7mb tflie file for the first time, it spend 20ms. After second times, it spend 3~4ms. Would this have significant impact on performace?

I think it matters... Imagine we have 500+ tflite files and latency gets 20ms * 500 = 10sec to load the view.
But it still sounds like a good thing that you discovered this one.

I think it's why we need to try modulization, to ensure the basic functionality of ONE-vscode while we try some optional functions for our backend.

[Hyeon-Uk]

@dayo09
#1630 This is a draft that solves the problem by referring to the attributes in the .cfg file without flatbuffer.

This is not the perfect way to solve this problem. so, we keep thinking about optimization of "flat buffers", or other methods,

[dayo09]

To make long short; we concluded that using _edgetpu.tflite prefix for edgetpu compiled models.