ctuning/ck-request-asplos18-mobilenets-tvm-arm

Benchmark on Firefly-RK3399

Closed this issue · 10 comments

psyhtest commented

Now that we have the latest Firefly-RK3399 BSP (xubuntu1604_201711301130), we should record benchmarking results on this platform in the CK format.

psyhtest commented

As I understand from the README, we have three programs all using the same command key:

$ ck run program:tvm-arm  --cmd_key=all
$ ck run program:tvm-nnvm  --cmd_key=all
$ ck run program:tvm-mxnet --cmd_key=all

How about we perform and record the experiments as follows:

$ ck benchmark program:tvm-<program name> \
--cmd_key=all --repetitions=10 \
--record --record_repo=local \
--record_uoa=ck-request-asplos18-tvm-<program name>-<model name>-<precision> \ 
--tags=request,request-asplos18,tvm,<program name>,<model name>,v<precision>

where:

  • is one of arm, nnvm, mxnet;
  • is one of mobilenets, alexnet, vgg16;
  • is one of fp32, fp16.
    (I'm writing this from memory so please correct me if I'm wrong.)
gfursin commented

Looks ok to me ;) ! Thanks! Just before taking off ;) ...

fvella commented

Tvm the support

model [mobilenet || resnet18 || vgg16] 
dtype=[float16 || float32]

ARM

backend[cl || neon]
model[mobilenet || vgg16] 
conv_method[gemm || direct] 
dtype[float16 || float32]"

Mxnet

model=[mobilenet || resnet18 || vgg16]
fvella commented

List of Experiments:

<name><backend><model><data_type>

MXNET

mxnet-openblas-mobilenet-gemm-float32
mxnet-openblas-resnet18-gemm-float32
mxnet-openblas-vgg16-gemm-float32

TVM

tvm-opencl-mobilenet-tvm-mali-float16
tvm-opencl-mobilenet-tvm-mali-float32
tvm-opencl-resnet18-tvm-mali-float16
tvm-opencl-resnet18-tvm-mali-float32
tvm-opencl-vgg16-tvm-mali-float16
tvm-opencl-vgg16-tvm-mali-float32

ARM

armcl-cl-mobilenet-direct-float16
armcl-cl-mobilenet-direct-float32
armcl-cl-mobilenet-gemm-float16
armcl-cl-mobilenet-gemm-float32
armcl-cl-vgg16-direct-float16
armcl-cl-vgg16-direct-float32
armcl-cl-vgg16-gemm-float16
armcl-cl-vgg16-gemm-float32
armcl-neon-mobilenet-direct-float16
armcl-neon-mobilenet-direct-float32
armcl-neon-mobilenet-gemm-float16
armcl-neon-mobilenet-gemm-float32
armcl-neon-vgg16-direct-float16
armcl-neon-vgg16-direct-float32
armcl-neon-vgg16-gemm-float16
armcl-neon-vgg16-gemm-float32
psyhtest commented

Can you please attach here a zip of the experimental data when it's ready?

fvella commented

ARM experiment had some problem. I have to re-run just them.

fvella commented

I have the results. WRT to the list posted before we have less experiments. The application does not allow to run some of them. i.e. neon with fp16..

gfursin commented

Cool! Thanks a lot @fvella - I am now preparing presentations for tomorrow and if I find time, I will check it tonight ...

gfursin commented

I unified all experiments and recorded them in the CK format.