XiaoMi/mace

Quantization Problems

didadida-r opened this issue · 3 comments

didadida-r commented

Before you open an issue, please make sure you have tried the following steps:

  1. Make sure your environment is the same with (https://mace.readthedocs.io/en/latest/installation/env_requirement.html).
  2. Have you ever read the document for your usage?
  3. Check if your issue appears in HOW-TO-DEBUG or FAQ.
  4. The form below must be filled.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): 16.04
  • NDK version(e.g., 15c): r16b
  • GCC version(if compiling for host, e.g., 5.4.0): 5.4.0
  • MACE version (Use the command: git describe --long --tags): v0.13.0-0-g9a06864
  • Bazel version (e.g., 0.13.0): 0.13.0

Model deploy file (*.yml)

  library_name: test
  target_abis: [host]
  model_graph_format: file
  model_data_format: file
  models:
    test:
      platform: onnx
      model_file_path: test.onnx
      model_sha256_checksum: 471a9fbc491929433ee43bc8d2222df19ea4626fa69eb315765e97cf5974222a
      subgraphs:
        - input_tensors:
            - input
          input_shapes:
            - 1,60,40
          output_tensors:
            - output.affine
          output_shapes:
            - 1,60,1024
          backend: kaldi
          validation_inputs_data:
            - test_input.npy
          validation_outputs_data:
            - kaldi_output.npy
      backend: kaldi
      runtime: cpu
      limit_opencl_kernel_time: 0
      nnlib_graph_mode: 0
      obfuscate: 0
      data_type: fp16_fp32

Describe the problem

Hi, i follow the doc about quantization, and here is how i construct my input data dir, just mimic the way in tools/image/image_to_tensor.py
a. clip the feat chunk with shape the same as the input_shapes in yml file.
b. save each chunk in numpy array to /path/to/input_data dir with suffix '.dat'.

Then i run the command, but it fails, and the log is in Error information / logs

python3 $MACE_ROOT/tools/converter.py run --config /path/to/yml \
    --quantize_stat --input_dir /path/to/input_data > range_log

And my questions are,
a. if followed the correct step, can i do quantization in mace with kaldi model.
b. where should i use the command in the doc rename 's/^/input/' *, because the file in /path/to/input_data is just some numpy array.
c. what is option quantize_stat do?

  • because in the doc, the fisrt step is 1. Convert original model to run on CPU ..., and i can see the --quantize_stat is used.
  • and then in the second step, 2. Log tensor range of each activation ..., i can see the --quantize_stat is used.

d. what is the different in:

  • quantize in yaml
  • quantize in tools/bazel_build_standalone_lib
  • quantize_large_weights in yaml, why this parameter can not be used when quantize in yaml is 1.
  • quantize_stat in tools/converter.py

Maybe the question is a lot and many thanks!

To Reproduce

Steps to reproduce the problem:

1. cd /path/to/mace
2. python tools/converter.py convert --config_file=/path/to/your/model_deployment_file

Error information / logs

Please include the full log and/or traceback here.

/path/to/mace ~/workstation/kaldi2mace
/path/to/mace/tools/converter.py:339: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  configs = yaml.load(f)
Loading:
Loading: 0 packages loaded
INFO: Analysed target //mace/tools:mace_run_static (0 packages loaded).
INFO: Found 1 target...
[0 / 3] [-----] BazelWorkspaceStatusAction stable-status.txt
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 0.678s, Critical Path: 0.01s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Traceback (most recent call last):
  File "/path/to/mace/tools/converter.py", line 1176, in <module>
    flags.func(flags)
  File "/path/to/mace/tools/converter.py", line 959, in run_mace
    device.run_specify_abi(flags, configs, target_abi)
  File "/path/to/mace/tools/device.py", line 791, in run_specify_abi
    # tuned=tuned)
  File "/path/to/mace/tools/device.py", line 830, in report_run_statistics
    for line in self.stdout.split('\n'):
TypeError: a bytes-like object is required, not 'str

Additional context

Add any other context about the problem here, e.g., what you have modified about the code.

lu229 commented

@groovemaxRong v0.13.0 is too old, please use the v1.0.4?

didadida-r commented

@lu229 hi, i'd updated the mace to v1.0.4, and run the command python3 $MACE_ROOT/tools/converter.py run --config /path/to/yml --quantize_stat --input_dir /path/to/input_data >range_log sucessfully, but the range log is not showed as expected.

And its it right to simply put the numpy matrix file in the --input_dir, something like 1.dat 2.dat 3.dat?

Detail Log:

WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
Loading:
Loading: 0 packages loaded
Analyzing: target //mace/tools:mace_run_static (5 packages loaded)
Analyzing: target //mace/tools:mace_run_static (10 packages loaded)
INFO: Analysed target //mace/tools:mace_run_static (28 packages loaded).
Building: no action
INFO: Found 1 target...
[0 / 2] [-----] BazelWorkspaceStatusAction stable-status.txt
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.devtools.build.lib.unsafe.StringUnsafe (file:/root/.cache/bazel/_bazel_root/install/c25ea2c3043bcba07b93dde10595066c/_embedded_binaries/A-server.jar) to constructor java.lang.String(byte[],byte)
WARNING: Please consider reporting this to the maintainers of com.google.devtools.build.lib.unsafe.StringUnsafe
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[26 / 309] Compiling mace/port/file_system.cc; 0s local ... (55 actions running)
[40 / 309] Compiling mace/utils/statistics.cc; 1s local ... (55 actions running)
[64 / 309] Compiling mace/utils/statistics.cc; 2s local ... (54 actions running)
[80 / 309] Compiling mace/utils/statistics.cc; 4s local ... (55 actions running)
[98 / 309] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc; 5s local ... (55 actions running)
[123 / 309] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc; 6s local ... (55 actions running)
[151 / 309] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc; 8s local ... (48 actions running)
[198 / 309] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.cc; 10s local
[205 / 309] Compiling mace/proto/mace.pb.cc; 1s local ... (55 actions running)
[248 / 309] Compiling mace/proto/mace.pb.cc; 4s local ... (55 actions running)
[308 / 309] [-----] Linking mace/tools/mace_run_static
Target //mace/tools:mace_run_static up-to-date:
  bazel-bin/mace/tools/mace_run_static
INFO: Elapsed time: 25.603s, Critical Path: 19.67s
INFO: 292 processes: 292 local.
INFO: Build completed successfully, 309 total actions
INFO: Build completed successfully, 309 total actions
lu229 commented

@groovemaxRong Please refer to this document:
Rename input tensors to start with input tensor name(to differentiate multiple
# inputs of a model), input tensor name is what you specified as "input_tensors"
# in yaml config. For example, "input" is the input tensor name of InceptionV3 as below.
rename 's/^/input/' *