[convert_keras.py] Failed generating descriptor for webassembly backend after executing convert.sh under webdnn/example/benchmark folder.
BruceDai opened this issue · 2 comments
test@test:~/workspace/webdnn/example/benchmark$ ./convert.sh
Generate Keras model
Collecting keras==2.1.2 (from -r ./requirements_kerasjs.txt (line 1))
Using cached https://files.pythonhosted.org/packages/68/89/58ee5f56a9c26957d97217db41780ebedca3154392cb903c3f8a08a52208/Keras-2.1.2-py2.py3-none-any.whl
Requirement already satisfied: tensorflow in /usr/local/lib/python3.6/dist-packages (from -r ./requirements_kerasjs.txt (line 2)) (1.4.0)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from -r ./requirements_kerasjs.txt (line 3)) (2.8.0)
Requirement already satisfied: protobuf>=3.4 in /usr/local/lib/python3.6/dist-packages (from -r ./requirements_kerasjs.txt (line 4)) (3.6.1)
Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.2->-r ./requirements_kerasjs.txt (line 1)) (1.15.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras==2.1.2->-r ./requirements_kerasjs.txt (line 1)) (3.13)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.2->-r ./requirements_kerasjs.txt (line 1)) (1.1.0)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.2->-r ./requirements_kerasjs.txt (line 1)) (1.11.0)
Requirement already satisfied: tensorflow-tensorboard<0.5.0,>=0.4.0rc1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->-r ./requirements_kerasjs.txt (line 2)) (0.4.0)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.6/dist-packages (from tensorflow->-r ./requirements_kerasjs.txt (line 2)) (0.31.1)
Requirement already satisfied: enum34>=1.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->-r ./requirements_kerasjs.txt (line 2)) (1.1.6)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.4->-r ./requirements_kerasjs.txt (line 4)) (39.1.0)
Requirement already satisfied: werkzeug>=0.11.10 in /usr/local/lib/python3.6/dist-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->-r ./requirements_kerasjs.txt (line 2)) (0.14.1)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->-r ./requirements_kerasjs.txt (line 2)) (2.6.11)
Requirement already satisfied: bleach==1.5.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->-r ./requirements_kerasjs.txt (line 2)) (1.5.0)
Requirement already satisfied: html5lib==0.9999999 in /usr/local/lib/python3.6/dist-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->-r ./requirements_kerasjs.txt (line 2)) (0.9999999)
Installing collected packages: keras
Found existing installation: Keras 2.1.3
Uninstalling Keras-2.1.3:
Successfully uninstalled Keras-2.1.3
Successfully installed keras-2.1.2
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
2018-08-20 16:34:39.278287: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
2018-08-20 16:34:47.808843: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
2018-08-20 16:34:52.778784: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
Encode Keras model into Keras.js model:
Saved to binary file /home/test/workspace/webdnn/example/benchmark/output/kerasjs/resnet50/model.bin
Saved to binary file /home/test/workspace/webdnn/example/benchmark/output/kerasjs/vgg16/model.bin
Saved to binary file /home/test/workspace/webdnn/example/benchmark/output/kerasjs/inception_v3/model.bin
/home/test/workspace/webdnn/example/benchmark
Encode Keras model into WebDNN model:
Collecting keras==2.1.3 (from -r ./requirements_webdnn.txt (line 1))
Using cached https://files.pythonhosted.org/packages/08/ae/7f94a03cb3f74cdc8a0f5f86d1df5c1dd686acb9a9c2a421c64f8497358e/Keras-2.1.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.3->-r ./requirements_webdnn.txt (line 1)) (1.11.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras==2.1.3->-r ./requirements_webdnn.txt (line 1)) (3.13)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.3->-r ./requirements_webdnn.txt (line 1)) (1.1.0)
Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.6/dist-packages (from keras==2.1.3->-r ./requirements_webdnn.txt (line 1)) (1.15.0)
Installing collected packages: keras
Found existing installation: Keras 2.1.2
Uninstalling Keras-2.1.2:
Successfully uninstalled Keras-2.1.2
Successfully installed keras-2.1.3
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
[convert_keras.py] Generating feedforward graph
2018-08-20 16:35:03.722711: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
[convert_keras.py] Generating graph descriptor
[convert_keras.py] BackendName: webgpu
/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/util/console.py:30: Warning: [WebGPUDescriptorGenerator] 'xcrun' command is not found. validation of generated source code in webgpu backend is skipped.
warnings.warn(message, category)
[convert_keras.py] BackendName: webgl
[convert_keras.py] BackendName: webassembly
WARNING:root:not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]
Traceback (most recent call last):
File "/home/test/workspace/emsdk/emscripten/incoming/emcc.py", line 3057, in
sys.exit(run())
File "/home/test/workspace/emsdk/emscripten/incoming/emcc.py", line 1080, in run
assert not shared.Settings.WASM, 'LEGACY_VM_SUPPORT is only supported for asm.js, and not wasm. Build with -s WASM=0'
AssertionError: LEGACY_VM_SUPPORT is only supported for asm.js, and not wasm. Build with -s WASM=0
Executing em++ command failed. Make sure emscripten is properly installed and environment variables are set.
[convert_keras.py] Failed generating descriptor for webassembly backend
Traceback (most recent call last):
File "../../bin/convert_keras.py", line 98, in main
graph_exec_data.save(output_dir)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webassembly/generator.py", line 50, in save
self._compile_fallback_asmjs(dirname)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webassembly/generator.py", line 101, in _compile_fallback_asmjs
raise ex
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webassembly/generator.py", line 97, in _compile_fallback_asmjs
subprocess.check_call(args, shell=self.platform_windows)
File "/usr/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['em++', 'output/webdnn/resnet50/non_optimized/kernels_webassembly.cpp', '-O3', '-std=c++11', '-s', "EXPORTED_FUNCTIONS=['_run','_init','_get_static_buffer','_allocate_dynamic_buffer','_get_dynamic_buffer','_set_placeholder_value']", '-s', 'TOTAL_MEMORY=603979776', '-s', 'LEGACY_VM_SUPPORT=1', '--pre-js', '/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webassembly/webassembly_header.js', '-o', 'output/webdnn/resnet50/non_optimized/kernels_asmjs.js']' returned non-zero exit status 1.
[convert_keras.py] BackendName: fallback
This issue is due to #874, @milhidaka PTAL, thanks.
When I try to modify https://github.com/mil-tokyo/webdnn/blob/master/src/graph_transpiler/webdnn/backend/webassembly/generator.py#L91 likes
args.append(f"LEGACY_VM_SUPPORT=0")continue to execute convert.sh script, I met the new issue " Failed generating descriptor for webgl backend":
test@test:~/workspace/webdnn/example/benchmark$ ./convert.sh
...
Successfully installed keras-2.1.3
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
[convert_keras.py] Generating feedforward graph
2018-08-20 16:45:00.539711: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
[convert_keras.py] Generating graph descriptor
[convert_keras.py] BackendName: webgpu
/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/util/console.py:30: Warning: [WebGPUDescriptorGenerator] 'xcrun' command is not found. validation of generated source code in webgpu backend is skipped.
warnings.warn(message, category)
[convert_keras.py] BackendName: webgl
[convert_keras.py] BackendName: webassembly
WARNING:root:not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]
[convert_keras.py] BackendName: fallback
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
[convert_keras.py] Generating feedforward graph
2018-08-20 16:45:22.648588: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
[convert_keras.py] Generating graph descriptor
[convert_keras.py] BackendName: webgpu
/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/util/console.py:30: Warning: [WebGPUDescriptorGenerator] 'xcrun' command is not found. validation of generated source code in webgpu backend is skipped.
warnings.warn(message, category)
[convert_keras.py] BackendName: webgl
[convert_keras.py] BackendName: webassembly
WARNING:root:not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]
[convert_keras.py] BackendName: fallback
Using TensorFlow backend.
/usr/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
return f(*args, **kwds)
[convert_keras.py] Generating feedforward graph
2018-08-20 16:45:47.794974: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
[convert_keras.py] Generating graph descriptor
[convert_keras.py] BackendName: webgpu
/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/util/console.py:30: Warning: [WebGPUDescriptorGenerator] 'xcrun' command is not found. validation of generated source code in webgpu backend is skipped.
warnings.warn(message, category)
[convert_keras.py] BackendName: webgl
**[convert_keras.py] Failed generating descriptor for webgl backend
Traceback (most recent call last):
File "../../bin/convert_keras.py", line 97, in main
graph_exec_data = generate_descriptor(backend, graph, constant_encoder_name=args.encoding)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/interface/generator.py", line 107, in generate_descriptor
return generator(graph, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webgl/generator.py", line 92, in generate
return WebGLDescriptorGenerator.generate(graph, kwargs)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webgl/generator.py", line 45, in generate
graph, _ = WebGLOptimizeRule().optimize(copy.deepcopy(original_graph))
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/graph/optimize_rule.py", line 110, in optimize
graph, flag_changed = sub_rule.optimize(graph)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/graph/optimize_rule.py", line 110, in optimize
graph, flag_changed = sub_rule.optimize(graph)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/graph/optimize_rule.py", line 110, in optimize
graph, flag_changed = sub_rule.optimize(graph)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webgl/optimize_rules/split_texture/split_variable.py", line 49, in optimize
axis = _choose_split_axis(v)
File "/usr/local/lib/python3.6/dist-packages/webdnn-1.2.5-py3.6.egg/webdnn/backend/webgl/optimize_rules/split_texture/split_variable.py", line 208, in _choose_split_axis
raise ValueError("No axis is splittable")
ValueError: No axis is splittable
[convert_keras.py] BackendName: webassembly
WARNING:root:not all asm.js optimizations are possible with ALLOW_MEMORY_GROWTH, disabling those. [-Walmost-asm]
[convert_keras.py] BackendName: fallback