Issues rebuilding bnn-pynq
gealy opened this issue · 2 comments
Hello,
This issue does not need a response. For the time being, I am moving to Vitis-AI. However, I wanted to let you know it seems like there are issues rebuilding the bnn-pynq examples.
I tested on two different build setups:
Setup 1
VirtualBox with Ubuntu 18.04, Vitis 2020.1
I followed the build instructions for bnn-pynq, however, I did use the pre-built docker image for version 0.7 of FINN. This may have caused some issues, but I am not positive.
My best guess here is that the VM was underpowered and simply not able to complete the build in a reasonable amount of time. Though, I am unclear why that would cause an issue.
I targeted building for both the ZCU104 and U200.
The main output in the docker container shows:
gealy@gealy-VirtualBox:~/finn-examples/build/finn$ ./run-docker.sh build_custom ~/finn-examples/build/bnn-pynq
WARNING: No swap limit support
Running build_custom: /home/gealy/finn-examples/build/bnn-pynq/build.py
Docker container is named finn_dev_gealy
Docker tag is named maltanar/finn:v0.7
Mounting /tmp/finn_dev_gealy into /tmp/finn_dev_gealy
Mounting /tools/Xilinx into /tools/Xilinx
Port-forwarding for Jupyter 8888:8888
Port-forwarding for Netron 8081:8081
Vivado IP cache dir is at /tmp/finn_dev_gealy/vivado_ip_cache
Using default PYNQ board Pynq-Z1
Obtaining file:///workspace/finn
Installing collected packages: finn
Running setup.py develop for finn
Successfully installed finn
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid (_ssl.c:1123)'))) - skipping
Found Vitis at /tools/Xilinx/Vitis/2020.1
XILINX_XRT : /opt/xilinx/xrt
PATH : /opt/xilinx/xrt/bin:/tools/Xilinx/Vitis/2020.1/bin:/tools/Xilinx/Vitis/2020.1/gnu/microblaze/lin/bin:/tools/Xilinx/Vitis/2020.1/gnu/arm/lin/bin:/tools/Xilinx/Vitis/2020.1/gnu/microblaze/linux_toolchain/lin64_le/bin:/tools/Xilinx/Vitis/2020.1/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin:/tools/Xilinx/Vitis/2020.1/gnu/aarch32/lin/gcc-arm-none-eabi/bin:/tools/Xilinx/Vitis/2020.1/gnu/aarch64/lin/aarch64-linux/bin:/tools/Xilinx/Vitis/2020.1/gnu/aarch64/lin/aarch64-none/bin:/tools/Xilinx/Vitis/2020.1/gnu/armr5/lin/gcc-arm-none-eabi/bin:/tools/Xilinx/Vitis/2020.1/tps/lnx64/cmake-3.3.2/bin:/tools/Xilinx/Vitis/2020.1/cardano/bin:/tools/Xilinx/Vivado/2020.1/bin:/tools/Xilinx/DocNav:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/workspace/oh-my-xilinx
LD_LIBRARY_PATH : /opt/xilinx/xrt/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
PYTHONPATH : /opt/xilinx/xrt/python:
Found XRT at /opt/xilinx/xrt
Building dataflow accelerator from models/cnv-w1a1.onnx
Intermediate outputs will be generated in /tmp/finn_dev_gealy
Final outputs will be generated in output_cnv-w1a1_ZCU104
Build log is at output_cnv-w1a1_ZCU104/build_dataflow.log
Running step: step_qonnx_to_finn [1/17]
Running step: step_tidy_up [2/17]
Running step: step_streamline [3/17]
Running step: step_convert_to_hls [4/17]
Running step: step_create_dataflow_partition [5/17]
Running step: step_target_fps_parallelization [6/17]
Running step: step_apply_folding_config [7/17]
Running step: step_generate_estimate_reports [8/17]
Running step: step_hls_codegen [9/17]
Running step: step_hls_ipgen [10/17]
Running step: step_set_fifo_depths [11/17]
Running step: step_create_stitched_ip [12/17]
Running step: step_measure_rtlsim_performance [13/17]
Running step: step_out_of_context_synthesis [14/17]
Running step: step_synthesize_bitfile [15/17]
ERROR: [Common 17-69] Command failed: Run 'impl_1' has not been launched. Unable to open
Traceback (most recent call last):
File "/workspace/finn/src/finn/builder/build_dataflow.py", line 166, in build_dataflow_cfg
model = transform_step(model, cfg)
File "/workspace/finn/src/finn/builder/build_dataflow_steps.py", line 621, in step_synthesize_bitfile
model = model.transform(
File "/workspace/finn-base/src/finn/core/modelwrapper.py", line 141, in transform
(transformed_model, model_was_changed) = transformation.apply(
File "/workspace/finn/src/finn/transformation/fpgadataflow/make_zynq_proj.py", line 356, in apply
model = model.transform(
File "/workspace/finn-base/src/finn/core/modelwrapper.py", line 141, in transform
(transformed_model, model_was_changed) = transformation.apply(
File "/workspace/finn/src/finn/transformation/fpgadataflow/make_zynq_proj.py", line 274, in apply
raise Exception(
Exception: Synthesis failed, no bitfile found. Check logs under /tmp/finn_dev_gealy/vivado_zynq_proj_bsc0nfeg
> /workspace/finn/src/finn/transformation/fpgadataflow/make_zynq_proj.py(274)apply()
-> raise Exception(
The end of the accompanying vivado.log file shows:
[Mon Apr 5 16:20:46 2021] Launched impl_1...
Run output will be captured here: /tmp/finn_dev_gealy/vivado_zynq_proj_bsc0nfeg/finn_zynq_link.runs/impl_1/runme.log
launch_runs: Time (s): cpu = 00:00:36 ; elapsed = 00:00:43 . Memory (MB): peak = 2392.496 ; gain = 184.086 ; free physical = 3606 ; free virtual = 10113
# wait_on_run [get_runs impl_1]
[Mon Apr 5 16:20:46 2021] Waiting for impl_1 to finish...
[Mon Apr 5 18:13:53 2021] impl_1 finished
WARNING: [Vivado 12-8222] Failed run(s) : 'top_StreamingDataflowPartition_1_0_synth_1'
wait_on_run: Time (s): cpu = 00:00:06 ; elapsed = 01:53:09 . Memory (MB): peak = 2392.496 ; gain = 0.000 ; free physical = 8291 ; free virtual = 10109
# open_run impl_1
ERROR: [Common 17-69] Command failed: Run 'impl_1' has not been launched. Unable to open
INFO: [Common 17-206] Exiting Vivado at Mon Apr 5 18:14:02 2021...
The Vivado log doesn't really indicate what the problem is, but perhaps it will mean more to you.
Setup 2
In this setup, I attempted to run the docker container on a large computing node that has 3 Alveo U200 boards attached.
This server is running a version of Red Hat Linux. On this node, I am able to successfully build with Vitis-AI docker containers.
On this system, I tested with both the pre-built FINN docker container (version 0.7) and with the custom-built version generated by the docker_run.sh script.
The issues seen here were slightly different than before.
Building dataflow accelerator from models/cnv-w1a1.onnx
Intermediate outputs will be generated in /ihome/ageorge/cag158/tmp/finn_dev_cag158
Final output will be generated in output_cnv-w1a1_ZCU104
Build log is at output_cnv-w1a1_ZCU104/build_dataflow.log
Running step: step_qonnx_to_finn [1/17]
Running step: step_tidy_up [2/17]
Running step: step_streamline [3/17]
Running step: step_convert_to_hls [4/17]
Running step: step_create_dataflow_partition [5/17]
Running step: step_target_fps_parallelization [6/17]
Running step: step_apply_folding_config [7/17]
Running step: step_generate_estimate_reports [8/17]
Running step: step_hls_codegen [9/17]
Running step: step_hls_ipgen [10/17]
Running step: step_set_fifo_depths [11/17]
ERROR: [BD 5-390] IP definition not found for VLNV: xilinx.com:hls: Thresholding_Batch_0:1.0
Traceback (most recent call last):
File "/workspace/finn/src/finn/builder/build_dataflow.py", line 166, in build_dataflow_cfg
model = transform_step(model, cfg)
File "/workspace/finn/src/finn/builder/build_dataflow_steps.py", line 450, in step_set_fifo_depths
model = model.transform(
File "/workspace/finn-base/src/finn/core/modelwrapper.py", line 141, in transform
(transformed_model, model_was_changed) = transformation.apply(
File "/workspace/finn/src/finn/transformation/fpgadataflow/set_fifo_depths.py", line 306, in apply
sim = pyverilate_stitched_ip (model)
File "/workspace/finn-base/src/finn/util/pyverilator.py", line 167, in pyverilate_stitched_ip
with open(vivado_stitch_proj_dir + "/all_verilog_srcs.txt", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/ihome/ageorge/cag158/tmp/finn_dev_cag158/vivado_stitch_proj_dd_fh5q
k/all_verilog_srcs.txt'
> /workspace/finn-base/src/finn/util/pyverilator.py (167) pyverilate_stitched_ip()
-> with open(vivado_stitch_proj_dir + "/all_verilog_srcs.txt", "r") as f:
(Pdb) |
It seems like some file that should have been autogenerated was not. It is curious to me that I was able to get farther in the build process on setup 1. I'm not sure if this is an issue with the docker image or with some other setup variable.
Since I was targeting different install locations, I did have to change my environment variables on the different setups. However, I did try to set the same ones on each machine. On the second setup, I had the following environment variables in my .bashrc:
export LC_ALL="C"
export FINN_XILINX_PATH=/ihome/crc/install/xilinx
export FINN_XILINX_VERSION=2020.1
export VIVADO_PATH=/ihome/crc/install/xilinx/Vivado/2020.1/
#export FINN_DOCKER_TAG=maltanar/finn:v0.7
#export FINN_DOCKER_PREBUILT=1
export PLATFORM_REPO_PATHS=/opt/xilinx
Again, I don't need an immediate update to this issue. I think FINN sounds like a cool tool, but this project is on a bit of a deadline, so I am switching to Vitis-AI for now.
Please let me know if I can provide any additional info.
Hi,
for 1) you'll have to dig deeper into the logs to find the cause. For example in "finn_zynq_link.runs/impl_1/runme.log" or the Vivado HLS logs inside the "code_gen_ipgen_..." temporary folders of problematic IP cores.
2 is likely caused by the Y2K22 bug (https://support.xilinx.com/s/article/76960?language=en_US). Did you apply the patch?
Closing this issue for now, please feel free to re-open if your problem isn't solved.