Image for Rock 4 SE RK 3399

Question

Image for Rock 4 SE RK 3399

ajay01994 opened this issue 2 years ago · 38 comments

ajay01994 commented 2 years ago

Hi ,could you please also help me. with releasing the same model for Rock pi 4SE ,RK 3399 architecture.this would be really helpful since there are supply chain issues with Raspberry pi

https://forum.radxa.com/c/rockpi4

Answer 1 · 2023-04-20T08:11:45.000Z

Dear @ajay01994,

If you look at Radxa's website, you are facing the same journey as I did.
Install a Debian operating system. I used the native Radxa Debian, as being as close to the hardware as possible.
Install OpenCV as described in our guide, just like the ncnn framework.
Not much can go wrong.

Answer 2 · 2023-04-21T04:57:16.000Z

Hi ,

Thanks for the reply , do you have a model zoo for YOLO V5,V7,V8 for RK 3399 , Also could you help with TNN how to install that ?

Answer 3 · 2023-04-21T11:45:33.000Z

The RK3399 has no NPU on board. You can use the ncnn models made for the Raspberry Pi. Or any other framework, like MNN, TNN, TensorFlow-Lite.
Please see the updated guide on how to install TNN on a ARM board like your Rock Pi 4 SE.

Answer 4 · 2023-04-23T14:36:45.000Z

Hi yes it worked for Rk 3399 thanks a lot !

I also bought a rk3588 with NPU , and was able to run yolov5 , do you have inference code for yolo v7,v8 for NPU ,and help with steps to install and inference on NPU for these models

Answer 5 · 2024-04-24T16:57:50.000Z

First of all great work @Qengineering . I appreciate your efforts.

Hi Ajay! can you tell me how can I run yolov5 as a demo on rk3588 with NPU. Thank you in advance.

And yes @Qengineering , I'm also interested in yolov8-seg inference with NPU. Thank you in advance you too!!

Answer 6 · 2024-04-24T17:11:50.000Z

I will port the NPU examples for the RK3566 to the RK3588 and upload them to GitHub next week.

Answer 7 · 2024-04-24T17:16:06.000Z

I have already ran build-linux_RK3588.sh file on rknpu2 examples but later I do not know what to do to make running yolov5 demo . If you guide me then it would be great. Thank you in advance

Answer 8 · 2024-04-25T10:31:19.000Z

@Qengineering Finally, I ran successfully yolov5 demo on the image you provided. Now, I would like to test yolov8_seg based on RK3588 whenever you put it. I'm really looking forward to testing it. Thank you once again!

Answer 9 · 2024-04-27T16:33:52.000Z

Please find: https://github.com/Qengineering/YoloV8-seg-NPU

Answer 10 · 2024-04-29T08:47:55.000Z

Hi @Qengineering , thank you so much for it. But I'm not able to download Rock 5 image from your sync site. This new image contains yolov8 examples right? And One more thing, if I would like to run my custom trained model inference on Rock 5b with NPU then should I change in .cpp right? and the model itself. Is there anything should I need to tale care of or?

Answer 11 · 2024-04-29T09:02:34.000Z

Sorry for the inconvenience. The nightly upload failed. Just restart it again.

I let you know when the link is active.

Answer 12 · 2024-04-29T09:06:09.000Z

To get your custom model up and running, you need to use the rknn_toolkit2 toolkit, installed on a Linux PC, you can port the model to an INT8 or FP16 rknn model. Please use the example that best fits your architecture. For instance, ~/rknn-toolkit2/rknn-toolkit2/examples/tensorflow. Here, use test.py to port your *.pd model to *.rknn. Sometimes you need to modify test.py a little to get everything working. Also, not every model can be ported to the NPU domain. And, of course, some models don't act well with INT8. Once running, postprocces.cpp is location where you alter the code to get your outputs. (Mostly)
Best to install the toolkit in a virtual environment, as it will pull in many thrid party packages, like TensorFlow, Pytorch, PaddlePaddle etc.

Answer 13 · 2024-04-29T09:21:15.000Z

The link is up and running.

Answer 14 · 2024-04-29T09:24:25.000Z

Thank you!! Sure! I will try it and will update you or ask you if I need any assistance. One more thing about the image. I'm using the rock5b image already from this repo. is it sufficient to test new examples or should I use this Rock5_ubuntu22.img for yolov8 seg?

Answer 15 · 2024-04-29T09:27:20.000Z

The latter. The Rock5B image doesn't have all the new examples. However, you could download the examples separately from the GitHub repo, as they were published this weekend. See the overview.

Answer 16 · 2024-04-29T09:31:52.000Z

So, basically, I can download the examples repo and can use in the existing Rock5 image and no need to use newer one. Right?

Answer 17 · 2024-04-29T09:44:00.000Z

OpenCV and the latest RKNPU2 library must be installed on your system.
OpenCV is available. Most likely, the RKNPU2 library needs to be upgraded.
Follow the instructions in the repo. It is only replacing one librknnrt.so and three *.h files.

Answer 18 · 2024-04-29T09:50:11.000Z

Thanks again. I will check.

Answer 19 · 2024-04-29T10:28:37.000Z

Hey @Qengineering, I'm trying to run the demo first and getting some error. could you please look into it please?

`rock@rock-5b:~/examples/YoloV8-seg-NPU$ ./YoloV8_seg busstop.jpg rk3588/yolov8n-seg.rknn
model input num: 1, output num: 13
input tensors:
index=0, name=images,
n_dims=4, dims=[1, 640, 640, 3],
n_elems=1228800, size=1228800, fmt=NHWC,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=375,
n_dims=4, dims=[1, 64, 80, 80],
n_elems=409600, size=409600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-55, scale=0.138304
index=1, name=onnx::ReduceSum_383,
n_dims=4, dims=[1, 80, 80, 80],
n_elems=512000, size=512000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.002873
index=2, name=388,
n_dims=4, dims=[1, 1, 80, 80],
n_elems=6400, size=6400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003169
index=3, name=354,
n_dims=4, dims=[1, 32, 80, 80],
n_elems=204800, size=204800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=26, scale=0.023277
index=4, name=395,
n_dims=4, dims=[1, 64, 40, 40],
n_elems=102400, size=102400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-40, scale=0.095424
index=5, name=onnx::ReduceSum_403,
n_dims=4, dims=[1, 80, 40, 40],
n_elems=128000, size=128000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003418
index=6, name=407,
n_dims=4, dims=[1, 1, 40, 40],
n_elems=1600, size=1600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=7, name=361,
n_dims=4, dims=[1, 32, 40, 40],
n_elems=51200, size=51200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=32, scale=0.020263
index=8, name=414,
n_dims=4, dims=[1, 64, 20, 20],
n_elems=25600, size=25600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-43, scale=0.075364
index=9, name=onnx::ReduceSum_422,
n_dims=4, dims=[1, 80, 20, 20],
n_elems=32000, size=32000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003874
index=10, name=426,
n_dims=4, dims=[1, 1, 20, 20],
n_elems=400, size=400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=11, name=368,
n_dims=4, dims=[1, 32, 20, 20],
n_elems=12800, size=12800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=26, scale=0.022538
index=12, name=347,
n_dims=4, dims=[1, 32, 160, 160],
n_elems=819200, size=819200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-119, scale=0.029378
model is NHWC input fmt
model input height=640, width=640, channel=3
Unable to init server: Could not connect: Connection refused
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(4.6.0) /home/rock/opencv/modules/highgui/src/window_gtk.cpp:635: error: (-2:Unspecified error) Can't initialize GTK backend in function 'cvInitSystem'

Aborted`

Answer 20 · 2024-04-29T11:38:08.000Z

Now, I'm not getting this error but it got stuck after this
...
...
model is NHWC input fmt
model input height=640, width=640, channel=3 and not proceeding further.

Answer 21 · 2024-04-29T11:53:03.000Z

Sorry to hear.
It seems all related to OpenCV. Your first issue was throwing an exception on the GUI module.
The second complains about the size of the input image not fitting the expected format.
The input image is also resized by OpenCV before hand over to the NPU.
I've tested all supplied examples twice. One for the RK3566 on a Radxa Zero 3w, and once for the RK3588 on a Rock 5 board.
All worked well.
You have two options.

Use the newly provided Rock 5 Ubuntu 22 image.
Debug OpenCV. Probably not in a healthy stage. A completely new installation may resolve the issue.
I would opt for 1.
If you are in desperate need of the original Armbian Rock 5 OS, perhaps I can install the examples and generate a new one this weekend. Not promising anything.

Answer 22 · 2024-04-29T12:09:50.000Z

My bad representation. In the second issue, it does not throw any error regarding image size. It just got stuck after this. and later not proceeding at all. Not throwing any error. I will try with the new image but I really want to run it on the Armbian Rock5 OS. But if it gets solved here then it would be great as it's not throwing any error. Just stucking on something.
model input num: 1, output num: 13
input tensors:
index=0, name=images,
n_dims=4, dims=[1, 640, 640, 3],
n_elems=1228800, size=1228800, fmt=NHWC,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=375,
n_dims=4, dims=[1, 64, 80, 80],
n_elems=409600, size=409600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-61, scale=0.115401
index=1, name=onnx::ReduceSum_383,
n_dims=4, dims=[1, 80, 80, 80],
n_elems=512000, size=512000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003514
index=2, name=388,
n_dims=4, dims=[1, 1, 80, 80],
n_elems=6400, size=6400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003540
index=3, name=354,
n_dims=4, dims=[1, 32, 80, 80],
n_elems=204800, size=204800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=27, scale=0.019863
index=4, name=395,
n_dims=4, dims=[1, 64, 40, 40],
n_elems=102400, size=102400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-15, scale=0.099555
index=5, name=onnx::ReduceSum_403,
n_dims=4, dims=[1, 80, 40, 40],
n_elems=128000, size=128000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003555
index=6, name=407,
n_dims=4, dims=[1, 1, 40, 40],
n_elems=1600, size=1600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003680
index=7, name=361,
n_dims=4, dims=[1, 32, 40, 40],
n_elems=51200, size=51200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=30, scale=0.022367
index=8, name=414,
n_dims=4, dims=[1, 64, 20, 20],
n_elems=25600, size=25600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-55, scale=0.074253
index=9, name=onnx::ReduceSum_422,
n_dims=4, dims=[1, 80, 20, 20],
n_elems=32000, size=32000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003813
index=10, name=426,
n_dims=4, dims=[1, 1, 20, 20],
n_elems=400, size=400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=11, name=368,
n_dims=4, dims=[1, 32, 20, 20],
n_elems=12800, size=12800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=43, scale=0.019919
index=12, name=347,
n_dims=4, dims=[1, 32, 160, 160],
n_elems=819200, size=819200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-119, scale=0.032336
model is NHWC input fmt
model input height=640, width=640, channel=3

Answer 23 · 2024-04-29T15:55:38.000Z

Hi @Qengineering , I tried the new image Rock5 ubuntu one and still getting the same error as mentioned below.
root@5:/home/rock/examples/YoloV8-seg-NPU# ./YoloV8_seg busstop.jpg rk3588/yolov8n-seg.rknn
model input num: 1, output num: 13
input tensors:
index=0, name=images,
n_dims=4, dims=[1, 640, 640, 3],
n_elems=1228800, size=1228800, fmt=NHWC,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
output tensors:
index=0, name=375,
n_dims=4, dims=[1, 64, 80, 80],
n_elems=409600, size=409600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-55, scale=0.138304
index=1, name=onnx::ReduceSum_383,
n_dims=4, dims=[1, 80, 80, 80],
n_elems=512000, size=512000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.002873
index=2, name=388,
n_dims=4, dims=[1, 1, 80, 80],
n_elems=6400, size=6400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003169
index=3, name=354,
n_dims=4, dims=[1, 32, 80, 80],
n_elems=204800, size=204800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=26, scale=0.023277
index=4, name=395,
n_dims=4, dims=[1, 64, 40, 40],
n_elems=102400, size=102400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-40, scale=0.095424
index=5, name=onnx::ReduceSum_403,
n_dims=4, dims=[1, 80, 40, 40],
n_elems=128000, size=128000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003418
index=6, name=407,
n_dims=4, dims=[1, 1, 40, 40],
n_elems=1600, size=1600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=7, name=361,
n_dims=4, dims=[1, 32, 40, 40],
n_elems=51200, size=51200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=32, scale=0.020263
index=8, name=414,
n_dims=4, dims=[1, 64, 20, 20],
n_elems=25600, size=25600, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-43, scale=0.075364
index=9, name=onnx::ReduceSum_422,
n_dims=4, dims=[1, 80, 20, 20],
n_elems=32000, size=32000, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003874
index=10, name=426,
n_dims=4, dims=[1, 1, 20, 20],
n_elems=400, size=400, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=11, name=368,
n_dims=4, dims=[1, 32, 20, 20],
n_elems=12800, size=12800, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=26, scale=0.022538
index=12, name=347,
n_dims=4, dims=[1, 32, 160, 160],
n_elems=819200, size=819200, fmt=NCHW,
type=INT8, qnt_type=AFFINE, zp=-119, scale=0.029378
model is NHWC input fmt
model input height=640, width=640, channel=3
terminate called after throwing an instance of 'cv::Exception'
what(): OpenCV(4.9.0-dev) /home/rock/opencv/modules/highgui/src/window_gtk.cpp:638: error: (-2:Unspecified error) Can't initialize GTK backend in function 'cvInitSystem'

Aborted (core dumped)

Answer 24 · 2024-04-30T08:08:12.000Z

@Qengineering I was able to run it. Instead of using imshow. I just saved image to see if it works or not. Atleast demo is working. Next thing is, to make it run for custom model. I will update you for it. Thank you once again.

Answer 25 · 2024-04-30T08:19:56.000Z

@apanand14,

Which monitor are you using? Or are you working with SSH?
When you see these two lines (model is NHWC input fmt, model input height=640, width=640, channel=3) your NPU is propper configured and an OpenCV cv::imshow() window must show your output.
Given the first error (Can't initialize GTK backend), OpenCV is the issue. It is not able to show something on the desktop.
Right now, I'm making a Rock-5 Armbian SD image with all the NPU examples.
I've tested the YoloV8-seg-NPU, and it is working fine. Creating an image, zipping and uploading will take several hours. I will let you know when we have a live link.

Answer 26 · 2024-04-30T09:50:47.000Z

Hi @Qengineering , I'm working with SSH only. And yes. GTK backend is the Opencv issue. But it's fine as long as I'm able to run and save the image with prediction and verify it. I'm trying to run my custom model. I will ask you if I need some assistance. Thank you for the help

Answer 27 · 2024-04-30T11:37:35.000Z

FYI. I've just published a new SD image with the RKNPU model zoo examples.

Answer 28 · 2024-04-30T11:55:02.000Z

Thank you! I will check it. One question to you. I have customized yolov8n-seg model. Customized I mean just different classes. Just 2 instead of 80. I think I do not have to change alot in cpp side right? I tried to run after converting my model but it didn't draw any mask but just showed FPS. Do you have any idea what could be wrong? Model conversion? or something needs to be changed in drawing mask? Thank you in advance.

Answer 29 · 2024-04-30T11:59:25.000Z

Now you're on your own. I have no idea. Both can be the case.
My problem is time. I'm very busy with other projects and, unfortunately, I can't dive into the code or the model to address the issue. How much I wanted to.