Fails to cross-compile for armv7/aarch64 architectures
quietlychris opened this issue · 10 comments
Hello,
First, just wanted to say that I've really enjoyed working with this crate so far! Being able to configure ROS nodes without needing to tie into the entire colcon/catkin build structure is really a joy.
I'm interested in deploying a simple monitoring node to an RPi, but have run into an issue with the compiling r2r for architectures besides x86 (my host computer is running an Intel i5-5300U). I've verified that my cross-compilation toolchain is working (per the rust-cross repository) and can compile and run simple armv7 programs on my target Pi.
The follow compiles for my host machine, but produces an error when I run the $ cargo build --target=armv7-unknown-linux-gnueabihf command.
use failure::Error;
use r2r;
use std::time::Duration;
fn main() -> Result<(), Error> {
let ctx = r2r::Context::create()?;
let mut node = r2r::Node::create(ctx, "echo", "")?;
let topic: String = "/rosout".to_string();
let cb = move |msg: r2r::rcl_interfaces::msg::Log| {
println!("raw_msg: {:#?}", &msg);
};
let _ws2 = node.subscribe(&topic, Box::new(cb))?;
loop {
node.spin_once(Duration::from_millis(100));
}
Ok(())
}The above does compile on my host computer using $ cargo build, but does not compile for armv7, producing the following errors:
error[E0063]: missing field `__bindgen_padding_0` in initializer of `rmw_message_info_t`
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:13:9
|
13 | rmw_message_info_t {
| ^^^^^^^^^^^^^^^^^^ missing `__bindgen_padding_0`
error[E0063]: missing fields `__bindgen_padding_0`, `__bindgen_padding_1` in initializer of `rmw_qos_profile_t`
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:27:9
|
27 | rmw_qos_profile_t {
| ^^^^^^^^^^^^^^^^^ missing `__bindgen_padding_0`, `__bindgen_padding_1`
error[E0308]: mismatched types
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:46:41
|
46 | let s = unsafe { CStr::from_ptr(self.data as *mut i8) };
| ^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
|
= note: expected raw pointer `*const u8`
found raw pointer `*mut i8`
error[E0308]: mismatched types
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:54:62
|
54 | rosidl_runtime_c__String__assign(self as *mut _, to_send_ptr);
| ^^^^^^^^^^^ expected `u8`, found `i8`
|
= note: expected raw pointer `*const u8`
found raw pointer `*const i8`
error[E0308]: mismatched types
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:127:62
|
127 | unsafe { std::ptr::copy(values.as_ptr(), self.data, values.len()); }
| ^^^^^^^^^ expected `u128`, found `f64`
...
143 | primitive_sequence!(rosidl_runtime_c__long_double, u128);
| --------------------------------------------------------- in this macro invocation
|
= note: expected raw pointer `*mut u128`
found raw pointer `*mut f64`
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
error[E0308]: mismatched types
--> /home/chrism/.cargo/git/checkouts/r2r-c48e5652e9219d1b/9bc75b6/rcl/src/lib.rs:134:21
|
130 | pub fn to_vec(&self) -> Vec<$element_type> {
| ------------------ expected `Vec<u128>` because of return type
...
134 | target
| ^^^^^^ expected `u128`, found `f64`
...
143 | primitive_sequence!(rosidl_runtime_c__long_double, u128);
| --------------------------------------------------------- in this macro invocation
|
= note: expected struct `Vec<u128>`
found struct `Vec<f64>`
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
error: aborting due to 6 previous errors
Some errors have detailed explanations: E0063, E0308.
For more information about an error, try `rustc --explain E0063`.
The following warnings were emitted during compilation:
warning: couldn't execute `llvm-config --prefix` (error: No such file or directory (os error 2))
warning: set the LLVM_CONFIG_PATH environment variable to a valid `llvm-config` executable
error: could not compile `rcl`
To learn more, run the command again with --verbose.
I get a similar, but not identical error message for the aarch64-unknown-linux-gnu architecture, which only has the u8 vs. i8 errors.
My compiler version is: rustc 1.49.0-nightly (3525087ad 2020-10-08). Any help or suggestions for tracking this problem down would be really appreciated!
Hi,
thanks for the kind words!
I suspect that we need to add some conditional compilation based on the target_arch in a few places. If there's not a ton of errors you could probably add them here and there to fix the problems above.
One of the hurdles here is also to cross compile ros because we link directly to the shared libs of rcl and the messages using the paths setup when sourcing ros. I think sourcing a cross-compiled ros installation should do the right thing, but its not something I've tested.
Do you have instructions for cross-compiling ros2 for armv7? I could try it out to see if I can make it work on a pi3 I have laying around.
I'd really appreciate that! As best I can tell, the cross-compilation instructions for ROS2 can be found at https://index.ros.org/doc/ros2/Tutorials/Cross-compilation/ , with some code for the cross-compilation in a repository here.
I'll take a look at adding those conditions compilation steps myself, but can't promise much success (at least, anything timely), as the library seems to use a lot of Rust code concepts that I haven't spent a lot of time working with (macros, large enums, etc.).
If there's anything else that I can to do support this (testing, bug reproduction, etc.), just let me know!
Thanks. I cross compiled ros2 for armv7 using the instructions you posted.
I have fixed the compilation problems you had above have in the experimental branch armv7. I did not manage to make it link in the end, so I don't know yet if it will actually work. But I thought I'd push it if you want to try.
Because we actually link to the message libraries at build time in order to get to the introspection functions (this is how we avoid the idl pipeline) to generate the rust code for the messages you need to do a bit of hacking to make it work. The steps you need to take are something like:
Source your (host) ros workspace on the host system and build r2r (no cross compilation) from the master branch. This will produce a directory in target/debug/r2r-somehash/out with the generated rust code for the message types. Copy this out directory somewhere.
Now open a fresh shell where you instead source your cross compiled ros workspace. Checkout the armv7 branch of r2r. I had to manually add include dirs for the cross compilation toolchain in build.rs, e.g. builder = builder.clang_arg("-I/usr/arm-linux-gnueabihf/include/");. You may need to change the path here, but given your intial post I guess you have this working already.
export OUT_DIR=/path/to/the/copied/out
cargo build --target=armv7-unknown-linux-gnueabihf
So you will use the rust code generated on the host system (so you need to have the same message types in your host ros2 ws as the cross compiled ws) to build the bindings for the cross compiled messages.
As I wrote above, for me this compiles but fails to link. I'm sure it's just because of my unfamiliarity with the cross compilation system, I just didn't find a way to set up the paths for the linker properly. Would be interested in knowing how to set it up.
/usr/lib/gcc-cross/arm-linux-gnueabihf/9/../../../../arm-linux-gnueabihf/bin/ld: warning: librosidl_typesupport_introspection_c.so, needed by /home/martin/cc_ws/ros2_ws/install/lib/librosgraph_msgs__rosidl_typesupport_introspection_c.so, not found (try using -rpath or -rpath-link)
Thanks for pushing that. I spent some time working on trying to get the armv7 branch to link yesterday, but also came up short. The rustc errors all dropped, though, so seems like a step in the right direction! I will probably put some time into looking a little more deeply into it over the next couple weeks to see if I'm missing something.
Thanks for trying it out. I managed to make it link by copying the output from cargo, i.e.
"arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" ....
and adding "-Wl,-rpath-link,/home/martin/cc_ws/ros2_ws/install/lib" to the begining of the command line. This feels like a hack but it seems to work for the moment. Maybe it's possible to get cargo to add that automatically, but I suspect its a gcc toolchain thing.
I tested with the "parameters" example. Running the resulting binary on the docker container I used for cross compilation I get
root@12f6f1056ce9:/ros2_ws# ./paramters
./paramters: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.28' not found (required by ./paramters)
Turns out my docker with ubuntu 18 has libc 2.27. I guess the next step is to figure out how to link against an older glibc :)
That makes sense. Do you think it's because ROS2 versioning is coming into play? If I remember correctly, I think that Eloquent is mostly tied to 18.04. At least just reading through the commit log, I was under the impression that r2r broke compatibility with Eloquent to match with Foxy, and (at least in my limited experience) Eloquent doesn't play especially well with 20.04 either, so maybe there's something happening on that side of things?
I don't think this particular dependency has to do with the ros2 version as I could build it (and run the examples) in the docker. I think it comes from the rust side. However I have not looked more into it since.
Should we try this out again in galactic or close the issue?
The current guidance (humble) for cross compiling ROS relies on running emulated builds using QEmu. Initial test running cargo in this same emulated environment with r2r appear to work also.
Cleaning up some old issues. Assuming this one to be ok now.