Support cross compilation to aarch64

[henriksod]

Solving the issue discussed here: #144

I also implemented a feature to cross compile towards aarch64
See #150 for more info.

• Patch cyclonedds to remove -flto
• Added gcc_toolchain and openssl cross compile support for aarch64

[ahans]

This is nothing that we strictly need in the global WORKSPACE file, as it is only used to test cross compilation. I'd prefer adding that to the examples workspace or even create a new WORKSPACE for this.

@mvukov What do you think?

[henriksod]

Hi! I suggest it is run through CI anyways, to ensure compilation towards aarch64 doesn't break.

[ahans]

Yes, I'm all for running it in CI, just setup a bit differently. We already have two jobs, one for the main workspace and one for examples. Would be easy to add a third. To trigger aarch64 compilation you need an extra job anyway.

[mvukov]

Before I started working on this repo, I was working on rules_ros. And there a holy goal for me was to be able to easily cross-compile a deployment (launch specs + nodes). Cross-compiling of the C++ code was fairly easy, but, then there was Python code. Back then I was mostly interested in C++ nodes (pretty much now as well), thus, on the Python side only for launch mechanisms. The problem was that some of the Python deps had compiled extensions. Python ecosystem is wild and I didn't want to invest much of my time figuring out all small details. TL;DR I got rid of some packages I didn't need in roslaunch and found python-only alternatives to replace packages with compiled extensions. Still hacky here and there, but this worked. (I didn't even bother configuring appropriate Python toolchain back then...)

In the meantime, arm64 dev stations became a commodity, e.g. https://www.ipi.wiki/products/ampere-altra-developer-platform / https://www.siliconhighwaydirect.com/product-p/945-13730-0055-000.htm -- at least for a company setting. Therefore, I am questioning cross-compilation these days. For hobbyists it still makes sense, naturally :)

On a high level: yay 💯 for this PR, but we should also think about how far we want to go with this. Do you want to work on figuring out how to make Python code working on arm64 as well? I am fine if we scope this that we're interested at the moment in Python packages only necessary for launching nodes: e.g. that top-level ros2_launch targets can be correctly cross-compiled and packaged for exec on an arm64 platform.

For this to work, we'll need a transitive closure for arm64 made from requirements.txt. pip-compile cannot resolve packages for another platform, btw. There exists something like https://pypi.org/project/pip-compile-cross-platform/, that I think uses poetry package under the hood. Then, let's say we have requirements_lock_amd64.txt and requirements_lock_arm64.txt. We'd need to figure out a convenient way to consume pip deps w/o forcing users to copy/paste selects in the build files.

Regarding CI. I think we want to have a workflow where we cross-compile e.g. a test target on a amd64 host for arm64 target and then in another job run that test on arm64 host to see whether it actually works. We can't do that with github actions at the moment, but we could do that on circleci. At the moment I think that would be an overkill to do for every PR, but once a week sounds good.

Regarding the lto patch: would this be necessary if we would natively compile code on an arm64 platform? I suspect not. If we wouldn't need that for native compilation, then it should be possible to tweak cross-compilation.

[henriksod]

On a high level: yay 💯 for this PR, but we should also think about how far we want to go with this. Do you want to work on figuring out how to make Python code working on arm64 as well? I am fine if we scope this that we're interested at the moment in Python packages only necessary for launching nodes: e.g. that top-level ros2_launch targets can be correctly cross-compiled and packaged for exec on an arm64 platform.

Hi! I am actually able to cross compile ros2_launch targets completely fine towards aarch64. Maybe there have been updates to rules_python since last time. Currently I have other struggles that I need to sort out. Seems like micro-ros-agent requires rmw_fastrtps_cpp and does not support rmw_fastrtps_dynamic_cpp as is currently implemented in #148. I need to either add support for that, rewrite the micro-ros-agent entirely to support introspection, or switch to Zenoh.

drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_catkin_pkg/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_docutils/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_empy/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_lark_parser/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_netifaces/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_numpy/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_packaging/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_pyparsing/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_python_dateutil/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_pyyaml/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_setuptools/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_six/
drwxr-xr-x  3 pi pi 4096 Jan  1  2000 rules_ros2_pip_deps_types_pkg_resources/
drwxr-xr-x  6 pi pi 4096 Jan  1  2000 rules_ros2_python_aarch64-unknown-linux-gnu/

pi@balancingrobot:/opt/balancing_robot $ ./launch_package
[INFO] [launch]: All log files can be found below /home/pi/.ros/log/2023-08-01-23-22-55-133995-balancingrobot-139566
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [micro_ros_agent-1]: process started with pid [139570]
[INFO] [micro_ros_listener_example_task-2]: process started with pid [139572]
[micro_ros_agent-1] [1690924975.863841] info     | TermiosAgentLinux.cpp | init                     | running...             | fd: 3
[micro_ros_agent-1] [1690924975.865139] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 4
[micro_ros_listener_example_task-2] [INFO] [1690924976.579878127] [micro_ros_listener_example_task]: I am alive

And as a side note, my requirement is to compile the application on my workstation and then deploy to a raspberry pi. This is to speed up compilation time.

[mvukov]

What does the output of ls rules_ros2_pip_deps_netifaces/site-packages look like? Do you get arm64 .so binary out of the box?

[mvukov]

Seems like micro-ros-agent requires rmw_fastrtps_cpp and does not support rmw_fastrtps_dynamic_cpp as is currently implemented in #148. I need to either add support for that, rewrite the micro-ros-agent entirely to support introspection, or switch to Zenoh.

IMO, it still makes sense to finish up #148 since you already made quite some progress. Off the top of my head, a rewrite of micro-ros-agent sounds more complicated than supporting rmw_fastrtps_cpp in this repo. :)

[mvukov]

So, I'm not sure Python stuff would just work out of the box. That's why I asked ⬆️

What does the output of ls rules_ros2_pip_deps_netifaces/site-packages look like? Do you get arm64 .so binary out of the box?

Also, I had a question:

Regarding the lto patch: would this be necessary if we would natively compile code on an arm64 platform? I suspect not. If we wouldn't need that for native compilation, then it should be possible to tweak cross-compilation.

Please reply when you find some time.

aspect_gcc_toolchain has some huge deps (e.g. a rootfs) and I would rather set up an example in a separate workspace. Next to that, would be great if we can test whether the cross-compilation works. I made a comment that we could try to set this up on circleci, eventually.

[ahans]

Regarding the lto patch: would this be necessary if we would natively compile code on an arm64 platform? I suspect not. If we wouldn't need that for native compilation, then it should be possible to tweak cross-compilation.

I have tried this out a couple weeks ago. It did work just fine. So the -flto -fPIC incompatibility seems to only apply to cross compilation. Most likely explanation to me is still a GCC bug.

At any rate, IMHO this issue is only remotely related to rules_ros2, so a FAQ entry to document that this issue is known and how to work around it would be good enough. An extra workspace that shows the workaround in detail (patching CycloneDDS's CMakeLists.txt or selecting RelWithDebInfo) would of course be even nicer than just the FAQ entry.

[mvukov]

Hm, does this looks like a gcc bug at a specific gcc version?