Test-driven development of multi-node Zephyr + micro-ROS solutions with Renode

Antmicro is actively involved in developing advanced applications, which may involve multiple subsystems communicating with each other, variable device configurations and various communication protocols. To handle such solutions, we often use the ROS (Robot Operating System) framework. It allows the developer to wrap the application’s subsystems in separate programs, called nodes. Nodes can communicate with each other in a publisher-subscriber (one-to-one) and client-service (one-to-one) manner. It helps create complex applications in a lean and modular way.

Testing and deploying such modular and especially multi-node systems requires a reliable deployment and testing approach using Continuous Integration pipelines. In this note, we will describe how using our open source Renode framework can help you develop a micro-ROS solution running on top of Zephyr with a robust test-based workflow.

Transition to ROS 2 and micro-ROS

Last year we wrote several notes about practical uses of ROS at Antmicro, including a tracking and detection tester based on ROS 1. In ROS 1, nodes were communicating with each other via TCP connections, and there was a core node responsible for managing the communication and connection of nodes to the system.

ROS 2 is practically a reimplementation of ROS 1, introducing a fully distributed communication between nodes via Ethernet (UDP), UART and other interfaces. It does not have a core node anymore, which with replacement of TCP connection with UDP connection makes this framework more fitting for real-life scenarios, where the connection between nodes in a system can be lossy. Among many new features, the ROS 2 provides a library called micro-ROS, which is meant to introduce ROS features to microcontrollers.

Microcontrollers are omnipresent in robotic systems - they are often used for reading sensory data (temperature or touch sensors) and manipulating the effectors (robotic arms, servos, lights). Micro-ROS allows to implement nodes running in microcontrollers and communicating with other nodes in the system. The data is by default transferred using a UART connection, but you can implement communication over other protocols.

The multi-node capabilities of Renode, and the ability to simulate various connections modes and protocols, make it ideal for testing ROS 2 and micro-ROS based systems.

When developing systems consisting of multiple interconnected nodes running on application processors - running an OS like Linux - and MCUs - based on e.g. Zephyr - communicating with each other using various protocols, lots of things can go wrong. To ensure the system’s correctness and stability, you need tests verifying software running various nodes in separation, different features of the system as a whole, as well as corner cases. Such tests should be reproducible, so if any bug or error occurs, it can be fixed through debugging.

To address the above, we can use Renode to provide a consistent, hardware-less environment for fast development and testing of the systems involving many applications and machines of various types. Without the hardware in the loop, testing with Renode can be also easily deployed in Continuous Integration pipelines.

To showcase how this could be done in a practical environment, we created the Renode micro-ROS demo, a skeleton example for test-driven development of multi-node micro-ROS solutions with Zephyr.

Simulating micro-ROS platforms with Renode

The Renode / micro-ROS demo presented in this note allows us to demonstrate Renode capabilities in micro-ROS development, simulating a complex setup of micro-ROS nodes communicating with the host machine running ROS 2. It is designed to build the firmware and required environment components inside of CI. We will take a sample application, build it and then, using Renode, simulate a platform where said firmware can run and communicate via a simulated UART queue with the host device and its services.

Micro-ROS is a ROS 2 library meant for microcontrollers which usually runs under the control of an RTOS - most notably Zephyr (which we will focus on today), with FreeRTOS also supported. Using an RTOS like Zephyr for hardware abstraction allows the library to accommodate many more platforms, since most hardware-specific code would be handled by the OS. Then, micro-ROS can communicate with the system’s API and interact with it in a standardized way.

The ROS 2 library is compatible with desktop systems and can easily interact with micro-ROS applications,Micro-ROS works on the same principle as ROS 2 - it allows the user to create a topic and publish to it or subscribe to the messages being sent there. One notable difference is that ROS 2 is based on C++ and allows for mixing it with Python, while micro-ROS firmwares are written in C. ROS 2 provides an extensive environment in which the firmware is being built alongside scripts that automate the process to just a couple of commands and the final ROS 2 package is an executable or a Python script to be executed in the context of a broader system. The micro-ROS firmware on the other hand contains both the application and the RTOS in one binary that is meant to be flashed onto whatever device it targets.

In Renode we are able to simulate microcontrollers and sets of interconnected devices - this makes it a perfect development and testing environment for micro-ROS (and ROS 2) solutions. With hardware-in-the-loop testing, it’s hard to efficiently check various configurations and border conditions in an efficient way, especially in non-standard use cases prone to generate conflicts within the firmware. Renode however enables this by letting you implement and parametrize wide-ranging test suites which can be deterministically executed and distributed at scale. This in turn enables things like compatibility charts for different hardware and configurations for micro-ROS on Zephyr, leading to widespread adoption and improved quality and stability of the integration.

The Renode micro-ROS demo

The demo application itself consists of a simple publisher/subscriber pair - the micro-ROS firmware publishes messages to a ROS 2 topic managed by the micro_ros_agent package that can be received either by a second micro-ROS instance or a ROS 2 subscriber application. There is only one firmware type - it takes up both roles of sending information periodically and receiving messages if one arrives. It allows you to test simultaneous execution of the tasks further checking compatibility with Renode. The message type that is being sent is a Float64MultiArray, since it allows for multiple values to be sent at once. In our case, we send a 1x2 array, which contains a device ID used to identify the sender of the message and a message value containing whatever information is to be published.

The firmware has one Renode-specific piece of code that would not run on real hardware and that is reading a Python device address used for randomizing the device ID. It is memory-mapped inside of an unused address space and ensures that the device IDs will be truly random. It can be easily replaced with a rand() call for hardware deployments. The device ID is needed since it is possible to receive your own messages in this system and a simple check will discard them when the sender is known. What is worth noting is that this type of message was meant to be used within C++, which would contain a constructor and an allocator that will automatically adjust the memory allocation while filling up the data section. Since micro-ROS is based on C, this allocation needs to be done manually, which requires carefully studying the available manual to avoid reading outside of the allocated space. Execution is controlled in a way that messages are published periodically with the aid of a timer and it is interrupted when data is received from other sources.

Running the demo

The following step-by-step guide will run the demo without running the tests themselves, essentially just publishing messages from the micro-ROS environment into a ROS 2 topic on the host OS.

1. Clone the repository:

   git clone https://github.com/antmicro/renode-microros-demo.git
   cd renode-microros-demo
   

2. In the cloned directory, run the named Docker container based on the ros:galactic image:

   docker run --rm -v $(pwd):/data --name renode-microros-demo -it ros:galactic
   

3. In the Docker container, install the remaining dependencies:

   apt-get update
   apt-get install -y python3-pip wget
   

4. Get Renode in the Docker container - the most convenient way is to get the latest portable release, extract it and add the executable to the PATH variable:

   wget https://builds.renode.io/renode-latest.linux-portable.tar.gz
   tar -xvf renode-latest.linux-portable.tar.gz
   mv renode_* renode_portable
   export PATH=/renode_portable:$PATH
   

5. In the Docker container, go to the directory with the repository and build the solution:

   cd /data
   ./build.bash
   

6. In another terminal, connect to the running container using:

   docker exec -it renode-microros-demo /bin/bash
   

7. In the second container shell, run the ROS 2 instance fetching data from the simulated microcontroller:

   source /opt/ros/galactic/setup.sh
   cd /data
   ./run_ros2_communication_demo.bash
   

8. In the first container shell, run Renode simulation of the machine running micro-ROS publisher-subscriber:

   renode -e "s @./renode/first_instance.resc"
   

9. The ROS 2 application in the second container shell and the micro-ROS application running in the first container shell should establish a connection. Upon successful connection, we should observe in the micro-ROS application the following log:

   

   And in the ROS 2 application:

   

The observed logs demonstrate a working micro-ROS application running in Renode, communicating with the ROS 2 application running in the host system.

Improved testing with Renode

As shown above, this demo provides a vital insight into the compatibility of micro-ROS with both the STM32 platform and Renode simulations in general. Renode’s vast platform support will allow for quick testing on other devices and with the Zephyr Dashboard, it will be also possible to check the support for micro-ROS, micro-ROS features and micro-ROS-based applications across various boards. The main focus of the project as a testing platform was to provide reproducible, accessible and continuous checks to validate the capabilities of the system as a development platform. Furthermore, thanks to the extensive support for device networking in Renode, it will also serve as a starting point for potential future projects containing a multitude of interconnected devices.

Most micro-ROS applications are running under the control of an RTOS. Some deployments where micro-ROS would be a great fit to connect a device network would therefore have the capability to merge the micro-ROS architecture with other solutions already running under the control of those systems. One example would be machine learning on the edge; using TensorFlowLite Micro or uTVM would allow to make a distributed system where the local processing of the models could be forwarded to other nodes without gathering an enormous amount of data on a central processing server. Such a solution would only require a small device acting as a topic manager for all the micro-ROS instances. Aside from that, integration with ROS 2 would preserve the ability to process the data on central machines with orders of magnitude more processing power, if such a use case is required.

If you are interested in building an AI system with quick and automated testing capabilities provided by Renode, be sure to reach out to us at contact@antmicro.com and find out how we can help.