Zephyr RTOS and Cortex-R5 on Zynq UltraScale+

The UltraScale+, a high-performance FPGA SoC designed for heterogeneous processing with 4 Cortex-A53 cores and 2 Cortex-R5 cores, is often used in Antmicro’s projects. For certain complex devices, the combined processing capabilities of the US+ FPGA SoC’s heterogeneous cores are ideal - with the R5 cores used for real-time processing, the A53s for running Linux with non-critical software, and the FPGA used for dedicated accelerators for large amounts of data, such as high-resolution video. A good example is our fast 3D vision system, X-MINE, currently being deployed in several valuable mineral mines across Europe.

For such AMP applications however, only FreeRTOS and bare metal are available as options to be run on the R5 core by default. Coming from a software-oriented and standards-driven perspective, Antmicro likes to work with the Linux-Foundation backed, vendor-neutral and scalable Zephyr RTOS, of which we are a member - and so porting Zephyr to the US+ was an obvious choice.

Why is Zephyr a good choice

Dedicated for all but the most resource-constrained devices, Zephyr can target a variety of use cases in real-time applications with the US+’ Cortex-R.

Zephyr allows for easy handling of multiple configuration options, APIs and external components, and is well suited to structured application development. We’d worked with many AMP Linux+RTOS applications on various platforms, including ones executed in TEEs, which makes us especially sensitive to mixing programming styles and code architecture, which differ immensely between Zephyr and more traditional RTOS.

Another benefit of Zephyr is that it targets some very serious protocol and standard implementations, being e.g. the first open source RTOS to introduce TSN support - by way of Antmicro’s contribution. The rising popularity of TSN in automotive and aerospace applications, and just about everywhere else, could be a very important reason to start using Zephyr in your TSN-capable product.

The Zephyr port

Just recently initial support for Cortex-R has been introduced in Zephyr, providing basic context switching and interrupts, as well as adding a testing platform in simulation.

Antmicro’s contribution, released to GitHub today, introduces the support for a first real hardware platform. Our choice was the Enclustra Mercury XU1 system-on-module, which is often used by ourselves and our customers, encapsulating the complexity of the UltraScale+ MPSoC in an easily swappable module. Antmicro has a standard devkit based on this SoM which you could use to recreate this demo, but of course it should be possible to run it with minor tweaks on any Zynq UltraScale+ MPSoC device.

It also helps that Antmicro has been in charge of developing the entire OS-level software stack for all the FPGA SoC modules from Enclustra, basing on Buildroot/OpenEmbedded and our deep cross-area HW/SW/FPGA expertise which helped build a very easy to use interface for thousands of customers purchasing SoMs from this vendor. For simplicity, we will leverage this building block, called Enclustra Build Environment, in this note, although of course that’s not a strict necessity.

How to run a demo

Zephyr can be run on Cortex-R5 either from Linux running on Cortex-A53, or using a JTAG adapter. Here we will use Linux to have supervisor control (power on, load firmware, power off) over the remote processor.

Zephyr comes with a number of demos you can run on the Mercury XU1 SoM, and we’ll focus on the philosophers demo here. It implements a solution to the Dining Philosophers problem, which in computer science is considered a classic multi-thread synchronization problem.

Building the Linux environment for Cortex-A53

The setup requires the Enclustra Build Environment - a tool by Antmicro that enables easy and fast build of Linux with necessary bootloaders and firmware. It provides a simple ncurses-based GUI and command line interface to fetch and build U-Boot, Linux and a Buildroot-based root file system. At present, EBE supports 10 modules of the two SoC families: Zynq-7000 and Zynq-UltraScale+ including the Mercury XU1 module that we used. To get you started with EBE, refer to its online documentation.

In order to use the Cortex-R5, you have to load the firmware to the Tightly Coupled Memory of the processor. For this, a few additions in the original devicetree, that are related to the remoteproc device, are required.
The necessary devicetree parts can be found in this document (pages 15-16).

A ready to use devicetree file for Mercury XU1 module can be found in the Xilinx Linux repository.

Building the ‘philosophers’ demo in Zephyr

The application was built with Zephyr version 1.14.99 and ZephyrOS SDK version 0.10.2. The code is available on GitHub.

The philosophers demo can be built using the following bash commands:

Go to the zephyr repository:
cd zephyrproject/zephyr
Set up your build environment:
source zephyr-env.sh
Go to the location of the demo and build it:

cd samples/philosophers
mkdir build
cd build
cmake -DBOARD=mercury_xu_board ..
make

Starting the Zephyr app from Linux

The Zephyr app should be copied to /lib/firmware in the root filesystem
By default, the driver for the remote processor is compiled as a module. It can be loaded into the kernel using the following command:

# modprobe zynqmp_r5_remoteproc

Load and start the application for Cortex-R5:

# echo zephyr.elf > /sys/class/remoteproc/remoteproc0/firmware
# echo start > /sys/class/remoteproc/remoteproc0/state

The image below presents the result of running the application.

Future development

In the near future, We plan to enable Inter Processor Communication between Linux running on the A53 core and Zephyr on the R5 core. Both Zephyr and Linux support OpenAMP (Open Asymmetric Multi Processing) which is a platform that implements homogenous API for asymmetric multiprocessing.

Currently, Zephyr has only one OpenAMP demo. It targets the LPC54114 SoC which features a Cortex-M4 core. Adding a demo with Zephyr running on R5 and Linux on A53 would be the very first of this kind, so it’s definitely a worthwhile endeavor.

Benefits of AMP on US+

Asymmetric multiprocessing (AMP) can be really useful to get the best of both worlds, allowing you to get predictable, real-time responses where they matter, while keeping the ease-of-use and richness of a standard Linux OS. We have built many FPGA and regular SoC based Linux devices which benefited from e.g. running Web-based control servers and GUIs on Cortex-A cores, while keeping a critical functionality running on another CPU core (be it Cortex-M, A or R) with an RTOS. And in terms of the programming experience, Zephyr is a good match for the Linux you’ll be running on the main application core.

If you want to develop a complex application for Xilinx’s Zynq UltraScale+ MPSoC, and could use HW-SW co-design capabilities of Antmicro, whether it is designing dedicated PCBs, creating well-structured and modern FPGA code and/or integrating this with Linux or Zephyr - or both, don’t hesitate to contact us at contact@antmicro.com.