Building advanced testing suites for Zephyr devices with Twister, Renode and Robot Framework

Creating real-world test cases for a project plays a key role in releasing bug-free and secure software, especially for mission-critical embedded systems. The Zephyr RTOS, which Antmicro often uses (and ports) for building real-world industrial devices for its customers, features many tools to facilitate better testing, including Twister, a dedicated test runner which can orchestrate testing on hardware and in simulation.

In this note, we will describe how the recently added integration between Twister and Renode, our open source simulator, through Robot Framework, a Python-based test automation and robotic process automation (RPA) tool, can help you build complex, real-world test suites for embedded devices which have to operate faultlessly in the field for extended periods of time. The interactive, keyword-driven workflow of Robot and Renode’s ability to freely scale the simulation of advanced devices without limits works great with the test orchestration capabilities of Twister, which is familiar to Zephyr developers.

Extending the Zephyr target testing workflow

Many of Antmicro’s projects revolve around testing, from multi-node robotics systems with e.g. Zephyr/micro-ROS and Linux nodes, through testing Video4Linux2 applications, to unit testing with the CMock/Unity module in Renode.

A multitude of projects and perspectives naturally sees Antmicro use a variety of tools and paradigms. But in Zephyr, the most common way to perform on-target testing and test case automation is with the Zephyr Test Runner, also known as Twister, which is responsible for finding and building test cases, executing them, and collecting test results.

While Twister could already run basic Renode tests for some time, there are certain use cases, like our customer’s project, where you need more interactive tests to better replicate real user input and more complex tests scenarios. Our goal was to give the user the ability to develop advanced test suites that can be executed in simulation to help address real-world problems. The integration we developed allows you to do this while staying within the familiar infrastructure of the Zephyr project so that you can adopt Renode and Robot Framework in a way that’s fully compatible with all previously created tests.

Robot Framework for building complex test suites

Robot scripts consist of keywords, which are custom, reusable sets of commands that Robot can execute each time it runs. They can be further combined into tasks to execute a group of keywords for more complex test suites. Robot also allows you to define variable blocks and helps you define constants that are assigned values at runtime and can be used throughout the test case for more interactive, conditional testing and better replication of user input. Renode can execute Zephyr samples like any other binary, and the scripts can be used alongside Robot to write tests and run them in simulation.
The following is a sample Zephyr test implemented in Robot using Renode:

*** Settings ***
Resource                      ${KEYWORDS}

*** Test Cases ***
Should Read Version From Shell
    Prepare Machine
    Start Emulation
    Wait For Prompt On Uart   uart:~$
    Write Line To Uart        version
    Wait For Line On Uart     Zephyr version

This simple test runs a sample with the Zephyr shell and provides input over the UART, expecting a proper response. It is possible to provide other types of input including e.g., synchronized multi-sensor information, button presses, or even packets on a network interface.

The current implementation of Robot support in Twister uses the renode-test tool, a Renode-specific convenience wrapper that handles running Renode and establishing communication with Robot. However, it is also possible to replace the keyword implementation and provide a custom solution for use with physical hardware or other tools.

Using Robot with Twister on your local machine

Testing your Zephyr-based software with Robot in Twister works the same as with any other testing tool. To run our sample hello_world test on the hifive1 target platform, use the following command:

./scripts/twister --platform hifive1 --test samples/hello_world/sample.basic.helloworld.robot -v

You can also run this test suite outside Twister using:

ROBOT_FILES=sample.robot west build -p -b hifive1 -s samples/hello_world -t run_renode_test

You can use wildcards for ROBOT_FILES to select multiple Robot tests at once (e.g., ROBOT_FILES=shell-*.robot ...).

After each run of a test suite, Robot will store all the logs in the build directory and additionally generate a report, which can help you trace the exact execution of your tests in an easy-to-read, visual form. When started with Twister, these results will also populate the Twister test report.

Build reactive and accurate test suites for embedded systems with Antmicro and Renode

When developing embedded devices that need high reliability, you can take advantage of Antmicro’s experience in developing and integrating custom, scalable test tools to add Zephyr support for new hardware, implement Zephyr drivers, libraries or applications. We can help you improve your existing test workflow or work with you to build a new one from the ground up to ensure it meets all your requirements and allows you to fully test your product in realistic scenarios.

If you are interested in using Renode and its integrations for complex, end-to-end testing in a fully repeatable environment, contact Antmicro at contact@antmicro.com.