Introducing a new flow for gathering Zephyr coverage data with Renode and Coverview

In its 10 years, there is little doubt that Zephyr has become the go-to open source RTOS for modern embedded systems, supporting a diverse and constantly growing range of architectures, SoCs, and boards. With this widespread adoption, thorough testing of Zephyr on target hardware must be seen as critical. In that context, code coverage analysis serves as useful metrics for identifying which parts of the embedded software are covered by tests, giving visibility into any potential “blind spots” that may need additional validation.

While Zephyr directly provides gcov-based code coverage for the native target along with some generic simulated boards, it covers only a handful of hardware platforms on which you can run the RTOS. This is where Renode, Antmicro’s open source simulation framework, comes into play - with its extensive support for Zephyr-based platforms featured in the Renode Zephyr Dashboard and code coverage reporting built on top of its advanced execution tracing features, taking away the concerns of RAM limitations in embedded hardware since everything happens in a fully deterministic simulated environment. Thanks to Renode’s integration with another open source tool by Antmicro, Coverview, coverage results can also be visually represented as interactive dashboards.

Antmicro continues to improve Renode’s capabilities for better testing and development of Zephyr-based platforms, from extending the range of supported platforms and UI improvements in the Zephyr Dashboard to introducing a new Renode GUI for more convenient interaction with the framework.

In this note, we describe the newly introduced flow for gathering Zephyr coverage data with Renode across all of its supported hardware targets and architectures (RISC-V, Arm, x86, and more), which results in ~80K more total lines covered as compared to the existing gcov-based approach. We also present coverage metrics on a GitHub-hosted dashboard rendered with Coverview.

Gathering Zephyr coverage data in a simulated environment

Renode allows you to work with unmodified software, generating traces of execution for coverage reports similar to how it’s done via Zephyr’s gcov integration, but without the need to instrument an executed binary. The coverage analysis is performed at the line hit count level, gathering data about any executable code. Instructions that run during testing are written to Renode’s trace file, which is used for generating coverage reports - thus, the simulated application doesn’t hold any measurement data.

Renode supports a vast list of hardware targets running Zephyr, and you can easily enable it as a simulator and a test runner within Zephyr on a target board of your choice using our west-renode-gen tool.

Implemented as an external Zephyr module, west-renode-gen can be easily added to any custom project. It uses the dts2repl tool developed by Antmicro to automatically generate REPL platform definitions for Renode from Zephyr devicetree files, making it possible to use Zephyr’s west meta-tool for managing a simulated environment. West-renode-gen also uses the complementary info-process tool for merging multiple coverage reports into one so that they can be visualized via Coverview.

For coverage specifically, west-renode-gen provides patches for Zephyr that enable Renode as a coverage backend in Twister, integrating its execution tracing features with Zephyr’s standard testing mechanisms.

Extended coverage analysis across hardware targets and architectures

Renode supports a large variety of hardware targets and architectures running Zephyr, and with the newly introduced coverage flow, you can run coverage in an auto-generated simulation environment on all of these hardware targets and architectures: Arm, RISC-V, x86, and more.

For example, there’s a tangible improvement for RISC-V-based platforms since they are extensively supported in our simulation framework as well as for the zephyr/drivers directory, which remained largely uncovered until now. In comparison, Zephyr has very limited coverage analysis for RISC-V, though this architecture, with its increasing popularity in hardware design, greatly matches the openness, structuredness, and flexibility of the RTOS.

The modular, plug-and-play nature of Renode also offers a much simpler and faster way for measuring coverage on custom hardware targets, allowing you to gather data for a simulated system as it’s being developed. Contrary to the existing flow in Zephyr, where a compilation target must be prepared specifically for simulator support or rely on, Renode allows you to run coverage analysis for a digital copy of a hardware using any custom tests, without any technical limitations of the real embedded platform.

Representing interactive coverage data with Coverview

Coverview, due to its seamless integration with Renode, allows you to view coverage measurements from the UI, either using the deployed GitHub pages or by building locally as part of your CI setup. You can view coverage measurements line-by-line, compare them across boards, and aggregate multiple reports (generated both via Zephyr’s gcov integration and Renode) into a single output. For details on how to generate code coverage reports in Renode, see the documentation.

The universal, language-agnostic nature of Coverview and its easy adoption across CI/DC workflows also make it a natural choice for visualizing coverage measurements across multiple systems and use cases, as in the case of generating coverage dashboards for RTL projects that we did a while ago.

Aiming to provide a comprehensive, unified view of Zephyr coverage data

We have created a dashboard that features combined Zephyr coverage results and is deployed on GitHub Pages. This dashboard demonstrates how the attempt to run all twister tests in an auto-generated simulation environment is the first step towards the development of a single coverage dashboard - an undeniably useful tool for all Zephyr developers.

As visible from the represented data, over half of broad Zephyr test suites can be run with an auto-generated simulation environment. We are still working on broadening test sets that contribute to the dashboard, which will increase the coverage information even further, supplementing the original, limited set of data generated by the upstream Zephyr CI.

The introduced dashboard also clearly demonstrates that pursuing the improvement of auto-generated simulation platforms is a worthwhile effort. More than that - it can be used in tandem with the current Zephyr coverage mechanisms, providing a unified, comprehensive representation of the Zephyr RTOS coverage state.

There’s another noteworthy update - for this release, we have increased the number of line hits from 156,174 (342,212 in total) that’s currently covered by Zephyr gcov-based approach to 234,558 (528,506 in total) with our Renode-based coverage, ultimately achieving ~80K more covered lines!

Comprehensive simulation-based code coverage analysis with Renode

The new simulation-driven flow of gathering coverage data for Zephyr, which relies on Renode’s extensive tracing capabilities, offers a reliable, comprehensive, and easy-to-set up process of conducting a post-mortem code analysis for an embedded system, outputting all the collected data on an interactive Coverview dashboard.

By introducing the new Renode-based flow, we’re now able to run coverage for Zephyr RTOS across all architectures and hardware targets that are supported in Renode, obtaining accurate measurements of the current “as-is” state of code coverage in Zephyr. This also allows us to get more coverage data for such areas as the zephyr/drivers directory, which has been practically uncovered by the existing Zephyr’s gcov-based approach.

Renode also allows you to build, test, and customize any simulated multi-architecture system, working concurrently on both hardware and software stacks, with the whole process happening in a fully deterministic environment.

If you would like to integrate Renode or Coverview into your broader ecosystem of tools or further extend Renode’s code coverage analysis capabilities to a specific platform or architecture, contact us at contact@antmicro.com.