Introducing code coverage reporting in Renode

One of the key metrics helping ensure code quality is test coverage, providing objective, automatic ways of making sure that all of the most important branches of the code are verified. Antmicro’s open source Renode simulation framework already offers advanced Python-driven scripting and automation as well as extensive execution tracing features, and the recent addition of code coverage analysis provides developers with even more control and better understanding of their software.

In this note, we introduce the newly developed code coverage solution in Renode, built on top of extensive execution tracing features available in our simulation framework. Like most coverage reporting tools, it is run post-mortem and uses precise information about actually executed fragments of code to generate a report with the number of executions for each line.

Execution tracing in Renode

The most common method for code coverage analysis has the compiler add additional calls to the coverage infrastructure, meaning the code you’re running to get the statistics is not the same as the code you’ll be running in production. These modifications not only affect execution timing but also require you to allocate buffers for this additional data (which might be problematic especially on resource-constrained devices) as well as gather data from the host over some sort of connection, which might be complicated in case of embedded systems. For these reasons, when adding code coverage support in Renode, we leveraged the framework’s capabilities to run tests on uninstrumented binaries.

With Renode, you can generate very detailed traces of execution, as described in detail in a previous blog note, including function name and peripheral access logging, opcode counting and guest application profiling. The gathered information can be analyzed with speedscope and Perfetto, and visualized using our Metrics Analyzer. Interactive flamegraphs of execution traces of software in time for thousands of binaries across hundreds of boards, pregenerated by our CI, can also be found in our Renodepedia database.

But most importantly for the code coverage use case, Renode allows you to track all the executed CPU instructions, using the cpu CreateExecutionTracing command. With the availability of DWARF data (debug information available in ELF files, which are typically used to load software to Renode) and access to the source code, we can connect this trace to specific lines of code.

Code coverage reporting in Renode

We created a custom tool that can take a generated trace and add counters for each visited line in a given file. The main script, execution_tracer_reader.py, combines data in the DWARF format with the number of executions of each instruction counted by Renode. dwarf.py acts as a wrapper for the elftools library that allows us to read crucial DWARF data from ELF files, i.e. mapping between instruction PCs (Program Counters) and source code lines.

Below you can see a demo consisting of the following stages:

• a shell script that downloads files (main.c and main.elf),
• commands that create a machine, set execution tracing and execute main.elf,
• a command used to call the script that generates the coverage report,
• part of the coverage report.

For detailed information on generating a code coverage report in Renode, refer to the documentation.

Ensuring test completeness with Renode

The code coverage reporting tool, developed as part of the EU-funded TRISTAN project which focuses on open and reusable IP and tooling for RISC-V SoC development, together with other quality control features available in Renode, can prove especially useful in industries such as automotive and space. With the recent addition of pyrenode3, Renode offers improved scripting and automation capabilities, making it a more extendable and versatile tool for systems development and debugging, both pre- and post-silicon.

Reach out to us at contact@antmicro.com if you’re interested in using and expanding Renode’s code coverage analysis capabilities or integrating Renode into your workflow.