Generating interactive coverage dashboards with Coverview

Code coverage is a useful metric to keep track of while developing test suites, providing engineering teams with an actionable overview of how broad their testing goes. This is true both for executable code and digital design, but the different nature of RTL and software execution requires developers to look at different types of coverage information. On top of tracking executed code lines and visited branches, RTL coverage will also need to include information e.g. about specific bit field toggles of all the signals in the design. The thorough testing of a sufficient number of combinations of those signals can be the make or break of an RTL design.

In order to facilitate automated generation of interactive coverage dashboards for various internal and customer projects of growing complexity, Antmicro developed Coverview. Coverview is an open source tool that lets you aggregate RTL and other test execution results as a single page in a clear, unified form, with comparative analysis and introspection capabilities as well as the ability to load data incrementally. It enables digesting output from different sources, including RTL simulators, can be deployed as a static webpage and integrated with existing CI/CD pipelines.

In this article we provide an overview of Coverview’s features and illustrate it with an example of a dashboard generated for the VeeR EL2 core project we are maintaining for the Caliptra working group of CHIPS Alliance.

Versatile coverage reporting with Coverview

Coverview utilizes several sources of information: the LCOV (Linux Coverage Visualizer) format (.info files), the source of the project being analyzed as well as a custom .desc format, similar to .info, but containing information on which test covers a specific line. It aggregates multiple coverage types into a single-page, unified dashboard.

Coverview was built to cover a variety of use cases involving both public and private dashboards, in different CI and data storage setups. Thus, it can generate a self-contained dashboard with embedded data, or you can simply load a specially crafted .zip archive to load coverage data in runtime - for this purpose we released a Coverview version deployed on GitHub pages which you can start using right away.

This lets you either generate standalone dashboards that get deployed with your code - useful for simpler CI pipelines where you just upload everything into one specific place/bucket, or create more advanced setups where you can navigate different datasets by simply switching an URL parameter. A built-in multi-dataset capability with a dropdown also lets you compare multiple data sources in one.

Take a look at the project’s README for basic usage instructions.

We also developed a complementary info-process tool which simplifies the process of creating Coverview input archives, including aggregating multiple .info files into one.

Generating coverage dashboards for RTL projects

Coverview’s primary initial use case was for ASIC design - to enable actionable insights into verification status of open source RTL development in collaborative scenarios which CHIPS Alliance enables.

It can be used to process results from different RTL simulators, such as but not limited to the open source Verilator which we are making better for our customer’s use cases in a variety of contexts including verification.

It includes a burndown list functionality to show which parts of the design need more urgent attention, drill-down individual file views with expandable signal lists, as well as tooltip links to specific tests covering the analyzed object.

The feature set is developed generically to avoid overfitting to the specific ASIC use case, e.g. supported coverage types are configurable.

For RTL development projects, we now enable four types of coverage:

• line - shows which lines were executed
• branch - evaluates statements such as if-else or ternary operators in both true and false cases
• toggle - shows whether a signal is toggled between 0 and 1 and back in order to ensure that transitions of all bits in a signal were covered
• condition - summarizes coverage of all combinations of conditions for a given expression

Coverview is already deployed in several RTL projects - including the VeeR EL2 core mentioned earlier - and we are working on adapting it to cover other use cases such as software simulation using Renode which will be covered in a separate article.

Comprehensive testing and verification with Antmicro

Antmicro’s work often requires us to visualize and analyze RTL simulation and software execution results effectively, which puts us in the ideal position to develop productivity tooling of this kind. Thanks to its open source nature, Coverview can be easily extended and customized to match your specific use case, and you can seamlessly integrate it with your existing workflows, including in CI/CD.

Coverview is part of a broader ecosystem of open source tools and workflows that Antmicro develops to help our customers at every stage of the digital design process - from testing and verification, through debugging to benchmarking. If you would like to learn more about Antmicro’s code coverage analysis services, as well as our complementary verification and test planning tools, don’t hesitate to contact us at contact@antmicro.com.