An open source SystemVerilog Test Suite
Published:
Topics: Open FPGA, Open source tools, Open ASICs
At Antmicro, we work with improving development flows for both software, hardware, FPGA and ASIC design, for which we use, contribute to and produce a wide variety of open source tools.
Verilog, SystemVerilog and open tooling
For FPGA development flows, our work has gravitated towards Verilog which has two advantages over its major alternative, VHDL: much better open source tooling support as well as new and exciting generators/paradigms like Chisel (used mostly in the RISC-V ecosystem, for both FPGAs and ASICs), SpinalHDL (a Chisel derivative optimized for FPGA) or (n)migen (gaining rapid popularity in the open source FPGA space due to its origin in Python), which allow you to build hardware in Scala or Python and ultimately generate Verilog.
When you look at the broader picture, Verilog has one more advantage - it’s bigger brother (technically, a superset), SystemVerilog, is widely used in the ASIC industry. The challenge is that SystemVerilog support in open source tooling typically used in FPGA development is not so great - most of the support has been implemented opportunistically and based on a limited set of use cases.
When we talk of Verilog-capable tools, we mostly mean parsers, simulators, synthesis tools, but ultimately also linters, IDEs and other higher-order tooling which could only benefit from supporting SystemVerilog as well.
Today different open source tools would usually ship with their own Verilog preprocessor and parser, based on their needs and use cases, and history. The result is that SystemVerilog support is at best partial, and at worst - non-existent, which makes those tools barely usable to ASIC designers. There is no common infrastructure, and no common benchmark to compare against.
First step to a radical improvement
The most commonly mentioned single ecosystem change that would radically improve the perspective for open source tooling and a more software-driven ASIC design flow is significantly improving SystemVerilog support for tools like Verilator or Yosys (this view is shared amongst most in e.g. CHIPS Alliance where we lead the tooling and marketing efforts).
But the first question to ask is - what does it even mean to support SystemVerilog?
The obvious place to look for an answer is of course the SystemVerilog standard (IEEE 1800-2017) where the language constructs are painstakingly described. Testing against the spec is useful, and a good source for many “unit tests”, but it’s the real-world usage that ultimately matters to developers, so other ideas for test cases for SystemVerilog compatibility came from the RISC-V Foundation’s cores and SoCs list and FOSSI’s librecores.
Based on that input, and with the aim to test open source’s best known Verilog tooling for FPGA and ASIC designs, we created sv-tests, a new addition to the SymbiFlow project family, a common infrastructure for testing SystemVerilog support in Verilog tooling.
An open source test suite - for testing open source, against open source
The purpose of the SystemVerilog Test Suite project (sv-tests for short) is to pinpoint all the supported and missing SystemVerilog features in various Verilog/SystemVerilog tools.
That is achieved by running a large number of tests over each of the tools examined.
The test suite introduces three types of tests:
- simple test-cases testing individual features as per the SystemVerilog standard (IEEE 1800-2017)
- existing third party test suites imported from external test suites or from the tools in question
- selected open source IP cores, like SweRV, Ibex and others.
The tests are organized using tags - each feature from the standard gets its own tag. Similarly, each external test suite or core is represented by a separate tag.
At the moment, the tools under testing include:
The outcome of this test suite is a regularly updated table showing the results each of the tools got for each tag. Clicking on any cell of the report table shows a window with some detailed information about the selected test case (including its source code) and its execution on the selected tool. All of the results are summarized at the bottom of the table, where we can see the total tests passed as well as the total number of tags for which all of the tests passed.
Next to the regular summary, some performance metrics can be seen.
These are the total time the tool needed to run all the tests as well as the maximum amount of memory the tool used.
The test suite has been very well received, with numerous fix suggestions, feature propositions and pull requests.
Some of the creators of the tools we are testing are already using it to improve their tools - which is of course the best imaginable result, and exactly the aim of the suite.
Summary
The sv-tests work was recently presented at ORCONF in Bordeaux and the CHIPS DV workshop in Munich, and is part of a wider, ongoing effort to show how hardware - both in the FPGA and ASIC sense - can be built in a more open, software-driven way. As part of both the RISC-V Foundation and CHIPS Alliance, we believe that the dynamics are there to make a leap forward towards a more varied, innovative and stronger hardware ecosystem using open source.
Next on the roadmap is finding ways to better coordinate between tools, and making the table as green as we possibly can!
If you are interested in adopting more open flows for your FPGA or ASIC design, reach out to us at contact@antmicro.com. We offer custom engineering services, open source tools and IP support and integration for real, radically innovative designs and products.
