Automatic CPU testing with Embench

The already high numbers of IoT devices shipped worldwide are expected to reach staggering levels in the nearest future. Yet, for many years, there wasn’t a high quality, fully reliable and widely reported benchmark for embedded computers that would accurately test CPU performance of IoT devices. With the emergence of new and open architectures like RISC-V however, the concept of a new, robust benchmark framework was born.

Filling the benchmark gap

Embench, maintained by the FOSSi Foundation we proudly participate in, is an open source embedded benchmark test suite which aims to respond to the needs of modern embedded systems. It is free (as in both “beer” and “freedom” - you can get it from GitHub), easily portable, simple to use and provides a set of real programs, as opposed to the synthetic workloads offered by some of its proprietary counterparts. This allows it to accurately reflect how the tested CPU is going to behave once implemented in hardware. Being a suite rather than a single program, it allows the user to do tests with a complex, realistic workload and is easy to develop to accommodate the changes in compilers and hardware.

May the best one win

At Antmicro we are constantly looking for opportunities to broaden our structured understanding of the tech landscape and streamline various aspects of our work, developing open source ecosystems in the process, especially those related to FPGA and ASIC implementations of open source ISAs. This approach translates into the ability to pick the best solutions for a given customer project requiring a specific implementation of an open source SoC system, where alternatives abound. Here, we’ve used Embench to compare CPUs in multiple LiteX-based SoCs in an automatic CI testing flow. We achieved that by creating a test suite consisting of Embench tests that runs software in a simulation environment, which is much more convenient than having to manually run multiple designs on hardware, as tests are generated automatically on a server in Continuous Integration.

Thanks to this work, we can now quickly compare several CPUs in various specific contexts before choosing the most suitable one for any given project.

How it works

Using Embench we can now automatically test CPUs in LiteX SoCs by creating CPU simulation targets, compiling a test suite and executing it in the simulation environment, while the obtained results are saved as CSV or pre-formatted human-readable reStructuredText (rst) tables, which are then put together and presented on a website. At the first stage of the flow, the simulated hardware and software is compiled. Next, the script collects information about the SoC’s core, such as architecture, capabilities, compiler suite, as well as environment information (core version, toolchain version etc.). As the third step, benchmarks are compiled with a dedicated linker script that is common for all tests and architectures. Then, simulations are run and log files with benchmark results are created. Benchmarks are run twice, first to get relative performance in comparison to the Embench baseline Arm Cortex-M4, and then once more to get absolute results. At the next stage, JSON data is extracted from logs and is stored in JSON files. Finally, two data tables are created - one with relative core performance, the other with absolute performance results. The process described above is carried out for every tested CPU.

You can get the underlying code from our GitHub repository.

Flexible FPGA SoC development services based on LiteX

LiteX is our go-to soft SoC generator and library that we use to build a wide range of FPGA-based systems powering advanced devices operating in various areas of industry. It is open source, based on Migen Python DSL and provides the common components of a SoC that we are constantly developing and expanding, enabling our clients to easily kick-off FPGA development projects with us.

LiteX supports a wide array of FPGA platforms and offers a range of I/O options, such as the UART, SPI or I²C, as well as Ethernet, PCIe, USB and SATA for larger systems. With LiteX ports in Zephyr, Linux and our open source simulation framework Renode - all three developed by Antmicro - this versatile Core/SoC builder gives us a lot of leverage in delivering specialized solutions to our clients. Having a means to compare various CPU implementations that can be used with LiteX allows us to choose the best solution for a given application, and Embench, with the recently added CI testing capability, is an invaluable tool for that.