Viewing and sharing CI build results with the distant-bes client

The ability to conveniently measure the quality and performance metrics associated with heterogeneous codebases spanning many edge and cloud software and hardware components are key in today’s collaborative and distributed development ecosystem. The more convenient and intuitive the testing and benchmarking process, the easier it is to pinpoint issues early and tackle them without the need to overhaul the system at a later stage.

What’s essential in ensuring quality is a Continuous Integration-driven testing and development flow with a clear and well-structured dashboard that allows the collaborating teams to keep track of their development process. Various CI systems have their own back-ends for displaying results, however they are usually tightly-coupled with their internal runner provisioning and job orchestration mechanisms.

Today, hybrid cloud infrastructure setups consisting of dedicated on-premise servers and scalable compute cloud services are becoming increasingly popular, and we help our customers adopt the right mix of internal resources and on-demand cloud processing for their development workflows.

Also the nature of the codebases we work with, which predominantly consist of open source software but may include proprietary components - call for a way to enable viewing and sharing test results without exposing the entire CI infrastructure.

To solve this, we have developed an easy-to-use Python library for pushing test results of any CI to any back-end that implements the BES protocol, originally created for Google’s Bazel buildsystem, and integrated it with an open source build result viewer, as described below.

Encapsulating complexity with BuildBuddy

At Antmicro we use open source CI systems such as GitLab or Jenkins for recording and aggregating test results from various projects that we’re involved in, including our open source simulation framework Renode. Despite those CI frameworks being excellent tools in their own right, they can’t easily be used to provide an independent back-end for reporting and viewing results since they are tied to a lot of internal infrastructure (and obviously not intended to display results produced by one another). However, in the course of our internal testing work as well as collaborations with Google, Arm, QuickLogic, Microchip and other Renode users, the need to expose the results to external parties emerged and prompted us to turn to BuildBuddy - an open source implementation of the BES protocol used by Google’s Bazel build system. Its open source licensing allowed us to create what we’ve called distant-bes - a portable BES protocol implementation which can compose Bazel-like events using simple Pythonic methods and inject them into a service implementing Build Event Protocol, e.g. BuildBuddy.

One of the servers that we are making available to the public and at the same time a very compelling use case is the public results of Renode integration tests. Renode is an interesting challenge, as it’s not only a framework that you need to test as such but also a testing and development tool for a plethora of embedded systems, simulating many computer architectures such as Arm, POWER, RISC-V and SPARC that can run an unlimited amount of guest firmwares/software. The tests include software as diverse as Tock OS, Linux, Zephyr, mbed, TF Lite, FreeRTOS, RIOT, Android, Contiki, eCos, micropython, Wolfboot and more. You can access the Renode test results from Github and the Renode testing/demo dashboard.

System overview

At the center of distant-bes, there is a multiplexing service that various clients can contact in order to send build metadata. The service then publishes the results to user-specified BES servers. Under the hood, distant-bes uses gRPC as the application layer, which is gaining popularity among various companies and projects as a framework for communication between microservices.

Test results structure & artifacts

Each invocation consists of a series of events, with each event packed as a single protobuf message. Such messages are streamed to the back-end in real time during the build process. The BES protocol specifies various types of events corresponding to different steps of a build process, e.g. compiling a target, generating artifacts, spitting out messages to standard output and so on.

Normally, such messages are generated in Bazel to represent its internal state and it was a natural step to devise a way to expose them and present them in a human-readable form. Additionally, the protocol turned out to be flexible enough that our library, distant-bes, is able to generate such events to represent the state of arbitrary builds, not related to Bazel itself.

Artifacts are uploaded to a content-addressable storage service and are linked in an event describing the artifact, mapping it to a target that generated said artifact.

The files themselves are transmitted over gRPC as well, packed in a ByteStream message.

Antmicro’s cloud services and activity

This is not the first time we have implemented a piece of the Bazel ecosystem as separate software, having already developed the Distant Remote Execution Client. In fact, our cloud-oriented activity and services span a whole range of technologies, as we help our customers create connected systems of field deployed devices interfacing seamlessly to public and on-premise cloud infrastructure, and scale into production.

We use open source to provide a range of cloud solutions and services, such as CI infrastructure for automated testing with our Renode framework and cloud-based, scalable, heterogeneous computing resources (e.g. with FPGA or Arm accelerators). We also build over-the-air update and remote management systems as well as GPU-based AI pipelines and cloud-driven FPGA & ASIC development flows.

If you would like to enhance your workflow with modern cloud-based solutions, do not hesitate to send us a message.