Protoplaster - an automated hardware and BSP testing framework

Antmicro’s Linux-based embedded device development projects often contain many moving parts, including custom hardware (often in multiple revisions), kernels with custom driver configurations, secure boot and OTA update systems as well as user space / application software and AI models whose complexity grows along with the size of a project.

All these elements require testing, and in initial stages of a project, before more hardware revisions, features and use cases appear, it could seem that it’s fine to do this by hand, with all components and interfaces verified manually one by one, and likely re-tested after a new software feature is added.

This may work well at first or for projects of lower complexity, but tends to become an issue when the scale of a project grows. The attempts to mitigate the difficulties of manual testing often include introducing checklists and procedures to ensure thoroughness and repeatability, as well as end-to-end testing that covers the hardware and software use cases in a given project.

Ultimately however, in the fast-paced and collaborative projects that Antmicro is running, where complex software and hardware are developed in parallel between multiple teams, there is no time for chasing ephemeral bugs somewhere at the hardware-software divide which manual testing, however stringent, always misses. This need for a universal, repeatable and efficient methodology that can be replicated both on Antmicro’s and the customer’s end led to developing Protoplaster, an automated open source Python framework for platform testing.

Protoplaster’s basic functionality

Antmicro’s goal when developing Protoplaster was to introduce a highly customizable and flexible, permissively open source tool that would allow users to properly and thoroughly test and verify all of the moving parts mentioned above, beginning with the earliest stages of a project.
Protoplaster does this by providing a basic set of tests that are common (or at least similar) for various platforms that run Linux-based Board Support Packages (BSPs). These tests are meant to be run on the device itself, and can be easily adapted to a specific platform, and different groups of tests for different purposes are described as input in a .yaml file. An example configuration is presented below:

---
base:                   # A group specifier
  i2c:                  # A module specifier
  - bus: 1              # Multiple instances of tests can be specified in one module
    addresses: [ 0x3c ] # Every test instance should contain parameters needed to initialize the test
  - bus: 2
    addresses: [ 0x50, 0x30 ]
  camera:
  - device: "/dev/video0"
    camera_name: "Camera name"
    driver_name: "Driver name"
  - device: "/dev/video2"
    camera_name: "Camera2 name"
    driver_name: "Driver2 name"
    save_file: "frame.raw"
additional:
  gpio:
  - number: 20
    value: 1

There are three types of test modules specified in this configuration: I2C, camera and GPIO. A test module is a set of tests that checks the functionality of a specific piece of hardware or software. The I2C module tests whether we can detect devices at specific addresses on a specified bus. The camera module verifies if the device was detected correctly, checking the name of the camera, the name of the driver and whether it is possible or not to get a frame from that camera. The user can also specify a file where the frame is going to be saved (save_file). The GPIO module checks if it is possible to read and write from a specified GPIO. In the example above, two groups of tests are specified: base and additional. These groups define a separate set of tests that can be run either together or separately. Groups are especially useful in situations where different tests need to be run in different intervals, like stress tests.

Use cases

At a higher level, we can distinguish several test groups based on use cases:

• Base tests, re-run frequently in various scenarios
• Bringup tests, run during hardware bringup
• Stress tests, e.g. long-running tests to verify hardware heating issues in enclosures
• Software feature tests, run after each change in the feature set to verify proper functioning of older features

The primary overarching scenario in which Protoplaster is currently used is with early stage hardware handovers , allowing Antmicro’s customers to replicate a suite of tests on their side after hardware is delivered to them. The test results, in case of potential problems encountered by the customer either for a specific prototype or a batch, then enable us to locate the root cause of an issue and address it much more easily than in the case of a typical back-and-forth associated with a less normalized testing process. What is more, the framework allows us to continue collaborative development of software and running tests on both sides further down the road.

In order to make the tester easier to use with the variety of Antmicro’s open source hardware boards and their derivatives developed for customers, we have open sourced Protoplaster OpenEmbedded recipes in the meta-antmicro repository described in more detail in a recent blog note. The tester can also generate parts of project documentation, thus ensuring that testing reports are up to date. The generated documentation describes the subject as well as methodology of the tests performed, which helps establish a mutual understanding in the scope of required and provided functionality between Antmicro and the customer.

Extendability

In addition to the basic tests shipping with Protoplaster, it is possible to easily add modules that can extend testing functionalities. These external modules can include specific tests for a custom platform, or add tests that were not included in the base implementation. A new module needs to contain a test file named test.py, in which all of the tests are ascribed to a class decorated with ModuleName(“”) from protoplaster.conf.module. This decorator enables Protoplaster to recognize the name of a given module, so that test parameters can be dynamically injected from a .yaml file into a test class as its attributes, e.g. self.attribute. The name of a test class should start with Test and the names of individual tests should start with test. A new module can then be included in the .yaml tests definition, with an additional parameter __path__, specifying a path to that module. Below you can see an example of an extended module test and its .yaml definition.

from protoplaster.conf.module import ModuleName

@ModuleName("additional_camera")
class TestAdditionalCamera:
    def test_exists(self):
    	assert self.path == "/dev/video0"

---
base:
  additional_camera:
    __path__: "/home/user/tests/additional_camera"
    tests:
    - path: "/dev/video0"
    - path: "/dev/video1"

Improve your platform testing with Protoplaster

The examples presented above show how Protoplaster can be implemented for automating certain aspects of work with custom hardware, and so, for simplifying platform-wide testing, especially in the cases of development board preparation, BSPs and adding new software features to an existing platform. In the future, Antmicro is planning to continuously add further tests to include additional interfaces and modules, e.g. CAN or FPGA, as well as various stress tests.

If you are interested in automated platform testing or looking for assistance with efficient development of software-driven solutions in the areas of ASIC, FPGA, please reach out to us at contact@antmicro.com, and we will gladly use Protoplaster and our other open source tools to help you get the most out of your platform.