Remote reporting and logging in Protoplaster for easier product debugging

When developing complete systems with custom hardware, operating systems, drivers and userspace software, it is essential to have a comprehensive testing framework to be able to quickly pinpoint whether specific issues are related to problems in the software stack, manufacturing issues with a particular hardware unit or the hardware design itself. At Antmicro, besides our own comprehensive simulation testing framework called Renode, we have also developed our own automated physical device testing framework, Protoplaster, which we use for our own needs, as well as in various customer projects. With the growing adoption of the framework, new features are being added to support new use cases, including the most recent addition of a reporting mode, which will be described in this note.

Bug reporting in Protoplaster

Our customers include multinational hyperscalers, industrial juggernauts and high-tech startups. For different reasons, all of them value our operational efficiency and complete traceability of both our software and hardware work, and with every manufactured prototype, we need to make sure we can quickly pinpoint any potential issues. While Protoplaster was already able to test specific hardware interfaces against concrete scenarios, for less obvious cases, we were looking for ways to speed up the troubleshooting process so that geographically distributed teams could more easily share information on what is happening. We achieved that by implementing the ability to generate real time snapshots of a system by means of a reporting mode named protoplaster-system-report, launched from within Protoplaster. It obtains current information about the system state and configuration by launching a list of commands, and saves the output in a single ZIP archive together with an HTML summary.

With each addition to a BSP, whether it is a new OS feature or a new userspace app, we can easily add a new, dedicated Protoplaster test along with a corresponding Protoplaster report entry, meaning that by examining detailed logs, we can easily identify where a bug or other failure occurred. The flexibility of the new report mode means that we can easily update it with any additional commands or logs that we feel might be useful when diagnosing problems, including commands that take into account specific operating conditions for a given customer.

When working on a customer project, we simultaneously ensure that Protoplaster is configured correctly for the purpose of providing system reports. As a result, when troubleshooting, we only need to request that the customer executes a single command on the device, which will generate the required output/logs. This allows us to investigate problems using specific and relevant data rather than having to sift through a large amount of data, most of which does not pertain to the problem at hand.

This approach also takes advantage of the fact that a part of a device’s configuration will be common across the vast majority of projects, meaning that it is only required to configure logging things that are specific to an individual project, such as dedicated applications used to control a given device. The modularity and expandability of Protoplaster and the new reporting mode makes it very easy to adapt the tool to each new project created at Antmicro.

Using protoplaster-system-report

Generating a full system report using Protoplaster is as simple as executing a single command, including the following parameters:

protoplaster-system-report [-h] [-o OUTPUT_FILE] [-c CONFIG] [--sudo]

The available options are:

• -h, --help - show help message and exit
• -o OUTPUT_FILE, --output-file OUTPUT_FILE - path to the output file
• -c CONFIG, --config CONFIG - path to the YAML config file
• --sudo - run as sudo, required when Protoplaster has been installed by a non-root user.

Alternatively, an input file can also be specified.

The YAML config contains a list of actions to perform. A single action is described in the following format:

report_item_name:
  run: script
  summary:
    - title: summary_title
      run: summary_script
  output: script_output_file
  superuser: required | preferred
  on-fail: …

A detailed description of each option in the configuration file can be found in the Protoplaster README, available on the Antmicro Open Source Portal. For the purpose of troubleshooting our projects, we provide YAML config files to our customers as part of our projects, customized for their specific needs and problems.

After running the protoplaster-system-report command, a report is generated, containing the required information for diagnostic purposes. An example of a typical system report is included below, containing current information about the operating system in use, USB devices, device drivers and network connections. As mentioned above, these reports can be highly customized to meet the precise requirements of the project.

Enhanced and expedited technical support for Antmicro’s products

With the latest addition of the reporting mode to Protoplaster, we are now able to render faster and more accurate technical support to our customers without requiring extensive manual intervention from their side. This has resulted in tangible improvements for customers, including faster response times and rapid deployment of fixes for their specific use cases. We can also develop testing suites for existing projects with the assistance of Protoplaster, which can be helpful for customers looking to upgrade an existing project to utilize new hardware.

If you wish to take advantage of our full project development services, including hardware and software engineering for your specific use case, do not hesitate to get in touch with us at contact@antmicro.com