Renode-based CI for TensorFlow Lite MCU

Renode is an open source simulation framework enabling effortless and collaborative software development of embedded and IoT devices in a deterministic environment. A wide toolbox of features for debug, trace and experimentation make it extremely useful in interactive code development scenarios, but the full potential of the tool is revealed in a Continuous Integration environment. With Antmicro’s help, a number of organizations are currently adopting Renode to radically improve their testing and CI situation for developing their products.

One such group is Google’s TensorFlow Lite for Microcontrollers team, which turned to Renode to run and reliably test machine learning models targeting microcontroller-based TinyML applications in a CI environment.

Accelerating TinyML development with Renode CI

Our collaboration with Google around TF Lite and Renode dates back to 2018 and, apart from our current wide-ranging rework of TF Lite Micro’s CI and enhancing its demo capabilities, has involved many joint projects, such as helping the to team create the initial TF Lite Micro port for RISC-V or running their popular ML library on a soft RISC-V CPU in a popular low-cost FPGA board, both on real hardware, virtually and in CI.

The current collaborative project between Google and Antmicro answers several needs that have emerged in the TF Lite for MCUs ecosystem with the increasing adoption of the framework.

The most challenging aspect of developing TinyML - as Google’s Pete Warden calls this field in the eponymous book - is the relative complexity of the development environment which typically involves physical boards spanning many computer architectures, with various input sensors running diverse minified ML models and a plethora of RTOS software.

This complex ecosystem calls for a novel, software-driven approach and a robust development flow, with advanced test, debug and trace capabilities as well as a high degree of observability and control over the development process. Most embedded CI flows are limited to checking if your code builds or not. Renode, on the other hand, offers a lot more extensive CI experience, enabling you to measure software metrics to detect anomalies early, monitor how your development is progressing and see how the software interacts with the platform it is being developed for. And whatever features are available in CI - determinism, custom sensor data input, detailed logging, state saving and loading, event hooks etc. - can also be used in an interactive environment to nail down specific problems in day-to-day development as well as for demonstration and education purposes.

Ensuring the fitness of TensorFlow Lite Micro

At an early stage of our CI-related work, Google’s TF Lite for MCU developers ran only synthetic tests in their Renode-based CI. These tests verify if the ML algorithms produce the expected results when fed with predefined data sets as input, which was already a huge improvement over relying on real HW testing, which does not scale so well.

This was not, however, all that Renode had to offer; the open source simulation framework supports complete platforms with complete I/O. And ML applications running on MCUs usually have to be provided with data from sensors: accelerometers, cameras, microphones and others. For a framework like TensorFlow Lite Micro, which aims to support a wide variety of hardware platforms, this means integrating with various third-party Hardware Abstraction Layers, with incompatible APIs and processing flows - which opens a large number of potential vectors for various hard-to-trace bugs.

In order to enable the full scope of TF Lite for MCU, on top of supporting typical inputs such as temperature, humidity sensors or accelerometers we worked with Google to add video and voice data input capability into Renode and implemented CI setups verifying real-life scenarios featuring sensors.

Our main focus was the very popular Arduino Nano 33 BLE Sense platform based on Nordic’s nrf52840, as well as a the FPGA-oriented LiteX SoC with VexRiscv CPU (implementing the RISC-V ISA), running TensorFlow Lite for MCU in Zephyr RTOS.
Such complex scenarios are virtually impossible to test regularly without proper automation, and Renode allows the TF Lite MCU team to ensure that each and every commit works as expected.

For the purpose of these tests we created dedicated repositories: one for Arduino and one for LiteX running Zephyr - which have become the prototype implementations for the TF Lite MCU continuous integrations using GitHub Actions as well as good, self-contained demo repositories in themselves.

To make the it compatible with the remaining part of TF Lite’s CI infrastructure, the results of the tests running in Renode are uploaded with our distant-bes-client library to Google’s Results Store back-end.

TinyML in 5 minutes

The wide-ranging CI work was only a starting point for further collaboration around making TinyML more widely available, understood and approachable. The above-mentioned repos, apart from serving our CI purposes, also contain a number of examples and demos, such as speech detection, person detection, and gesture recognition to give new users a quick and easy way to get started with TF Lite Micro. These interactive notebooks are runnable without installing any software, completely on Google’s Colab cloud servers, and give you a taste of how you can use virtual development for building TinyML applications without need for hardware thanks to Renode.

As another offshoot of the project, also aimed at bringing TinyML to a wider audience, we’ve also integrated our open source simulator with the Arduino IDE, which makes it possible to upload and run software on the Nano 33 BLE Sense directly from the IDE, while further work is being done to enable more interactive demos, visualization and other exciting features.

Platforms support and integrations

Renode supports multiple TinyML boards based on multiple ISAs, like the sensor-packed, Arm-based Nano 33 BLE Sense, the RISC-V Kendryte’s K210 or QuickLogic’s tiny QuickFeather FPGA SoC. Also, the TensorFlow repository includes Robot-based Renode tests for several other supported platforms, some of which have been supplied by external contributors, such as Arm. We are continuously working on adding support for more boards from vendors interested in widening the adoption of their hardware through Renode, so watch this space to find out when the platform you are eyeing becomes available in our open source development framework.

Much more than just CI

Renode’s wide ranging functionalities feature events recording, allowing the user to replay them to get the same, deterministic execution with user input; state saving, which can be used to save the state of the whole emulation into one file that can be shared with your team to solve problems collaboratively; as well as extensive and customizable logging capabilities, such as fine-tuned filtering depending on the log source and target, a built-in graphical log analyzer, and support of various data sources, e.g. executed software, peripherals analysis, framework events and user-defined events. Other options that are interesting in the TinyML development context include FPGA/hardware/software co-simulation as well as wired and wireless connection simulation. What’s not to love?

Whether you want your organization to adopt a Renode-based CI testing and development flow, or are interested in us developing support for your hardware in Renode, get in touch at contact@antmicro.com to discuss how our open source framework can help you achieve your goals.