Fuzzing Zephyr with AFL and Renode

Fuzzing is an automated testing technique aimed at detecting problems like crashes or memory leaks in software by feeding it with invalid, often random input. It is especially valuable in safety-critical use cases, e.g. in the medical or automotive industries.

Recently, we used this technique to find, report and help fix several bugs in Zephyr’s Bluetooth stack using the RTOS’s integration with the libFuzzer tool, which shows the practical value of that method and having an RTOS that comes with a rich tool ecosystem.

Independently, as part of the aSSIsT project (Secure Software for the Internet of Things) led by the Uppsala University and RI.SE and funded by the Swedish Foundation for Strategic Research (SSF), Antmicro has added Renode support for Google’s open source fuzzing tool called american fuzzy lop (AFL) and its more evolved fork - AFL++. As part of the project, Uppsala University originally used AFL to perform fuzzing on the Contiki-NG RTOS. The work yielded some interesting results and Antmicro’s involvement was meant to use Renode to add the capability to perform fuzzing that is software-agnostic, on layers closer to hardware.

Using Renode to simulate real-world, unmodified binaries means that the technique can be applied independent of the software payload, which meant we could use Zephyr as our test subject. This work adds another item to Renode’s Zephyr testing toolkit which already includes Robot and Twister integrations, as well as the CMock/Unity module. Below, we describe the basics of fuzzing in simulation, how AFL++ integrates and communicates with Renode and provide a quick demo.

Fuzzing in simulation

Thanks to the way Renode is built, the integration with AFL++ is possible without changes to the simulator itself by leveraging hooks exposed by Renode and through the framework’s Python integration.

Fuzzing is typically performed on a specific application layer (e.g. network stack) and in order to do it, you need to prepare your software payload to make it possible to interact with a fragment of it. Renode, however, operates on a premise that you run unmodified software - just like what you would normally use with actual hardware. The integration aims at leveraging this, along with other features brought to the table by Renode, to test entire systems, including the driver layer. While the demonstrated approach is not necessarily “better” in any way, it is complementary to typical subsystem fuzzing. Renode also mimics real hardware by providing a way to input data directly to a physical interface. This can be done conveniently because of Renode’s extendibility, letting you create helper functions to trigger specific conditions and events on targeted peripherals.

The functionality of a fuzzer can be described simply as generating streams of bytes. It is then up to the user to decide how to interpret and feed the byte stream to whatever part of a system they are testing. In the section below, we provide a demo, where this input data is treated as simple UART input, but in other scenarios, we could assume this data as having an overlying structure, e.g. first byte is the length of a network packet and subsequent bytes are the data.

The simulator then runs the system using dedicated hooks for each executed block of instructions. The hooks provide feedback to AFL by filling a “coverage map” stored in shared memory - an unordered set of visited locations. The user then decides what to interpret as an exit condition. In most cases, a default of jumps to address 0 is interpreted as a failure exit (bug found). The fuzzer then inspects the feedback from Renode and decides what to mutate in its generated byte array. Additionally, we have implemented initial support for optional bus fault generation that can be enabled by the user to indicate an error.

Fuzzing Zephyr on UART demo

To illustrate the communication between AFL++ and Renode, we prepared a demo which you can replicate locally by following the CI mechanism defined in the AFL++ fork from Antmicro’s GitHub, and the GitHub actions script defined there.

The CI script describes three main steps:

• building AFL++
• building a Zephyr sample to test
• running AFL++ with Renode.

You can follow the instructions below to run the setup on your Linux host (this has been tested on Debian).

Building AFL

Building AFL++ is relatively straightforward. You need to clone the repository and run make:

git clone https://github.com/antmicro/AFLplusplus.git
cd AFLplusplus
make -j $(nproc)

For a description of AFL++ compilation options please refer to the project’s documentation.

Building Zephyr

Antmicro’s AFL++ fork comes with a pre-compiled echo Zephyr sample targeting the efr32mg_sltb004a board but you can also build it locally which will give you the benefit of easier debugging.

To do that, you need to install Zephyr SDK and west, Zephyr’s meta-tool used for repository management, building, flashing etc. To learn how to get these tools on your system, follow Zephyr’s Getting Started Guide or replicate the instructions from our CI workflow. Please note that for full replicability we use a specific version of Zephyr and its SDK.

Once you have the prerequisites installed and Zephyr cloned, you need to apply the patch that will effectively introduce a bug in the samples/subsys/console/echo sample:

cd zephyrproject/zephyr
patch -p1 < ${PATH_TO_AFLPLUSPLUS}/renode_mode/echo_failure.diff

The echo demo is quite simple - it prints out the characters it reads on the serial. With the added patch, whenever it encounters the letter a, it jumps to 0x0, crashing the application.

To build the demo, use west and place the binary in the AFL++ directory:

west build -p -b efr32mg_sltb004a samples/subsys/console/echo
cp build/zephyr/zephyr.elf ${PATH_TO_AFLPLUSPLUS}/zephyr-echo.elf

Running in Renode

Getting Renode is very easy - you can either use one of our prebuilt packages from builds.renode.io (we recommend using the “linux-portable” package) or use our Python helper renode-run:

pip3 install --upgrade --user git+https://github.com/antmicro/renode-run
renode-run download
alias renode=renode-run

With Renode in place, you can start the fuzzing demo:
Start by preparing the sample test case:

cd ${PATH_TO_AFLPLUSPLUS}
mkdir INPUTS
echo “bcdefxyz” > INPUTS/sample-testcase

And start AFL++:

AFL_SKIP_CPUFREQ=1 AFL_SKIP_BIN_CHECK=1 ./afl-fuzz -D -t 15000 -i INPUTS -o OUTPUTS -R -- renode_mode/example-uart.resc

With this command, you should see an output similar to the one below:

Renode - AFL++ communication

We started the fuzzing process providing the renode_mode/example-uart.resc script name as the parameter for afl-fuzz. The script looks as follows:

py "import afl_uart"

i $ORIGIN/efr32mg_sltb004a.resc
emulation SetGlobalQuantum "0.0001"
emulation SetGlobalAdvanceImmediately true
s

This script is a typical Renode script that does the following:

• Loads the afl_uart module
• Loads a generic EFR32MG script, that in turn loads the platform description and the Zephyr binary
• Sets Renode execution configuration parameters
• Starts the simulation.

AFL++ provides Renode with seemingly random data and how this data is interpreted depends on the user’s decision. For the purpose of this demo, we decided to treat the incoming bytes as data transferred via a serial port. This is handled by the afl_uart.py module loaded in the script above.

This module defines a single function, named quantum_hook, which is called every time Renode’s virtual time counter reaches the “quantum” value and handles reading out data provided by AFL and feeds it to Renode:

WriteChar = mach["sysbus.usart0"].WriteChar
if len(visited) < IDLE_COUNT:
    n = read(INFD, data, DATA_SIZE)
    for byte in bytearray(data.raw[:n]):
        WriteChar(byte)

To prepare your own scenario, you can replace the quantum_hook implementation with your own logic. It is up to you if you interpret the AFL data as serial characters, BLE packets or memory content - Renode will provide you with the API to access all peripherals in the system. Depending on the complexity of your workflow, you will also want to change the SetGlobalQuantum value used in the script. This value represents the virtual time that elapses between the iterations during which we expect all data processing to finish (in the example above, it is 0.0001 of a virtual second).

The infrastructure of the integration between AFL++ and Renode is implemented in the afl_renode.py script. You can tailor it to match your specific requirements, but you can also use it as-is.

The entire process begins with the start_fuzzing function:

def start_fuzzing():
    monitor.Machine.LocalTimeSource.SinksReportedHook += do_quantum_hook
    for cpu in monitor.Machine[sysbus_name].GetCPUs():
        cpu.SetHookAtBlockBegin(log_basic_block)

    do_one_fuzz()

It is responsible for setting up hooks to events in Renode: do_quantum_hook is executed every quantum and log_basic_block is executed on each code block. The first one will ensure that your quantum_hook function gets called, and the second one is responsible for informing AFL++ about the trace of the execution, thus guiding the generation of input data.

In its current implementation, afl_renode.py resets the emulation only in the event of an application failure, which means that the scenario assumes each subsequent input not to be influenced by previous results. Should you want to ensure that your application starts from scratch after each test (and you’re willing to sacrifice some performance), simply remove this condition in one_fuzz_complete:

def one_fuzz_complete(status):
    [...]

    # always reset? add `or True` to the condition, as below
    if status != STATUS_SUCCESS or True:
        monitor.Machine.Reset()
        reset = True
    do_one_fuzz()

Fuzzing in Renode for more comprehensive development and testing

The integration opens up the door to interesting fuzzing scenarios that have not been possible earlier, all for the sake of improving code quality and catching tricky but possibly dangerous bugs.

With Renode’s extensive testing capabilities, vast experience in the Zephyr project and customizability enabled by open source workflows, Antmicro can help you co-develop comprehensive, thoroughly tested hardware/software solutions with ZephyrRTOS or other operating systems like Linux or Android.

If you have any questions about the solutions we can offer and develop for your project, reach out to us at contact@antmicro.com.