Warp Pipe: library for simulation-driven development of Zephyr and Linux-based PCIe devices

At Antmicro, part of our customer work relates to creating and improving open source tooling and utilities for developing hardware, often in simulation.

The flexibility and high throughput of PCIe made it a foundation element of datacenter systems, and to keep up with the growing demand for computation in compute-heavy applications like video transcoding or AI, hyperscalers like Meta and Google, as well as silicon industry behemoths like Intel, and up-and-coming startups like Tenstorrent - are developing PCIe-based accelerator cards with customized silicon on board.

While developing a PCIe device, in order to meet specific performance requirements, simulation-based tooling can enable software teams to quickly prototype and optimize efficient communication channels using bulk memory transfers and other useful optimizations while waiting for silicon to arrive.

This is why Meta, in cooperation with Antmicro, developed Warp Pipe a library for simulation-driven development and linking of PCIe devices allowing designers of PCIe peripherals to view the detailed flow of packets on the memory bus, and to optimize their designs accordingly.

Node-to-node communication through simulated PCIe links

The Warp Pipe library is a Swiss Army knife for developing PCIe-based systems with numerous use cases, including the ability to work directly with host applications by mocking PCIe devices. The library enables node-to-node communication between two operating systems, creating PCIe links between instances of different simulators/emulators such as Renode or QEMU, running operating systems such as Zephyr or Linux while also handling Message Signaled Interrupts (MSIs), packet routing and flow control, with Warp Pipe taking care of communication between the instances. The structure of a typical Warp Pipe flow can be seen below:

One of the primary design objectives with Warp Pipe was to let it tightly integrate with Zephyr but keep it software agnostic and usable with other operating systems such as Linux, the main use case being testing-driven development of Zephyr-powered devices, such as high-throughput PCIe accelerators. Additionally Warp Pipe connects to simulation tools for intercepting PCIe traffic, and part of the initial release includes a WireShark dissector that allows users to observe and analyze PCIe traffic on the simulated bus.

Simulating, Testing and Analyzing PCIe traffic with Warp Pipe

Warp Pipe provides users with the ability to simulate PCIe traffic on the level of packet transactions within an instruction set simulator. The library also makes it possible to create complex PCIe networks within a distributed simulated environment, including the possibility of packetizing and depacketizing simulated PCIe transactions into two different types of packets, specifically PCIe Data Link-Layer Packets (DLLPs) and PCIe Transaction Layer Packets (TLPs), included as part of the PCIe standard. An additional benefit is that users can manipulate PCIe traffic, such as by performing out of order transactions, dropping packets and injecting various errors, which is useful when checking how a device behaves when confronted with real life scenarios.

As Warp Pipe is cross-architecture and cross-OS, it enables use cases such as automated testing of APIs and protocols over PCIe. One endpoint runs in a simulated environment and the other in a custom application on the host machine, executing the desired test scenario.

The library is also integrated with Twister, the Zephyr RTOS testing suite, allowing users to easily run Zephyr-based test scenarios, such as the one displayed below that illustrates using the Ztest framework with Warp Pipe to send and receive data from separate processes.

To provide memory emulation, we also created the memory-mock application which simulates simple memory as a PCIe device. As an endpoint, it can be used with several tests, providing a useful demonstration of how the Warp Pipe API can be used in practice, such as with the PCIe DMA test, which performs DMA transactions between the Zephyr app (acting as a host) and the memory-mock device, or with the PCIe Native test, which presents communication between memory-mock and a Zephyr app compiled for the native_sim target.

Other ready-to-use testing examples are also available, including PCIe Pipe, which connects two Zephyr instances, one working as a server (PCIe device), one as a client (PCIe host) and exchanges data between them and handles MSI-X packets.

Warp Pipe can also be used for the purpose of verifying performance, as shown in the graphic below which illustrates a typical initial bottleneck analysis through the “Sandwich” view, suggesting CRC32 as a potential target for optimization.

In future, Warp Pipe will include PCIe controller models implementing routing and arbitration algorithms, hardware control flow like device resets and Virtual PCIe device support. Warp Pipe is also planned to be integrated with our Pipeline Manager visual system diagramming and design framework, speeding up design of complex PCIe networks and enabling the ability to monitor communication between devices.

Custom built simulation tools and hardware devices

The Warp Pipe project helps with rapid prototyping of PCIe-based devices; most firmware development can thus be carried out pre-silicon using reliable emulation and simulation tools. By providing a bridge between simulated devices and systems, the Warp Pipe library allows more of the development process to be shifted into simulation, to enable quicker development, testing and debugging of PCIe links.

By developing new open source libraries such as Warp Pipe and combining them with established tools like Wireshark or Renode, Antmicro can help you accelerate your development for creating new hardware devices, especially in ML-focused and security-first scenarios. To learn more about the transition towards simulation-driven development of PCIe devices, you can also watch our recent talk about the Warp Pipe library at the 2024 Embedded Open Source Summit, or reach out to us at contact@antmicro.com.