The popular edge AI NVIDIA Jetson Orin NX and Nano SoMs offer up to four PCIe controllers, sharing up to seven lanes. The 4-lane PCIE0 interface is usually used for the NVMe drive, leaving other peripherals three PCIe lanes, which can be configured as either three x1 PCIe interfaces or x1 + x2. As PCIe allows for interfacing with AI accelerators, NVMe, Ethernet adapters, cameras, and the like, additional PCIe ports can greatly improve the capabilities of NVIDIA Jetson Orin-based devices, for which Antmicro offers full-stack development services.
To increase the number of PCIe ports available from NVIDIA Jetson Orin SoMs, we developed an expansion board compatible with the open hardware Antmicro Baseboard for Jetson Orin, featuring a PCIe 3.0 packet switch. The switch enables connecting several PCIe endpoint devices to a single PCIe root complex.
In this article we describe the open hardware PCIe Switch Expansion Board for Antmicro Baseboard for NVIDIA Jetson Orin, and demonstrate its capabilities with a use case where two NVMe drives connect to the same PCIe interface of an NVIDIA Jetson Orin Nano, forming a simple RAID drive.
Open hardware PCIe Switch Expansion Board for Antmicro Baseboard for Jetson Orin
The board is designed as an 100 x 60 mm (3.9 x 2.4 inch) adapter, matching the Expansion Connector of the Antmicro Baseboard for Jetson Orin. The Expansion Connector exposes 2-lane PCIA/B/C form the NVIDIA Jetson Orin, which is split on the adapter with the PI7C9X3G606GPCFCAEX packet switch. The switch offers PCIe 3.0 support and several configurations of 6 PCIe lanes. For our purposes, we chose the 2-lane upstream, dual 2-lane downstream, i.e. splitting a single PCIe x2 interface from the host into two independent PCIe interfaces.
For the downstream PCIe interfaces, the board offers M.2 key-M slots, supporting both 2242 and 2280 form factors, and an OCuLink connector, which can be selected instead of one of the M.2 slots, in order to connect a PCIe device over cable. The selection can be made manually or via the NVIDIA Jetson Orin SoM GPIO.
For an interactive version of the diagram, visit the desktop version of the website. The PCIe Switch Expansion Board is powered directly from the Expansion Connector; the design also meets low-power requirements, making it suitable for embedded applications. The board design is available as Open Source Hardware - for the Expansion Board’s design files, a complete list of used components and additional assets, visit Antmicro’s Open Hardware Portal.
Testing the PCIe Switch Expansion Board in a RAID configuration
We tested the PCIe switch in a dual-SSD redundant data storage scenario. The setup consisted of the Antmicro Baseboard for Jetson Orin with the NVIDIA Jetson Orin Nano 4 GB SoM, the PCIe Switch Expansion module, and two Samsung 970 Evo NVMe drives. On the Jetson host, the drives were configured as a single RAID 1 drive (storage with data mirroring on both drives) and accessed with a single 2-lane PCIe gen. 3 interface. Data throughput was measured as the average net read/write speed of large ext4 files. Latency was measured with a custom benchmark, performing read/write operations on PCIe device registers.
Initially, we compared that to the performance of a single drive connected either directly to the PCIe interface or through the PCIe Switch, as seen in the diagram above (scenarios A and B, respectively). We observed similar data throughput of about 1/1.6 GB per second. Additionally, the PCIe switch introduced about 350 ns round-trip latency in the PCIe device access. The results were the same for both downstream channels, so the passive PCIe multiplexer (used for switching between M.2 and the OCuLink connector), which is present in one of the channels, had no impact on the performance in this scenario.
| Scenario | Single drive, direct (A) | Single drive, PCIe Switch (B) | RAID 1 configuration (C) |
|---|---|---|---|
| Write speed [MB/s] | 1023 | 1062 | 804 |
| Read speed [MB/s] | 1675 | 1669 | 1666 |
The write speed of the virtual RAID 1 volume dropped to 0.8 GB per second, as the RAID 1 driver had to write twice as much data to the drives, effectively creating a bottleneck on the common, host-side PCIe interface. The read speed was unaffected, since as long as the stored data are not corrupted, RAID 1 needs to read only one drive.
Bringing flexible and efficient PCIe connectivity to NVIDIA Jetson-based systems
Antmicro helps customers build complete products based on every member of the NVIDIA Jetson Orin and NVIDIA Jetson Thor families. The PCIe Switch concept demonstrated in this article illustrated just how scalable and flexible such systems can be, allowing for expanded PCIe connectivity without the need to redesign the baseboard. The compact, low-power PCIe Expansion Board is a perfect building block for developing embedded applications that require multiple PCIe devices connected simultaneously. It can be easily customized to meet specific requirements, for example by applying larger PCIe switch chips from the same family, thus enabling even more PCIe device interfaces to accommodate AI accelerators, further NVMe drives, camera adapters, and more.
To discuss your use case directly, you can visit antmicro.com/nvidia-jetson to learn more about Antmicro’s end-to-end engineering services for the NVIDIA Jetson ecosystem, or reach out at contact@antmicro.com.
