Blueprint: SDI/HDMI to HDMI/MIPI smart converter and streamer

The flexibility of FPGAs is an excellent match to any format conversion tasks, and at Antmicro we’ve been using FPGAs from various vendors to convert fast data streams between interfaces like MIPI CSI-2, HDMI, SDI, USB, PCIe and others. The increasing capabilities and decreasing cost of larger FPGAs means that new applications are becoming feasible in markets which previously could not afford to use an FPGA, and use cases that previously seemed to be reserved to large server rooms or heavyweight PCI-based industrial systems can be converted to compact, low-power and low-cost embedded devices.

The Internet is full of devices which at least nominally convert almost any input to any output, a majority of such data conversion and streaming solutions are however proprietary and come with their own closed bitstreams (often called firmware), binary drivers (even if there is a concept of Linux support, this is done with binary kernel modules) and little documentation. It’s extremely hard - or downright impossible - to integrate such devices in any product that does not exactly match whatever application the original vendor had in mind, let alone guarantee long-term availability and upgrade paths for other use cases. That is why at Antmicro we take an entirely different approach and help our customers create freely programmable and well-documented hardware, firmware and software for all types of data conversion.

Concept: smart conversion/streaming device

This document presents a concept of a universal conversion device like many similar ones we had built for our own and our customers’ use cases.

A smart SDI and HDMI to HDMI, MIPI or Ethernet converter / streamer, this device uses the freely programmable on-board FPGA to enable video format conversion, live processing, encoding and streaming. The video stream coming from an SDI or HDMI source can be processed with minimal latency and then output as HDMI - directly onto a display, or MIPI CSI-2 - for further processing on an embedded device.

Linux and Zephyr capabilities

The processing FPGA runs a compact, open source soft SoC system - LiteX - with a RISC-V CPU that is both Linux and RTOS capable, offering incredible flexibility. By default, the device uses a compact Zephyr RTOS configuration, a natural choice since Antmicro is a member of the Zephyr project and maintains the Zephyr LiteX port.

In the Linux configuration, it is very simple to build on top of standard Linux software for an embedded web control server, live analysis and reporting software. The built-in RISC-V core can be easily extended with custom accelerators, and the FPGA can be used for entirely custom processing blocks, such as codecs. For software development purposes, the processing system is possible to simulate in Antmicro’s open source Renode framework, which allows for software/FPGA co-simulation.

The device makes it possible to perform live processing on the video signal, including overlays, color processing and AI tasks such as object detection/classification. It provides an Ethernet port for control, debugging and - potentially - encoding and streaming data over the network.

Integration with more complex systems

The converter can be used in standalone form, but its MIPI CSI-2 also allows to connect it to a variety of Antmicro’s MIPI-capable hardware, such as our open source Jetson Nano / Jetson Xavier NX board, or the Apalis iMX8/TK1 Smart Vision kit, creating a complete edge AI video processing platform with SDI and HDMI camera input capabilities.

Antmicro provides both FPGA and OS/driver development services, host software development and customized hardware services to help you build your custom project similar to the one described above. If you are interested in building your next project with Antmicro, give a shout at contact@antmicro.com.