grabthecam and farshow - libraries for rapid video acquisition pipeline development

Antmicro’s work often revolves around developing computer vision applications running on the edge. It involves component selection, hardware design, creating custom camera drivers and developing end applications that use the camera pipeline in practice.
There, computer vision is used to process images incoming from the camera with both classical algorithms such as filtering, thresholding, image segmentation, feature matching, as well as deep neural networks for image classification, object detection, object tracking, or text detection and recognition.

Tailored solutions like these often require very low-level access to camera structures and configuration calls to address specific use cases. To get the most out of the embedded platform, as is often necessary in the cutting-edge products Antmicro helps build, access to frame data may need to be optimized using platform-specific memory management structures and calls that are not essential for more typical use cases.

When working between multiple levels of computer vision application development, we should be able to analyze and observe intermediate results of such processing in order to quickly and comfortably determine root causes for computer vision flow issues, e.g. undetected objects, misclassified images or empty images. Such errors can occur at any stage of video processing, from frame fetching (e.g. with exposure time set too low), through certain common operations (such as image normalization or thresholding) to neural network output parsing. While such errors may not be easy to find through code inspection or analyzing data structures containing the images, a proper visualization immediately uncovers the issues within a processing pipeline.

In this note, expanding the collection of Antmicro’s tools for working with video processing, we introduce new releases:

• grabthecam - a small and easily adjusted C++ library for video frame capture and camera configuration, and
• farshow - a lightweight C++ tool for streaming and displaying frames in real time for computer vision development purposes (and more).

These two tools can be used in tandem to simplify and automate your video pipeline development. In a typical workflow, grabthecam becomes a part of your application, which simplifies communication with the driver and captures frames, and farshow later enables you to fetch intermittent frames and view them remotely during development.

Interfacing cameras using grabthecam

While in most cases existing libraries, such as OpenCV, are very good for camera management and frame grabbing, for some tailored scenarios we often need to:

• Access custom ioctls to manipulate settings specific to a current camera,
• Access a camera file descriptor to manipulate a camera directly.,
• Trigger frame fetching manually, e.g. using a dedicated GPIO,
• Provide a dedicated place in memory where a driver should store the frame, e.g. to utilize platform-specific shared memory with an accelerator,
• Use platform-specific hardware acceleration for raw frame preprocessing, e.g. debayerization.

In order to address the above, Antmicro developed grabthecam, a configurable library utilizing the v4l2 API, which allows you to:

• Manipulate camera controls directly using existing (and newly added) ioctl calls instead of library-defined enums,
• List available camera settings,
• Read frames in raw format and process them using existing or newly implemented frame converters,
• Easily add new preprocessing functions for raw frames in order to support new pixel formats and bit widths,
• Customize the way the frames are stored in memory,
• Store and restore camera settings from JSON files (a format supported by Antmicro’s pyvidctrl TUI tool for camera parameter control).

The grabthecam library can be installed as follows:

git clone --recursive https://github.com/antmicro/grabthecam.git
cd grabthecam
mkdir build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j`nproc`
sudo make install

Follow the project’s README to check for necessary dependencies.

Later, you can easily add it as a dependency to an existing project using CMake’s find_package function:

# require grabthecam package
find_package(grabthecam REQUIRED)

# add include directory with grabthecam headers
include_directories(
    ${grabthecam_INCLUDE_DIRS}
    …
)

# add sample executable using grabthecam library
add_executable(example-app
	example.cpp
)
target_link_libraries(example-app PUBLIC grabthecam)

Then, you can create a grabthecam::CameraCapture object, e.g. for the /dev/video0 device:

#include <grabthecam/cameracapture.hpp>

...

grabthecam::CameraCapture camera("/dev/video0");

At this point, you can:

• set frame format:

  camera.setFormat(1920, 1080, V4L2_PIX_FMT_YYUV);
  

• print available camera settings:

  camera.printControls();
  

• obtain camera setting using V4L2 ioctl codes:

  int value;
  camera.get(V4L2_CID_BRIGHTNESS, value);
  

• set camera setting using V4L2 ioctl codes:

  camera.set(V4L2_CID_BRIGHTNESS, 128);
  

• save and load a current camera configuration:

  camera.saveConfig();
  ...
  camera.loadConfig();
  

• grab and read frames from a camera:

  std::shared_ptr<grabthecam::MMapBuffer> rawframe;
  camera.grab();
  camera.read(rawframe);
  

• create and use custom preprocessing classes for processing frames (i.e. demosaicing).

What is more, due to its lightweight nature, the library can be easily added as a submodule and adjusted for the particular needs of the application.

Introducing farshow

Computer vision application development and debugging involves inspecting computer vision pipelines on various steps. It is usually done by observing intermediate images created as a result of the algorithms mentioned in the first paragraph.

In the case of edge devices, it is common for the access to the device itself to be limited to an Ethernet or a Wi-Fi connection only, without the possibility for connecting a display.
With this limitation in mind, one possible way to is to save frames or videos within an application, upload them to a PC and analyze them there. This type of development is, however, quite slow and tedious.

Another possibility to observe the processing pipeline can be to create an SSH session with display forwarding or to use VNC or other tool to stream the display to a PC. This method also has drawbacks, especially for GUI-less devices, where installing additional software and libraries on the edge device would be needed just to enable the debugging flow.
In order to address this issue and introduce a lightweight, real-time preview tool for video processing pipelines, we have created the farshow library.

farshow is a streaming library consisting of a server and a client, where the server runs on an edge device. It is easily created using a simple class which then allows for providing frames ascribed to their “topics” - names that can describe relevant information, e.g. a given processing pipeline stage, the name of a variable, etc. The server then sends the provided frames along with their topics to the client, which displays the frames in windows with names corresponding to their topics:

In order to accelerate frame rendering and provide users with a real-time experience, the client uses OpenGL. The windows with frames are generated using the ImGui library. The data is sent using UDP in a compressed form in order to reduce data packet size.

The farshow library can be fetched and built as follows:

git clone --recursive https://github.com/antmicro/farshow.git
cd farshow
mkdir build && cd build
cmake ..
make -j`nproc`

Follow the project’s README to check for necessary dependencies.

The commands above should create a farshow executable, which is a client application receiving UDP frames.

The server application runs on the edge platform and is implemented by the user.

A sample server application built in farshow is named farshow-server-example.

To run a demo, first run the server application (in the build directory):

./farshow-server-example 0.0.0.0 -s <video-file>

Where <video-file> is an MP4 file or camera device, e.g.:

./farshow-server-example 0.0.0.0 -s /dev/video0

Then, run a client app displaying the received frames:

./farshow -i <server-ip>

Where <server-ip> is an IP address to the server. This shows frames as in the video example above.

When introducing farshow to your own project, you need to add it as a third-party dependency to your project using the add_subdirectory CMake function and link the farshow-connection library to targets utilizing its features, or install farshow in the system and find it using the find_package function in CMake, as described in grabthecam.

Later, include the farshow/framesender.hpp header and create a farshow::FrameSender object:

#include <farshow/framesender.hpp>
...
farshow::FrameSender streamer("0.0.0.0", 12345);

Where 0.0.0.0 is the server’s IP address, and 12345 is the port for the application.

After this, to send a cv::Mat frame under topic name topic_name, run:

streamer.sendFrame(frame, "topic_name");

With this, in the client’s farshow window, a topic_name view with the current frame appears.

Simplify and accelerate computer vision workflows with Antmicro’s tools

grabthecam and farshow are the newest additions to Antmicro’s growing repertoire of tools for work with v4l2 pipelines that aim at facilitating the development of computer vision solutions.

Recipes for grabthecam and farshow are available in our Yocto layer meta-antmicro and they can be easily added to embedded projects.

If you are interested in building tailored computer vision applications, edge AI systems, camera drivers and processing pipelines, or you would like to discuss your needs for commercially supported open source software and hardware solutions, feel free to contact us at contact@antmicro.com.