Jetson Image Processing¶

(click images to enlarge; source: Nvidia)

Jetson image processing for camera applications¶

Jetson Ecosystem is a unique platform from NVIDIA.
This is essentially a mini PC with extremely powerful and versatile hardware.
Apart from ARM processor, it has a sophisticated high performance GPU with CUDA cores, Tensor cores (on Xavier) and includes software for CPU/GPU and AI.
Below you can see an example of a possible way how to build a camera system using Jetson.
This is an important task to create realtime solutions for mobile, portable imaging applications.
Through thoughtful design, one can even implement a multicamera system on just a single Jetson, and XIMEA is one of the NVIDIA partners to showcase how to achieve it.

How image processing could be done on Jetson¶

Here we consider just ISP (Image signal processor) and CUDA-based image processing pipelines to describe how the task could be solved, which image processing algorithms could be utilized, etc.

• ISP inside Jetson (libargus library on the top of hardware solution)
• V4L2 framework instead of argus/nvcamerasrc to get bayer data like v4l2-ctl
• Image processing on CUDA (NPP library, Fastvideo SDK)
• Image processing on ARM (C++, Python, OpenCV)
• Hardware-based encoding and decoding with NVENC
• AI on CUDA and/or Tensor cores

To begin, you can explore the NVIDIA camera architecture.

Camera Architecture Stack¶

The NVIDIA camera software architecture includes NVIDIA components for ease of development and customization:

Fig.1. Diagram from Development Guide for NVIDIA Tegra Linux Driver Package (31.1 Release, Nov.2018)

NVIDIA Components of the camera architecture¶

NVIDIA provides OV5693 Bayer sensor as a sample and tunes this sensor for the Jetson platform.
The drive code, based on the media controller framework, is available at:
./kernel/nvidia/drivers/media/i2c/ov5693.c

• libargus - provides low-level API based on the camera core stack
• nvarguscamerasrc - NVIDIA camera GStreamer plugin that provides options to control ISP properties using the ARGUS API
• v4l2src - standard Linux V4L2 application that uses direct kernel IOCTL calls to access V4L2 functionality

NVIDIA further offers additional sensor support for BSP software releases.
Developers must work with NVIDIA certified camera partners for any Bayer sensor and tuning support.

Tasks involved in Development work¶

These tools and operating mechanisms are NOT part of the public Jetson Embedded Platform (JEP) Board Support Package release.

• Sensor driver development
• Custom tools for sensor characterization
• Image quality tuning

For more information on sensor driver development, see the:
NVIDIA V4L2 Sensor Driver Programming Guide.

Jetson ISP¶

Jetson includes internal hardware-based solution - ISP which was created for realtime mobile camera applications with high performance and low latency.
To control these features on Jetson hardware, there is libargus library. Camera application API (libargus) offers:

• low-level frame-synchronous API for camera applications, with per frame camera parameter control
• multiple (including synchronized) camera support
• EGL stream outputs.

RAW output CSI cameras requiring ISP can be used with either libargus or GStreamer plugin.
In either case, the V4L2 media-controller sensor driver API is used. Sensor driver API (V4L2 API) enables:

• video decode
• encode
• format conversion
• scaling functionality

V4L2 for encode opens up many features like bit rate control, quality presets, low latency encode, temporal tradeoff, motion vector maps, and more.

Libargus library features for Jetson ISP¶

To summarize, ISP is a fixed-function processing block which can be configured through the Argus API, Linux drivers, or the Technical Reference Manual which contains register information for particular Jetson.

• Bad pixel correction
• Bayer domain hardware noise reduction
• Per-channel black-level compensation
• High-order lens-shading compensation
• 3A: AF/AE/AWB
• Demosaic
• 3x3 color transform
• Color artifact suppression
• Downscaling
• Edge enhancement (sharp)

All information about utilized algorithms (AF, AE, demosaicing, resizing) is closed and user needs to test them to evaluate quality and performance.

How to choose the right camera¶

To be able to utilize ISP the camera needs a CSI interface.
If the camera does not offer CSI support (for example GigE, USB3, CameraLink, Coax, PCIe camera) the CSI driver needs to be created.
Even without CSI driver, there is still a way to connect the camera to Jetson using suitable carrier board with correct hardware output slot. Usually, this is either USB-3.x or PCI Express.
As NVIDIA partner, XIMEA offers both USB and PCIe interface cameras.

Fig.2. Jetson TX2 carrier board for XIMEA cameras

The minimum requirement to connect the camera to Jetson via carrier board is a camera driver for L4T and ARM processor.
NOTE: Keep in mind that in this case, ISP is not available. Next part deals with such a situation.

How to work with non-CSI cameras on Jetson¶

One solution is to utilize Fastvideo SDK for Jetson GPUs.
This SDK exists for:
Jetson TK1, TX1, TX2, TX2i and AGX Xavier
You just need to send data to GPU memory and to create a full image processing pipeline on CUDA.
This is the way to keep CPU free and to ensure fast processing due to the excellent performance of mobile Jetson GPU on CUDA.
Based on this approach you can create multicamera systems on Jetson with Fastvideo SDK together with USB-3.x or PCIE cameras.

Fig.3. NVIDIA Jetson with Embedded vision camera model on TX2 carrier board from XIMEA

Image processing on Jetson with Fastvideo SDK¶

Fastvideo SDK is intended for camera applications and it has a wide choice of features for raw image processing on GPU.
This SDK also exists for NVIDIA GeForce/Quadro/Tesla GPUs and consists of high quality algorithms which require significant computational power.
Usually, ISP/FPGA/ASIC image processing modules have low latency and high performance, but because of hardware restrictions, utilized algorithms are relatively simple and have moderate image quality.
Apart from image processing modules, Fastvideo SDK has high speed compression codecs which are implemented on GPU:
JPEG (8/12 bits),
JPEG2000 (8-16 bits),
Bayer (8/12 bits)

These codecs are working on CUDA and they were heavily tested being both reliable and very fast.
For the majority of camera applications, 12 bits per pixel is a standard bit depth and it makes sense to store compressed images at least in 12-bit format or at 16-bit.
Full image processing pipeline on Fastvideo SDK is done at 16-bit precision, but some modules which require better precision are implemented with float.
To check the quality and performance of RAW image processing with Fastvideo SDK, the user can download GUI application for Windows which is called Fast CinemaDNG Processor together with sample image series in DNG format: www.fastcinemadng.com

Fig.4. Image processing workflow on CUDA at Fastvideo SDK for camera applications

This application has benchmarks window to check time measurements for each stage of image processing pipeline on GPU.
The same application for Linux is expected by the end of Q1/2019.

High-resolution multicamera system for UAV Aerial Mapping¶

Application Example: 5K vision system for Long Distance Remote UAV
Manufacturer: MRTech company

Fig.5. Multiple cameras connected to Jetson TX2 carrier board from XIMEA

Cameras:
• One XIMEA 20 MPix PCIe camera MX200CG-CM
• Two XIMEA 3.1 MPix PCIe cameras MX031CG-SY

Hardware:
• NVIDIA Jetson TX2 or TX2i module with a custom carrier board
• NVMe SSD 960 PRO M.2 onboard

Image processing:
• Full workflow on CUDA: acquisition, black level, white balance, LUT, high quality demosaicing, etc.
• H.264/265 encoding, RTSP streaming via the radio channel
• Streaming of 4K images at 25 fps and 2× Full HD 1080p (1920 × 1080) images at 30 fps simultaneously
• Save high resolution snapshot images to SSD
• Power usage 35W (including all cameras)

AI imaging applications on Jetson¶

With the arrival of AI solutions the following task needs to be solved: how to prepare high quality input data for such systems.
First, choose high resolution color camera with bayer pattern images that come in realtime.
Next, implement fast RAW processing to be able to feed AI solution with high quality pictures fast enough.
The latest Jetson AGX Xavier has high performance Tensor cores for AI applications and these cores are ready to receive images from CUDA software.

Credentials
Fastvideo Blog:
https://www.fastcompression.com/blog/jetson-image-processing.htm