Documentation

Install DgiStreamer

DgiStreamer works on top of GStreamer and, as such, installing GStreamer is a prerequisite.
At the moment, GStreamer is not included in the installation bundle and you will need to install it manually or use the DgiRemote Installer.

DgiStreamer’s user interface itself does not require the presence of GStreamer to work, allowing you to keep the UI on a machine completely separated from the runtime. However, DgiStreamer uses a separate application called DgiRemote that needs to be installed on the machine and on which a more advanced integration with GStreamer is required. In the future, this will allow the development of advanced functionality and provides features including node synchronization where DgiStreamer can load the effective available GStreamer nodes provided on the target machine.

Install

The DgiStreamer installation is straightforward; simply go here and request the executable for your system: Download DgiStreamer!

Install DgiRemote (only on target devices)

Installation of DgiRemote requires GStreamer pre-installed.

Remote installation of DgiRemote is automated in the current version of DgiStreamer, but is only supported on Linux devices.

  1. Launch DgiStreamer
  2. Set up the remote connection, if the local machine is selected the tool will be installed locally
  3. Press on the Tools menu and then select install DgiRemote
  4. DgiStreamer will start to run an installation script, and you will be prompted to accept the end-user license agreement (EULA) (Note: if you are performing the installation on a remote machine you will be requested to prompt the password for the log-in before the installation begins)
  5. After accepting the EULA, your password will be requested since the installation requires administrator privileges.
  6. The tool will download DgiRemote and install it.

An introduction to system-to-system pipeline development

Software & Hardware

  • Server: Linux Machine
  • OS: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
  • GStreamer: 1.20.3
  • Client: Linux Machine
  • OS: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
  • GStreamer: 1.20.3

1. Introduction

This tutorial will demonstrate the development of two separate pipelines that are connected over the network, specifically a local LAN connection across two machines.

In order to make this tutorial work, we need to ensure that both machines are set up properly. Both machines will need to have GStreamer installed Official GStreamer Installation Guide.

If required, we can also install the DgiStreamer on the remote device to enable the extra features: Install DgiRemote!

Finally, we need a machine with DgiRemote to develop our pipeline; this can be either of our machines.

2. Developing the pipeline

We need to develop two pipelines: one to stream a video source and a second one to receive the stream and display it on the machine. However, before continuing we need to introduce some technicalities on how User Datagram Protocol (UDP) streaming in GStreamer works.

GStreamer and UDP streaming

In GStreamer, UDP streaming can work in two ways:

  • broadcast: by providing a valid broadcast IP
  • point-to-point: where the streaming device streams directly to the client’s IP.

In this tutorial, we will use the point-to-point method because we need to recover the IP of the client machine by using the command ifconfig in the terminal.
However, streaming raw video over the internet is not advisable since packet loss or errors in delivery can completely kill the received stream. Moreover, no information about video synchronization is present, thus continuity issues can arise on the receiving end. GStreamer proposes video encapsulation in RTP payloads, a protocol to handle real-time video. GStreamer provides a series of dedicated payload wrappers depending on the chosen encoding, so you can change the encoding according to your liking.

Note: it should also be possible to provide streaming in a more conservative manner, by streaming on a machine using the 0.0.0.0 IP and providing the IP of the machine to the client(s). However, this has some known issues and we, therefore, leave it out of the scope of this tutorial.

Source pipeline (streaming pipeline)

  • v4l2src: this is a node available only for Linux. It is part of the video4linux2 plugin that enables the reading of Linux files for video, or reading of the camera output. By default, data acquired by the first camera connected to the computer are written in /dev/video0 directory.
  • videotestsrc: this is an alternative node in cases where a camera is not available.
  • jpegenc: this node will encode the raw video using the jpeg encoder, generating a jpeg video stream.
  • rtpjpegpay: this node subsequently wraps the encoded jpeg video in the RTP protocol in order to be ready for streaming.
  • udpsink: this is the final node that will serve the video for the client to connect and receive the stream. We haven’t modified anything; however, we would like to point out that by default this module streams on the localhost IP (127.0.0.1) and has the auto-multicast setting enabled, which allows the streaming to automatically join multicast-groups.

Sink pipeline (Reading pipeline)

  • udpsrc: network source that reads UDP packets from the network. We have set the value of the port=0 to enable the automatic port search, and enforced the caps=application/x-rtp in order to listen for the expected input; this can be useful if you have more than 1 stream in the same network.
    (ARE THE BELOW BULLET POINTS CORRECT??)
  • capsfilter: UDP only provides raw bits and we do not know their content, therefore we will put a capsfilter to specify the format of the received stream. This information can be recovered from the pay node used, which in our case is rtpjpegpay. To do this we will be using the following filter: application/x-rtp,encoding-name=JPEG,payload=96 in the caps field. The application/x-rtp specifies that we are receiving an rtp stream, the encoding-name=JPEG tells the depay node that the encoding used is JPEG, while payload=96 is the output payload size of the rtpjpegpay node used in the streaming server.
  • rtpjpegdepay: this node removes the encoded video from the RTP protocol packets, and enforces various processes such as synchronization based on the protocol.
  • jpegdec: this node decodes JPEG-encoded video.
  • videoscale: scales the raw and unencoded video in order to be better visualized. Usually, streaming the video in low resolution and scaling it back can improve the streaming speed while maintaining good quality. The default scaling method is bicubic.
  • autovideosink: is used to show the result; it automatically searches for a valid GStreamer node to show the video on the current machine.

3. Running the pipeline

We now have to run the pipeline separately. We first need to launch the streaming pipeline in the source machine, and then the client pipeline in the second machine. We can do this in two ways: (1) open two instances of DgiStreamer and connect to the two machines, or (2) export the pipeline in GStreamer format using the Preview Button and launch them in the terminal:

  1. Source Pipeline:
    • Using camera on Linux: gst-launch-1.0 v4l2src ! jpegenc ! rtpjpegpay ! udpsink
    • Using test video source: gst-launch-1.0 videotestsrc ! jpegenc ! rtpjpegpay ! udpsink
  2. Client Pipeline: gst-launch-1.0 udpsrc caps='application/x-rtp' ! rtpjpegdepay ! jpegdec ! videoscale ! autovideosink

After a few seconds, you should be able to see your video!

Downloads

Streaming Pipeline: With Camera! / No Camera! (the No Camera! is a modification to support machines with no attached cameras, a test video generator will be used instead)

Receiving Pipeline: Download!

DeepStream tutorial

In this tutorial, we will go through the process of developing a Deep Learning powered pipeline using the Nvidia® DeepStream plugin for GStreamer. We will implement an already-developed YOLO model for basic object detection. In our case, we will be using a Jetson Nano, however, you can implement the system on any machine on which you can install DeepStream.

Software & hardware

  • Remote: Jetson AGX Orin (Dev Kit)
    • OS: Ubuntu 20.04.5 LTS (Nvidia Jetson Distro)
    • GStreamer: 1.20.3
    • Nvidia-Jetpack: 5.0.1-b118
    • Nvidia-Deepstream: 6.1
  • Client: Linux Machine
    • OS: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
      • GStreamer: 1.20.3

Setting up the systems

We expect the user reading this tutorial to have DgiRemote installed; if this isn’t the case you can look at the top of the page: Install DgiStreamer!

Install DeepStream:

We redirect you to the official tutorial: Install DeepStream.

This is an important step since we need some premade configurations from Nvidia. You can find the necessary resources under /opt/nvidia/deepstram/deepstram-<VERSION>/samples/, which contains a set of trained models and configuration files for the DeepStream tool, as well as some videos to test them in case you are missing a camera device.

Developing the pipeline

The pipeline will run on the Jetson system and will implement the DeepStream system from Nvidia; we will be using a pre-developed model from Nvidia to get started on its implementation and use.

Before continuing we would like to introduce the concept of memory location which, in the case of GStreamer, refers to the location between systems and GPU memory where our current data are placed. This is important since the DeepStream system expects the pipeline to store the video in video memory in order to work.

Please note that, at the moment, we haven’t developed a version of DgiStreamer for ARM processors. Therefore, we will not be able to use the tool directly on the Jetson and we will solve this by streaming the video processed on the Jetson back to us, or export the pipeline and launch it by hand on the device.

Develop the pipeline:

  • uridecodebin3: This is an advanced node that can get a stream from any provided uniform resource identifier (URI), while at the same time trying to decode it in raw format. Here, we will be referring to a file provided by Nvidia but you can also place the link to a video on the web
    • Set the URI field to : file:///opt/nvidia/deepstream/deepstream/samples/streams/sample_1080p_h264.mp4
  • queue: A simple node to add a queue before the next element, which is very useful for avoiding losing some frames due to lagging or cases where the next node is very slow.
  • nvstreammux: This component prepares the data in order to apply deep-learning inference to them, and allows you to define batches and to set up the shape of the tensor to give it as input to the nvinfer node.
  • nvinfer: This node applies the inference step on the data using a model converted in NVIDIA® TensorRT™. In this case, we use an example setup with a simple 2-class object classification provided by NVIDIA®, however, the user can develop a custom classifier and use it instead; we are providing a guide to do so: [Develop your classifier for Deepstream](). Watch Note 2 for issues about compilation time.
    • Set the config-file-path setting with the Deepstream config path /opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt (this may change in newer versions of DeepStream)
  • nvvideoconvert: This plugin performs video colour format conversion and enables us to upload and/or download data from the GPU memory. Moreover, it will perform the raw data conversion to a format accepted by the Nvidia nodes.
  • nvdsosd: This plugin draws bounding boxes, text, and region of interest (RoI) polygons (presented as a set of lines). The plugin accepts an RGBA buffer with attached metadata from the upstream component. The bounding boxes may be shaded based on custom configurations (e.g. width, colour and opacity), and text or RoI polygons can be placed at specified locations in the frame. Text and polygon parameters are configurable through metadata.
  • ximagesink: A sink node based on x-server for Linux to display the video output.

Note 1: Different cameras have different capabilities some cameras can provide raw data while others provide a pre-encoded video. In the former case, you need to use only the relative video caps and then a videoconvert node. In the latter case, you have to decode the video in raw format but the decoding depends on your camera specifications (same goes for video files: if they are raw we have to change the raw format using videoconvert, if they are encoded we have to decode them).

Note 2: nvinfer contains a field called model-engine-file, which is the path to an engine file of the model which is a compiled and optimized version of your model done by DeepStream. By default, if this file is not present, nvinfer will compile our raw model and create an engine file (this process can take up to 10 minutes). We advise you to point this file (also in the configuration text file) to a folder where you can write without the need for sudo, otherwise the engine file will not be created making you wait several minutes between each run.

Running the pipeline

We can deploy the pipeline as always using the deploy target in the target machine (expecting it to be a Jetson), or deploy it ourselves.
However, an extra step is required here: we need to move the configuration files and models used in the nvinfer plugin in the target system and ensure the path to them is correct in our pipeline.

Once this is done we can run our pipeline!

Add the video loopback

Sometimes it’s not possible to have a screen connected to your Jetson, or you simply want to work on it remotely. This, however, will block us from using any node that relies on the User Interface on the target system; for example, if we run remotely a pipeline with the ximagesink node, the pipeline will crash.

We can replace such a display node by streaming back the video to our machine using some network solution. However, this is not straightforward and for streaming over IP we need to set up a WebServer on the remote device.

Method 1 – Use the DeepCamera loopback node

Since developing a system to stream video across the web is not straightforward, we developed a custom node present in the DgiStreamer called debugvideosink that works alongside DgiRemote (that has to be installed on the remote device), to automatically do all the required setup for streaming the video.
To use it you only need to put it in the pipeline by feeding it a raw video (video/x-raw). It will then generate a pipeline to stream the video using the nodes available on GStreamer and communicate with the DgiRemote executable to set up the streaming back to our machine.
If you didn’t install it on DgiRemote, you should do it before proceeding: Install DgiRemote!

To use the debug node you have only to connect it to a raw video feed in the pipeline:

Before continuing further, we should note that the debugvideosink will work only if you launch the pipeline using the debug button, which will request the target machine to launch the pipeline using the DgiRemote tool; without this the debugvideosink will be replaced by a fake sink in the final pipeline. This allows you to keep it on the pipeline for debug without removing it when you need to deploy such pipeline.

Now let’s run the pipeline using the debug button and wait for the pipeline to start…

Now let’s run the pipeline using the debug button, And after having waited for the pipeline rollout, we will be able to receive the stream directly on the node inside DgiStreamer.

Method 2 – Develop a custom pipeline

The alternative option is to develop a pipeline that streams over UDP using the required encoding. We will be using h264 encoding in this example.
Let’s change our pipeline to add the encoding (which in this case is a special Jetson module provided by Nvidia) and then wrap it with the RTP payload before streaming it over UDP.

  • nvv4l2h264enc: will encode the raw video in h264 format while allowing you to use hardware acceleration on Jetson devices and/or on machines with an Nvidia GPU.
  • h264parser: Is needed to parse the h264 and prepare it for the format required by the next node.
  • rtph264pay: Puts the video bits in RTP payloads.
  • udpsink: Streams the RTP feed.
    • Here we need to set up the host field to the IP of the machine on which we want to receive the stream in the Local Network. You can use the ifconfig on Linux terminal or the ipconfig on Windows command prompt.
    • We should ensure the port on the udpsink and updsrc are the same ones.

Furthermore, we need to develop a pipeline that receives the video as follows:

  • udpsrc: Read the video from the UDP channel.
  • capsfilter: UDP only provides raw bits and we do not know their content, therefore we will put a capsfilter to specify the format of the received stream.
  • the filter is application/x-rtp,encoding-name=H264,payload=96 (the application/x-rtp specifies that we are receiving an rtp stream, the encoding-name=H264 tells to the depay node that the encoding used is h264, while they payload=96 is the output payload size of the rtph264depay node using in the streaming server)
  • rtph264depay: Remove the RTP payload from the stream and recover the encoded video in h264.
  • avdec_h264: Decode the h264 to raw video format.
  • autovideosink: Display the output in a system window choosing the best video sink for the current system.

Now we are ready to run the whole system; for this we need to deploy the server pipeline with the object detection on the Jetson, and the client pipeline that shows the stream on the machine we are working from. We can do this in 2 ways:

  1. We can manually run the server pipeline by getting the gst-launch format pipeline using the preview button, and run it by hand on the jetson (e.g., using an ssh connection)
  2. Open two instances of DgiStreamer on the working machine and connect the first one with the Jetson and the second one with the local machine.

As you can see, by using the special debugvideosink node you can avoid the detailed setup needed in the other case and enable you to focus on the really important part of your new system!

Downloads

Local Pipeline: Download!

Loopback with DgiRemote: Download!

Loopback with custom pipeline: Source! / Sink!

Use a custom YOLO model in your pipeline

Software & hardware

  • System: Jetson AGX Orin (Dev Kit)
    • OS: Ubuntu 20.04.5 LTS (Nvidia Jetson Distro)
    • GStreamer: 1.20.3
    • Nvidia-Jetpack: 5.0.1-b118
    • Nvidia-DeepStream™: 6.1

Set up the systems

This tutorial is based on the pipeline built on a previous example; before continuing, we strongly recommend to check the previous tutorial: Here!

Preview

We will be using the YOLOv7. In this example, we will be using the original model weights but the user is free to fine-tune the model, as well as change the model output classes.

This version of YOLO is developed in PyTorch and, as such, we need to extract the configuration for DeepStream. For this purpose, we will be using a utility script provided by DeepStream-Yolo.

Finally, we will develop a custom setting file for the nvinfer node to let the pipeline system know how to classify our model output and how it should handle the model.

Develop the pipeline

(Image of the older pipeline)

We will be working with the same pipeline we developed for object detection using the model provided by Nvidia, but we will be changing the nvinfer node to use our model.

You can download the pipeline from: here!.

Compile our models

We will be using the weights provided by the model’s author (yolov7.pt); you can find these here: https://github.com/WongKinYiu/yolov7#testing

Note: you can also download the other weights but you have to change the configuration of nvinfer accordingly.

Also, we need to download the code to have access to the conversion script; we can either:

  • Download the project zip: DeepStream-Yolo.
  • Clone the code using git:
    • Open the terminal
    • Run git clone https://github.com/WongKinYiu/yolov7.git or git clone https://github.com/marcoslucianops/DeepStream-Yolo.git

Then we need to install PyTorch to use the script (any version): Install PyTorch.

After the installation we are ready to convert our model using the authors’ script:

  • Open the terminal in the project folder
  • Run the following: python ./utils/gen_wts_yoloV7.py <path to the downloaded .pt model>

In the end, we should have two new files: the yolov7.cfg that contains the model description and the yolov7.wts, ready to be loaded in the configuration.

Set up the nvinfer node

We now need to develop the configuration file for our pipeline.

To help us, we can use the yoloV7 re-config provided by Nvidia config_infer_primary_yoloV7.txt and change it to use our weights.

We can organize it more appropriately by renaming it to config_custom_yoloV7.txt, and change the following within the text file:

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-color-format=0
custom-network-config=<path to yolov7.cfg>
model-file=<path to yolov7.wts>
model-engine-file=model_b1_gpu0_fp32.engine
## The file containing the labels for the classes
labelfile-path=labels.txt
batch-size=1
network-mode=0
## Classes in the coco dataset
num-detected-classes=80 
interval=0
gie-unique-id=1
process-mode=1
network-type=0
cluster-mode=2
maintain-aspect-ratio=1
symmetric-padding=1
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
engine-create-func-name=NvDsInferYoloCudaEngineGet
[class-attrs-all]
nms-iou-threshold=0.45
pre-cluster-threshold=0.25
topk=300

If you are working outside Jetson, remember to place all the generated files on the machine where the pipeline will run. The required files are:

  • yolo.cfg
  • yolo.wts
  • config_custom_yoloV7.txt

After copying them, ensure the path config_custom_yoloV7.txt is still correct.

Finalize the pipeline

Now we need to change the path in our pipeline to the new config_custom_yoloV7.txt file.

Inside nvinfer, change the config-file-path setting with the location of config_custom_yoloV7.txt.

We can then run our pipeline!

Integrate OpenCV with your pipeline

Software & Hardware

  • Linux Machine
    • OS: Ubuntu 22.04.1 LTS (Jammy Jellyfish)
    • GStreamer: 1.20.3

1. Introduction

In this tutorial, we will demonstrate how you can integrate a pipeline built with DgiStreamer with your project.
We will be doing this using OpenCV to read (or write) the video feed produced by the GStreamer pipeline developed with DgiStreamer.

2. Setup

We expect you have already set up DgiStreamer and installed GStreamer on the target machine. If you have skipped this step, please follow look here:

Install Python

This tutorial is based on Python; we suggest using a virtual environment manager (e.g. conda) because some specific properties of OpenCV are needed to make the code work.

  1. Install python (plain or using conda)

Install OpenCV

There are different ways to install OpenCV but we need the version with the additional plug-ins:

  1. Compile it yourself: you have to download and compile OpenCV with the GStreamer and QT modules (however, this is out of the scope of this tutorial; if you are interested, we have a dedicated tutorial)
  2. Using pip: pip install opencv-contrib-python (note: please uninstall any older installations with pip)
  3. Using Conda: conda install -c michael_wild opencv-contrib

We can now test our installation:

  1. run the python shell by running the python command in a terminal
  2. run the following code: import cv2 print(cv2.getBuildInformation())
  3. under Video I/O the GStreamer value should be: YES, as follows:
   QT: YES (ver 5.14.2)
   GStreamer: YES (1.16.2)

NOTE: be careful if you are using conda or virtual enviorments. It is possible that some conda libraries will install a custom version of GStreamer that could override the one we installed in the base system, and therefore the one used in DgiStreamer. You will end up working with desynchronized nodes and GStreamer versions.

3. OpenCV and DgiStreamer

After we have installed GStreamer and OpenCV with the required plugins, we can create our pipelines and include them in our code.
We can use OpenCV to allow us to run a GStreamer pipeline inside our code and also connect to it to retrieve or feed the video data we need.
We can do this using the following two methods:

  • cv2.VideoCapture() if we want to read the data from the pipeline
  • cv2.VideoWriter() if we want to write the data into the pipeline

We can therefore develop a pipeline using DgiStreamer and then import this pipeline into our code.

4. Developing the pipeline

We will use a simple pipeline that receives the input data from GStreamer, but we can modify it to read a stream or read the feed after some type of processing, e.g. an object detection step.

Before starting, however, we should introduce two GStreamer app nodes, appsink, and appsrc.

  • appsink: allows you to read data generated from the pipeline; via OpenCV this can be done using cv2.VideoCapture()
  • appsink: allows you to write data to the pipeline by generating a stream of information; via OpenCV this can be done using cv2.VideoWriter()

Nevertheless, our pipeline will simply read some data from the stream. The pipeline is shown below:

Let’s break down what the different nodes do:

  • autovideosrc: given an input source it will try to generate an internal pipeline to handle such stream, e.g. in a Linux system it will try to use video4linux.
  • videoconvert: this is a popular node that tries to convert the video in a format that the next node supports, e.g. if the input is in RGB format but the next node wants it in BRG it will attempt to convert it in that format if possible.
  • appsink: this is the node that enables us to read the video feed from GStreamer from our code using CV2.

Now that we have developed our pipeline, we can export it in order to use it in our code. Since OpenCV expects a pipeline like the one used in the gst-launch-1.0 command, DgiStreamer will enable you to convert the graphical pipeline in the GStreamer text format.
You can use the preview button to do so:

4. Developing the OpenCV code

Before we can use the generated pipeline, we need to develop the code to access the output of GStreamer from OpenCV and display it.

The following code will then execute the pipeline and connect to the appsink to extract the video from it (you can place the app sink in any location throughout the pipeline, therefore the captured output changes depending on the data type you feed to the appsink node). Then, it displays it in the OpenCV QT integration until the streaming ends or the q button is pressed.

import cv2 as cv
pipeline = ''
if __name == "__main":
   video = cv.VideoCapture(pipeline)
   while True:
      try:
         ret, frame = video.read()
         cv.imshow("Frame", frame)
         if cv.waitKey(1) == ord(’q’):
            break
         except Exception as e:
            print(e)
            breakvideo.release()
         cv.destroyAllWindows()

We can now paste the copied pipeline in the pipeline variable; remember to remove the gst-launch-1.0 since it is not needed in OpenCV.

import cv2 as cv
pipeline = 'autovideosrc ! videoconvert ! appsink'
if __name == "__main":
   video = cv.VideoCapture(pipeline)
   while True:
      try:
         ret, frame = video.read()
         cv.imshow("Frame", frame)
         if cv.waitKey(1) == ord(’q’):
            break
         except Exception as e:
            print(e)
            breakvideo.release()
         cv.destroyAllWindows()

We can now run our code and we should be able to see the feed recorded from GStreamer in our OpenCV viewer.

Downloads

Pipeline: Download!