YOLOv5 Object Detection

Based on a GStreamer + RKNN plugin chain, this example runs the full YOLOv5s pipeline on the board: image decode → preprocessing → NPU inference → post-processing (NMS) → OSD box drawing → MPP hardware encode → mp4 output.

Overall block diagram:

test.jpg (multifilesrc loop)
    │
    ▼
jpegdec → videoconvert → videoscale → BGR 640×640
    │
    ▼
rknninfer (YOLOv5s RKNN, RK1828 NPU inference)
    │
    ▼
rknnpostprocess (libyolov5spostprocess.so, NMS + decode)
    │
    ▼
rknnosd (box drawing + class labels on BGR)
    │
    ▼
videoconvert → NV12 640×640 → mpph264enc → h264parse → mp4mux → /tmp/yolov5_50frames.mp4

Two processing domains:

RK3588: jpegdec / videoconvert / videoscale / mpph264enc / mp4mux (all hardware encode/decode, VPU/RGA/MPP)
RK1828 (PCIe co-processor): rknninfer performs inference via /dev/dri/renderD128 (PCIe address 0004:41:00.0)

1. Install dependencies

sudo apt-get install -y \
    gstreamer1.0-rockchip1 \
    gstreamer1.0-plugins-good \
    gstreamer1.0-plugins-bad \
    ffmpeg

2. Verify the plugins

gst-inspect-1.0 rknninfer        | head -3
gst-inspect-1.0 rknnpostprocess  | head -3
gst-inspect-1.0 rknnosd          | head -3
gst-inspect-1.0 mpph264enc       | head -3
ls /usr/lib/libyolov5spostprocess.so

Measured output:

Factory Details:
  Long-name                RKNN infer
  Version                  1.0.0
Factory Details:
  Long-name                RKNN postprocess
  Version                  1.0.0
Factory Details:
  Long-name                RKNN osd
  Version                  1.0.0
Factory Details:
  Long-name                Rockchip Mpp H264 Encoder
  Version                  1.14.4
/usr/lib/libyolov5spostprocess.so

All 4 GStreamer plugins + the post-processing library are available.

2.1 Plugin input/output formats

rknninfer input:

gst-inspect-1.0 rknninfer 2>&1 | grep -A 5 "SINK template"

Output:

SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)GRAY8, (string)RGB, (string)RGBA, (string)BGR, (string)BGRA }

rknninfer input supports GRAY8 / RGB / RGBA / BGR / BGRA (not NV12 — run videoconvert first).

rknnpostprocess properties:

gst-inspect-1.0 rknnpostprocess 2>&1 | grep -E "config-path|library-path"

Output:

  config-path         : Location of the config file
  library-path        : Location of the library of customer post-proccessor

rknnpostprocess supports both the config-path and library-path properties.

3. Model files (bundled with the SDK)

/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/
    yolov5s.rknn             230K   YOLOv5s RKNN structure
    yolov5s.weight           8.6M   YOLOv5s weights
    detect_classes.txt               80-class COCO labels
    config.json                      SDK default config (paths need fixing)

4. config.json path fix

The SDK's config.json points to /userdata/models/yolov5s/ (a directory that does not exist on the board); install.sh fixes it automatically with cp + sed:

# What install.sh does automatically
cp /userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/config.json \
   /home/linaro/yolov5/yolov5s_config.json
sed -i 's|/userdata/models/yolov5s|/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s|' \
   /home/linaro/yolov5/yolov5s_config.json

Content after the fix:

{
  "model_path": "/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/yolov5s.rknn",
  "weight_path": "/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/yolov5s.weight",
  "label_path": "/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/detect_classes.txt",
  "conf_threshold": 0.45,
  "nms_threshold": 0.25
}

5. One-click deployment

cd /home/linaro/yolov5
bash install.sh           # interactive (precheck → y/n → package install/path fix/verify)
bash install.sh --yes     # skip confirmation

install.sh has 4 steps:

Scan the 4 system packages (install whatever is missing via sudo apt-get install -y)
Check the rknn plugins + post-processing library
Check the SDK model files
Fix config.json with cp + sed, then re-run the plugin checks to confirm

6. Daily usage

6.1 `make info` — inspect the environment

make info

6.2 `make pipeline` — run the 50-frame pipeline

make pipeline    # NUM_BUFFERS=50, i.e. 5 seconds at 10fps

Test image:

yolov5 test image

Output:

yolov5 test output 1

yolov5 test output 2

50 frames / 640×640 / h264 / done in 3.5 s, producing a 325K mp4.

6.3 `make probe` — inspect the output with ffprobe

make probe

6.4 `make clean` — remove the mp4 files

make clean    # rm -f /tmp/yolov5_*.mp4

7. Full pipeline command

gst-launch-1.0 -v \
  multifilesrc location=/userdata/RK1820_RK1828_AI_SDK/package/rknn3-test/sh/test.jpg \
                 loop=true caps="image/jpeg,framerate=10/1" num-buffers=50 ! \
  jpegdec ! videoconvert ! videoscale ! \
  video/x-raw,format=BGR,width=640,height=640 ! \
  rknninfer config-path=/home/linaro/yolov5/yolov5s_config.json ! \
  rknnpostprocess config-path=/home/linaro/yolov5/yolov5s_config.json \
    library-path=/usr/lib/libyolov5spostprocess.so ! \
  rknnosd ! videoconvert ! video/x-raw,format=NV12,width=640,height=640 ! \
  mpph264enc ! h264parse ! mp4mux ! filesink location=/tmp/yolov5_50frames.mp4

8. Performance

Metric	Measured
Input	1 jpg (test.jpg) looped 50 times
Resolution	640×640
Frame rate	10 fps
End to end (incl. mp4 mux)	3.5 s
Output size	325 KB
Encoding	h264 High profile (MPP hardware encode)
Post-processing	libyolov5spostprocess.so (NMS + decode)

9. Key facts quick reference

Item	Value
Model path	`/userdata/RK1820_RK1828_AI_SDK/examples/gstreamer/yolov5s/`
config.json (after fix)	`/home/linaro/yolov5/yolov5s_config.json`
Post-processing library	`/usr/lib/libyolov5spostprocess.so`
RKNN plugin version	1.0.0 (rknninfer / rknnpostprocess / rknnosd)
MPP encoder version	1.14.4 (mpph264enc)
Output	`/tmp/yolov5_50frames.mp4`
install.sh package install	4 gstreamer packages (the only example that installs packages)
Input format	BGR (converted from NV12 by `videoconvert`)
OSD output formats	BGR / BGRx / BGRA / RGB16 (`rknnosd` limitation)
Model classes	80-class COCO
conf / nms thresholds	0.45 / 0.25 (changeable in config.json)

10. FAQ

Symptom	Cause	Fix
`rknninfer` reports `model file not found`	Wrong config.json path	Run `bash install.sh` before `make pipeline` so it fixes config.json
mp4 `moov atom not found`	`filesink` never received EOS	Add `num-buffers=50` (or any number) to multifilesrc
`rknnosd` reports `caps not negotiated`	Input format is not BGR/BGRx/BGRA/RGB16	Put `videoconvert ! video/x-raw,format=BGR` before `rknnosd`
`rknnpostprocess` can't find the library	Wrong `library-path` or library not in `/usr/lib/`	`ls /usr/lib/postprocess.so`, use an absolute path
mpph264enc reports `baseline profile doesn't support 4:4:4`	testsrc defaults to RGB	When pushing RTSP, add `-pix_fmt yuv420p` or convert to NV12 with `videoconvert`
Want JSON coordinates/class text	`rknnpostprocess` doesn't print by default	Add an `appsink` and parse the GstBuffer meta in Python; or write your own NMS with `rknn3lite`
Want RTSP streaming	Full pipeline is available	See RTSP Streaming + AI Analysis; replace the mp4 part with `h264parse ! rtph264pay ! udpsink`

RTSP Streaming + AI Analysis — real-time video-stream detection with the same rknn plugins + RTSP streaming
PaddleOCR-VL OCR — text recognition (a completely different pipeline)