YOLOv5 (Object Detection)

1. Introduction

YOLOv5 is a single-stage object detection algorithm released by Ultralytics in 2020. It is not the work of Joseph Redmon, the original author of the YOLO series, but improves and optimizes the ideas of YOLOv3 and YOLOv4. It is implemented in the PyTorch framework and features easy deployment, high speed, and good accuracy. Its network structure includes the input end (Mosaic data augmentation, adaptive anchor calculation, etc.), the Backbone (using CSPDarknet53 as the feature-extraction network, with CSP structure and residual connections), the Neck (using an FPN+PAN structure for multi-scale feature fusion), and the output end (computing bounding-box loss via the GIOU loss function, supporting multi-class object detection). It performs excellently on datasets such as COCO.

Project Directory

YOLOV5
│  libsophon_soc_0.5.1-LTS_aarch64.tar.gz	##交叉编译所需环境
│  sophon-mw-soc_0.12.0_aarch64.tar.gz		##交叉编译所需环境
│  sophon-sail_3.8.0.tar.gz					##sail接口所需环境
│  stream_dev.tar							##libc为2.31的docker环境
│
├─cpp	##C++例程
│  ├─dependencies	##所依赖的库
│  │
│  ├─yolov5_bmcv	##bmcv例程
│  │      CMakeLists.txt
│  │      main.cpp
│  │      yolov5.cpp
│  │      yolov5.hpp
│  │      yolov5_bmcv.soc	##提供的执行文件
│  │
│  └─yolov5_sail	##sail例程
│          CMakeLists.txt
│          main.cpp
│          yolov5.cpp
│          yolov5.hpp
│          yolov5_sail.soc	##提供的可执行文件
│
├─datasets		##数据集
│
├─docs			##相关帮助文档
|
├─models		##bmodel文件
│  └─BM1684X
│          yolov5s_v6.1_3output_fp16_1b.bmodel
│          yolov5s_v6.1_3output_fp32_1b.bmodel
│          yolov5s_v6.1_3output_int8_1b.bmodel
│          yolov5s_v6.1_3output_int8_4b.bmodel
│
├─python		##python例程文件
│      postprocess_numpy.py
│      utils.py
│      yolov5_bmcv.py		# 使用SAIL解码、SAIL.BMCV前处理、SAIL推理的Python例程
│      yolov5_opencv.py		# 使用OpenCV解码、OpenCV前处理、SAIL推理的Python例程
│
├─sophon-sail	##提供的已经编译好的sail库
│
└─tools			# 存放精度测试、性能比对等python脚本
        compare_statis.py
        eval_coco.py

2. Running Steps

1. Python Example

1.1 Configure the Python Environment

bmcv environment (for running yolov5_bmcv.py)

Edit the .bashrc file to import the sophon Python environment:

sudo vim ~/.bashrc

Add the following line at the end of the file:

export PYTHONPATH=$PYTHONPATH:/opt/sophon/libsophon-current/lib:/opt/sophon/sophon-opencv-latest/opencv-python/

After :wq to save and exit, reload the terminal:

source ~/.bashrc

You can run echo $PYTHONPATH to verify the field.

opencv environment (for running yolov5_opencv.py)

pip install opencv-python-headless

1.2 Inference Test

File Parameter Description

yolov5_opencv.py and yolov5_bmcv.py take the same command-line arguments. Using yolov5_opencv.py as an example, the parameters are:

usage: yolov5_opencv.py [-h] [--input INPUT] [--bmodel BMODEL] [--dev_id DEV_ID] [--conf_thresh CONF_THRESH] [--nms_thresh NMS_THRESH]

optional arguments:
  -h, --help            打印这个帮助日志然后退出
  --input INPUT         测试数据路径，可输入整个图片文件夹的路径或者视频路径
  --bmodel BMODEL       用于推理的bmodel路径，默认使用stage 0的网络进行推理
  --dev_id DEV_ID       用于推理的tpu设备id
  --conf_thresh CONF_THRESH
                        置信度阈值
  --nms_thresh NMS_THRESH
                        nms阈值

Image Test

The following is an image test example; testing an entire image folder is supported. Enter the corresponding directory, e.g. /data/YOLOv5/, to debug the files.

 python3 python/yolov5_opencv.py --input datasets/test --bmodel models/BM1684X/yolov5s_v6.1_3output_fp32_1b.bmodel --dev_id 0 --conf_thresh 0.5 --nms_thresh 0.5

After the test, predicted images are saved under results/images, and the prediction results are saved under results/yolov5s_v6.1_3output_fp32_1b.bmodel_test_opencv_python_result.json. The prediction results, inference time, and other information are also printed.

Results:

Video Test

python3 python/yolov5_opencv.py --input datasets/test_car_person_1080P.mp4 --bmodel models/BM1684X/yolov5s_v6.1_3output_fp32_1b.bmodel --dev_id 0 --conf_thresh 0.5 --nms_thresh 0.5

After the test, the predicted results are drawn into results/test_car_person_1080P.avi; the prediction results, inference time, and other information are also printed.

yolov5_bmcv.py does not save the video; instead, it draws the prediction results on images and saves them under results/images, and it does not currently support printing text on images.

1.3 Flowchart

The processing flow in yolov5_bmcv.py and yolov5_opencv.py both follow this flowchart:

2. C++ Example

1. Cross-Compilation Environment Setup

1.1 Build Environment

C++ programs need their dependencies compiled to run on the board (you can also directly use the provided executable). To reduce load on the edge device, we use an x86 Linux environment for cross-compilation.

Two ways to set up the cross-compilation environment are provided:

(1) Install the cross-compilation toolchain via apt:

If your system's libc version matches the target SoC platform's libc version (check with ldd --version), you can install with:

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

To uninstall:

sudo apt remove cpp-*-aarch64-linux-gnu

If your environment does not meet the above requirements, method (2) is recommended.

(2) Set up the cross-compilation environment via Docker:

You can use the Docker image we provide — stream_dev.tar — as the cross-compilation environment.

If you are using Docker for the first time, run the following commands to install and configure it (only needed once):

sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Load the image from the downloaded image directory:

docker load -i stream_dev.tar

You can view the loaded image with docker images; it is named stream_dev:latest by default.

Create a container:

docker run --privileged --name stream_dev -v $PWD:/workspace  -it stream_dev:latest
# stream_dev只是举个名字的例子, 请指定成自己想要的容器的名字
#已有docker容器可通过下面命令打开
docker run -v $PWD:/workspace  -it stream_dev:latest

The workspace directory inside the container is mounted to the host directory where you ran docker run; you can compile the project inside this container. The workspace directory is under the root directory, and changes in it are mapped to the corresponding files in the local directory.

Note: When creating the container, you must be in the parent directory of soc-sdk (the dependency build environment) or above.

1.2 Package Dependency Files

1. Package libsophon

   For libsophon_soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 创建依赖文件的根目录
   mkdir -p soc-sdk
   # 解压libsophon_soc_x.y.z_aarch64.tar.gz
   tar -zxf libsophon_soc_${x.y.z}_aarch64.tar.gz
   # 将相关的库目录和头文件目录拷贝到依赖文件根目录下
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/lib soc-sdk
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/include soc-sdk

2. Package sophon-ffmpeg and sophon-opencv

   For sophon-mw-soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 解压sophon-mw-soc_x.y.z_aarch64.tar.gz
   tar -zxf sophon-mw-soc_${x.y.z}_aarch64.tar.gz
   # 将ffmpeg和opencv的库目录和头文件目录拷贝到soc-sdk目录下
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/include soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/include soc-sdk

1.3 Perform Cross Compilation

bmcv mode: After the cross-compilation environment is set up, use the cross-compilation toolchain to build the executable.

cd cpp/yolov5_opencv
mkdir build && cd build
#请根据实际情况修改-DSDK的路径，需使用绝对路径。
cmake -DTARGET_ARCH=soc -DSDK=/workspace/soc-sdk/ ..
make

After the build, a .soc file is generated in the corresponding directory, e.g. cpp/yolov5_bmcv/yolov5_bmcv.soc.

sail mode: If you use the sophon-sail interface, you must first configure sophon-sail for the soc environment. The configuration steps are:

Via cross-compilation (the example uses WSL), compile a SAIL that includes bmcv, sophon-ffmpeg, sophon-opencv.

If the build platform's libc version differs from the target, enter the Docker environment:

docker run -v $PWD:/workspace  -it stream_dev:latest

#解压sophon-sail_3.8.0.tar.gz
tar -zvxf sophon-sail_3.8.0.tar.gz
#进入到sophon目录下：cd sophon
mkdir build && cd build

cmake -DBUILD_TYPE=soc -DBUILD_PYSAIL=OFF -DCMAKE_TOOLCHAIN_FILE=../cmake/BM168x_SOC/ToolChain_aarch64_linux.cmake -DLIBSOPHON_BASIC_PATH=../../libsophon_soc_0.5.1-LTS_aarch64/opt/sophon/libsophon-0.5.1/ -DFFMPEG_BASIC_PATH=../../sophon-mw-soc_0.12.0_aarch64/opt/sophon/sophon-ffmpeg_0.12.0/ -DOPENCV_BASIC_PATH=../../sophon-mw-soc_0.12.0_aarch64/opt/sophon/sophon-opencv_0.12.0/ ..

make sail

Install the SAIL dynamic library and header files. The program will automatically create build_soc under the source directory, and the build output is installed under build_soc:

make install

Copy the sophon-sail folder under build_soc to the /opt/sophon directory of the target SOC; it can then be invoked on the SOC.

After the cross-compilation environment is ready, use the cross-compilation toolchain to build the executable.

cd cpp/yolov5_sail
mkdir build && cd build
#请根据实际情况修改-DSDK和-DSAIL_PATH的路径，需使用绝对路径。
cmake -DTARGET_ARCH=soc -DSDK=/path_to_sdk/soc-sdk -DSAIL_PATH=/wrokspace/sophon-sail/build_soc/sophon-sail ..
make

After the build, yolov5_sail.soc is generated in the yolov5_sail directory.

Because sophon-sail is moved to /opt/sophon, if you transfer files via SSH you must enter the root account. The root account has no initial password; before using it, run sudo passwd root from the linaro account to set a password.

Most Linux systems disable remote root login by default. Perform the following:

Edit the configuration file:

sudo vim /etc/ssh/sshd_config
#文件中加入
PermitRootLogin yes

Save and exit, then restart SSH:

sudo service sshd restart

After copying the sophon-sail library files to the target SOC according to the tutorial, you also need to set the following environment variable:

echo 'export LD_LIBRARY_PATH=/opt/sophon/sophon-sail/lib/:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

2. Inference Test

Copy the cross-compiled executable and the required model and test data to the SoC platform (i.e. the BM1684X development board) for testing. Entering the /data/YOLOv5/cpp/yolov5_sail directory is recommended.

Parameter Description

The executable has a default set of parameters; pass parameters according to your actual situation. yolov5_bmcv.soc and yolov5_sail.soc take the same parameters. The parameters are:

Usage: yolov5_bmcv.soc [params]

        --bmodel (value:../../models/BM1684/yolov5s_v6.1_3output_fp32_1b.bmodel)
                bmodel file path
        --classnames (value:../../datasets/coco.names)
                class names file path
        --conf_thresh (value:0.001)
                confidence threshold for filter boxes
        --dev_id (value:0)
                TPU device id
        --help (value:true)
                print help information.
        --input (value:../../datasets/test)
                input path, images direction or video file path
        --nms_thresh (value:0.6)
                iou threshold for nms

Image Test

The following is an image test example; testing an entire image folder is supported. Using yolov5_sail.soc as an example:

##先对文件加上可执行权限
chmod +x yolov5_sail.soc

./yolov5_sail.soc --input=../../datasets/test --bmodel=../../models/BM1684X/yolov5s_v6.1_3output_fp32_1b.bmodel --dev_id=0 --conf_thresh=0.5 --nms_thresh=0.5 --classnames=../../datasets/coco.names

After the test, predicted images are saved under results/images, and the prediction results are saved under results/yolov5s_v6.1_3output_fp32_1b.bmodel_test_bmcv_cpp_result.json. The prediction results, inference time, and other information are also printed.

Video Inference

The video inference of the cpp example saves the final dog results frame-by-frame under results/images, and the prediction results are saved under results/yolov5s_v6.1_3output_fp32_1b.bmodel_test_bmcv_cpp_result.json. The prediction results, inference time, and other information are also printed.

./yolov5_sail.soc --input=../../datasets/test_car_person_1080P.mp4 --bmodel=../../models/BM1684X/yolov5s_v6.1_3output_fp32_1b.bmodel --dev_id=0 --conf_thresh=0.5 --nms_thresh=0.5 --classnames=../../datasets/coco.names