ResNet (Image Classification)

1. Introduction

The performance of deep neural networks (DNN) usually improves as the number of network layers increases, but when the number of layers exceeds a certain threshold, a "degradation problem" arises: the network accuracy saturates or even drops. This is not due to overfitting or vanishing/exploding gradients, but because deep networks are hard to optimize.

ResNet (Residual Network) was proposed by Kaiming He et al. in 2015. It solves this problem through a residual learning framework, achieving effective training of networks with more than 150 layers for the first time in the ImageNet competition and advancing the development of deep neural networks.

Project Directory

RESNET
│  libsophon_soc_0.5.1-LTS_aarch64.tar.gz	#烧录依赖环境
│  sophon-mw-soc_0.12.0_aarch64.tar.gz		#烧录依赖环境
│  stream_dev.tar							#交叉编译环境镜像
│
├─cpp		C++例程所需文件
│  ├─dependencies
│  │  ├─include
│  │  │      bmnn_utils.h
│  │  │      bm_wrapper.hpp
│  │  │      ff_decode.hpp
│  │  │      json.hpp
│  │  │      utils.hpp
│  │  │
│  │  └─src
│  │          ff_decode.cpp
│  │
│  ├─resnet_bmcv
│  │      CMakeLists.txt
│  │      main.cpp
│  │      resnet.cpp
│  │      resnet.hpp
│  │
│  └─resnet_opencv
│          CMakeLists.txt
│          main.cpp
│          resnet.cpp
│          resnet.hpp
│
├─datasets		#测试数据集
│  └─img
│
├─docs			#教程说明文档
│  │  ResNet.md
│  │
│  └─images
│
├─models		#模型文件
│  └─BM1684X
│          resnet50_fp16_1b.bmodel
│          resnet50_fp32_1b.bmodel
│          resnet50_int8_1b.bmodel
│          resnet50_int8_4b.bmodel
│
├─python		#python例程所需文件
│      resnet_bmcv.py
│      resnet_opencv.py
│
└─tools			#可能用到的工具
        compare_statis.py	#比较运行结果
        eval_imagenet.py	#精度测量

2. Running Steps

1. Python Example

1.1 Configure the Python Environment

bmcv environment (for running resnet_bmcv.py)

Edit the .bashrc file to import the sophon Python environment:

sudo vim ~/.bashrc

Add the following line at the end of the file:

export PYTHONPATH=$PYTHONPATH:/opt/sophon/libsophon-current/lib:/opt/sophon/sophon-opencv-latest/opencv-python/

After :wq to save and exit, reload the terminal:

source ~/.bashrc

You can run echo $PYTHONPATH to verify the field.

opencv environment (for running resnet_opencv.py)

pip install opencv-python-headless

1.2 Inference Test

File Parameter Description

resnet_opencv.py and resnet_bmcv.py take the same command-line arguments. Using resnet_opencv.py as an example, the parameters are:

usage: resnet_opencv.py [--input INPUT_PATH] [--bmodel BMODEL] [--dev_id DEV_ID]

--input: 测试数据路径，可输入整个图片文件夹的路径；
--bmodel: 用于推理的bmodel路径，默认使用stage 0的网络进行推理；
--dev_id: 用于推理的tpu设备id；

Image Test

The following is an image test example; testing an entire image folder is supported. Enter the corresponding directory, e.g. /data/ResNet/, to debug the files.

python3 python/resnet_opencv.py --input datasets/img --bmodel models/BM1684X/resnet50_fp32_1b.bmodel --dev_id 0

After execution, the prediction results are saved under results/resnet50_fp32_1b.bmodel_img_opencv_python_result.json, and the prediction results, inference time, and other information are printed. The output is as follows:

2. C++ Example

1. Cross-Compilation Environment Setup

1.1 Build Environment

C++ programs need their dependencies compiled to run on the board. To reduce load on the edge device, we use an x86 Linux environment for cross-compilation.

Two ways to set up the cross-compilation environment are provided:

(1) Install the cross-compilation toolchain via apt:

If your system's libc version matches the target SoC platform's libc version (check with ldd --version), you can install with:

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

To uninstall:

sudo apt remove cpp-*-aarch64-linux-gnu

If your environment does not meet the above requirements, method (2) is recommended.

(2) Set up the cross-compilation environment via Docker:

You can use the Docker image we provide — stream_dev.tar — as the cross-compilation environment.

If you are using Docker for the first time, run the following commands to install and configure it (only needed once):

sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Load the image from the downloaded image directory:

docker load -i stream_dev.tar

You can view the loaded image with docker images; it is named stream_dev:latest by default.

Create a container:

docker run --privileged --name stream_dev -v $PWD:/workspace  -it stream_dev:latest
# stream_dev只是举个名字的例子, 请指定成自己想要的容器的名字

The workspace directory inside the container is mounted to the host directory where you ran docker run; you can compile the project inside this container. The workspace directory is under the root directory, and changes in it are mapped to the corresponding files in the local directory.

Note: When creating the container, you must be in the parent directory of soc-sdk (the dependency build environment) or above.

1.2 Package Dependency Files

1. Package libsophon

   For libsophon_soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 创建依赖文件的根目录
   mkdir -p soc-sdk
   # 解压libsophon_soc_x.y.z_aarch64.tar.gz
   tar -zxf libsophon_soc_${x.y.z}_aarch64.tar.gz
   # 将相关的库目录和头文件目录拷贝到依赖文件根目录下
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/lib soc-sdk
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/include soc-sdk

2. Package sophon-ffmpeg and sophon-opencv

   For sophon-mw-soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 解压sophon-mw-soc_x.y.z_aarch64.tar.gz
   tar -zxf sophon-mw-soc_${x.y.z}_aarch64.tar.gz
   # 将ffmpeg和opencv的库目录和头文件目录拷贝到soc-sdk目录下
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/include soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/include soc-sdk

1.3 Perform Cross Compilation

After the cross-compilation environment is set up, use the cross-compilation toolchain to build the executable. resnet_opencv and resnet_bmcv use the same build method; here we use resnet_opencv as an example:

cd cpp/resnet_opencv
mkdir build && cd build
#请根据实际情况修改-DSDK的路径，需使用绝对路径。
cmake -DTARGET_ARCH=soc -DSDK=/workspace/soc-sdk/ ..
make

After the build, a .soc file is generated in the corresponding directory, e.g. cpp/resnet_opencv/resnet_opencv.soc:

2. Image Test

Copy the cross-compiled executable and the required model and test data to the SoC platform (i.e. the BM1684X development board) for testing.

Parameter Description

The executable has a default set of parameters; pass parameters according to your actual situation. resnet_bmcv.soc and resnet_opencv.soc take the same parameters. The parameters are:

usage:./resnet_opencv.soc [params]

        --bmodel (value:../../models/BM1684X/resnet50_fp32_1b.bmodel)
                bmodel file path
        --dev_id (value:0)
                TPU device id
        --help (value:0)
                print help information.
        --input (value:../../datasets/imagenet_val_1k/img)
                input path, images direction or video file path

Image Test

The following is an image test example; testing an entire image folder is supported.

##先对文件加上可执行权限
chmod +x resnet_opencv.soc
./resnet_opencv.soc --input=../../datasets/img --bmodel=../../models/BM1684X/resnet50_fp32_1b.bmodel --dev_id=0

After execution, the prediction results are saved under results/resnet50_fp32_1b.bmodel_img_opencv_cpp_result.json, and the prediction results, inference time, and other information are printed. The output is as follows: