ResNet (Image Classification)

1. Introduction

The performance of deep neural networks (DNN) usually improves as the number of network layers increases, but when the number of layers exceeds a certain threshold, a "degradation problem" occurs: network accuracy saturates or even decreases, not due to overfitting or vanishing/exploding gradients, but because deep networks are difficult to optimize.

ResNet (Residual Network) was proposed by He Kaiming et al. in 2015. It solved this problem through a residual learning framework. It was the first to successfully train networks with more than 150 layers in the ImageNet competition and promoted the development of deep neural networks.

Project Directory

RESNET
│  libsophon_soc_0.5.1-LTS_aarch64.tar.gz	#Burning dependency environment
│  sophon-mw-soc_0.12.0_aarch64.tar.gz		#Burning dependency environment
│  stream_dev.tar							#Cross-compilation environment image
│
├─cpp		C++ example required files
│  ├─dependencies
│  │  ├─include
│  │  │      bmnn_utils.h
│  │  │      bm_wrapper.hpp
│  │  │      ff_decode.hpp
│  │  │      json.hpp
│  │  │      utils.hpp
│  │  │
│  │  └─src
│  │          ff_decode.cpp
│  │
│  ├─resnet_bmcv
│  │      CMakeLists.txt
│  │      main.cpp
│  │      resnet.cpp
│  │      resnet.hpp
│  │
│  └─resnet_opencv
│          CMakeLists.txt
│          main.cpp
│          resnet.cpp
│          resnet.hpp
│
├─datasets		#Test dataset
│  └─img
│
├─docs			#Tutorial documentation
│  │  ResNet.md
│  │
│  └─images
│
├─models		#Model files
│  └─BM1684X
│          resnet50_fp16_1b.bmodel
│          resnet50_fp32_1b.bmodel
│          resnet50_int8_1b.bmodel
│          resnet50_int8_4b.bmodel
│
├─python		#Python example required files
│      resnet_bmcv.py
│      resnet_opencv.py
│
└─tools			#Possible tools used
        compare_statis.py	#Compare running results
        eval_imagenet.py	#Accuracy measurement

2. Running Steps

1. Python Examples

1.1 Configure Python Environment

bmcv Environment (for running resnet_bmcv.py)

Modify .bashrc file to introduce Sophon's Python environment

sudo vim ~/.bashrc

Add the following at the end of the file

export PYTHONPATH=$PYTHONPATH:/opt/sophon/libsophon-current/lib:/opt/sophon/sophon-opencv-latest/opencv-python/

Save and exit with :wq, then reload the terminal

source ~/.bashrc

You can check with echo $PYTHONPATH to see if it corresponds to the field.

opencv Environment (for running resnet_opencv.py)

pip install opencv-python-headless

1.2 Inference Testing

File Parameter Description

The command parameters for resnet_opencv.py and resnet_bmcv.py are the same. Taking resnet_opencv.py inference as an example, the parameters are described as follows:

usage: resnet_opencv.py [--input INPUT_PATH] [--bmodel BMODEL] [--dev_id DEV_ID]

--input: Test data path, can input the path of the entire image folder;
--bmodel: bmodel path for inference, default uses stage 0 network for inference;
--dev_id: TPU device id for inference;

Image Testing

Image testing example is as follows. Supports testing the entire image folder. Enter the corresponding directory, for example: /data/ResNet/, then you can debug the required files.

python3 python/resnet_opencv.py --input datasets/img --bmodel models/BM1684X/resnet50_fp32_1b.bmodel --dev_id 0

After execution, predicted results will be saved in results/resnet50_fp32_1b.bmodel_img_opencv_python_result.json, and predicted results, inference time and other information will be printed. Output is as follows:

2. C++ Examples

1. Cross-compilation Environment Setup

C++ programs need to compile dependency files to run on the board. To save pressure on edge devices, we choose to use an X86 Linux environment for cross-compilation.

Setting up cross-compilation environment, two methods provided:

(1) Install cross-compilation toolchain via apt:

If your system and target SoC platform have the same libc version (can be queried via ldd --version command), you can install using the following command:

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

Uninstall method:

sudo apt remove cpp-*-aarch64-linux-gnu

If your environment does not meet the above requirements, it is recommended to use method (2).

(2) Set up cross-compilation environment via docker:

You can use the provided docker image -- stream_dev.tar as the cross-compilation environment.

If using Docker for the first time, execute the following commands to install and configure (only required for first time):

sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Load the image in the downloaded image directory

docker load -i stream_dev.tar

You can view loaded images via docker images, default is stream_dev:latest

Create container

docker run --privileged --name stream_dev -v $PWD:/workspace  -it stream_dev:latest
# stream_dev is just an example name, please specify your own container name

The workspace directory in the container will mount to the host directory where you run docker run. You can compile projects in this container. The workspace directory is under root, changes in this directory will map to changes in corresponding files in the local directory.

Note: When creating a container, you need to go to the parent directory of soc-sdk (dependency compilation environment) and above

1.2 Package Dependency Files

1. Package libsophon

   Extract libsophon_soc_x.y.z_aarch64.tar.gz, where x.y.z is the version number.

   # Create root directory for dependency files
   mkdir -p soc-sdk
   # Extract libsophon_soc_x.y.z_aarch64.tar.gz
   tar -zxf libsophon_soc_${x.y.z}_aarch64.tar.gz
   # Copy related library directories and header file directories to the dependency root directory
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/lib soc-sdk
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/include soc-sdk

2. Package sophon-ffmpeg and sophon-opencv

   Extract sophon-mw-soc_x.y.z_aarch64.tar.gz, where x.y.z is the version number.

   # Extract sophon-mw-soc_x.y.z_aarch64.tar.gz
   tar -zxf sophon-mw-soc_${x.y.z}_aarch64.tar.gz
   # Copy ffmpeg and opencv library directories and header file directories to soc-sdk directory
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/include soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/include soc-sdk

1.3 Perform Cross-compilation

After setting up the cross-compilation environment, use the cross-compilation toolchain to compile and generate executable files. resnet_opencv and resnet_bmcv have the same compilation method. Taking compiling the resnet_opencv program as an example:

cd cpp/resnet_opencv
mkdir build && cd build
#Please modify -DSDK path according to actual situation, use absolute path.
cmake -DTARGET_ARCH=soc -DSDK=/workspace/soc-sdk/ ..
make

After compilation completes, a .soc file will be generated in the corresponding directory, for example: cpp/resnet_opencv/resnet_opencv.soc.

2. Image Testing

You need to copy the executable files generated from cross-compilation and required models and test data to the SoC platform (i.e., BM1684X development board) for testing.

Parameter Description

The executable program has a default set of parameters. Please pass parameters according to actual situation. resnet_bmcv.soc and resnet_opencv.soc have the same parameters. Specific parameters are as follows:

usage:./resnet_opencv.soc [params]

        --bmodel (value:../../models/BM1684X/resnet50_fp32_1b.bmodel)
                bmodel file path
        --dev_id (value:0)
                TPU device id
        --help (value:0)
                print help information.
        --input (value:../../datasets/imagenet_val_1k/img)
                input path, images direction or video file path

Image Testing

Image testing example is as follows. Supports testing the entire image folder.

##First add executable permission to the file
chmod +x resnet_opencv.soc
./resnet_opencv.soc --input=../../datasets/img --bmodel=../../models/BM1684X/resnet50_fp32_1b.bmodel --dev_id=0

After execution, predicted results will be saved in results/resnet50_fp32_1b.bmodel_img_opencv_cpp_result.json, and predicted results, inference time and other information will be printed. Output is as follows: