LPRNet (License Plate Recognition)

1. Introduction

LPRNet (License Plate Recognition via Deep Neural Networks) is a lightweight convolutional neural network that enables end-to-end license plate recognition without character segmentation.

The advantages of LPRNet can be summarized in three points: (1) LPRNet does not require pre-segmentation of characters; license plate recognition accuracy is high, the algorithm is highly real-time, and variable-length character license plate recognition is supported. It can be trained end-to-end for license plates of various countries with significantly different characters. (2) LPRNet is the first real-time lightweight OCR algorithm that does not use RNN; it can run on a variety of devices, including embedded devices. (3) LPRNet is robust enough to perform well under complex conditions such as viewing angle and camera distortion, poor lighting, and viewpoint changes.

Project Directory

LPRNET
│  libsophon_soc_0.5.1-LTS_aarch64.tar.gz	#编译所需依赖
│  sophon-mw-soc_0.12.0_aarch64.tar.gz		#编译所需依赖
│  stream_dev.tar							#交叉编译环境镜像
│
├─cpp		##C++例程所需文件
│  ├─dependencies
│  │  ├─include
│  │  │      bmnn_utils.h
│  │  │      bm_wrapper.hpp
│  │  │      ff_decode.hpp
│  │  │      json.hpp
│  │  │      utils.hpp
│  │  │
│  │  └─src
│  │          ff_decode.cpp
│  │
│  ├─lprnet_bmcv	##使用bmcv所需文件
│  │      CMakeLists.txt
│  │      lprnet.cpp
│  │      lprnet.hpp
│  │      main.cpp
│  │
│  └─lprnet_opencv	##使用opencv所需文件
│          CMakeLists.txt
│          lprnet.cpp
│          lprnet.hpp
│          main.cpp
│
├─datasets	##测试数据集
│  └─test
│
├─docs		#教程说明文档
│  │  LPRNet.md
│  │
│  └─images
│
├─models	#模型权重
│  └─BM1684X
│          lprnet_fp16_1b.bmodel
│          lprnet_fp32_1b.bmodel
│          lprnet_int8_1b.bmodel
│          lprnet_int8_4b.bmodel
│
├─python	#python例程所需文件
│      lprnet_bmcv.py
│      lprnet_opencv.py
│
└─tools		#开发中可能需要的工具
        compare_statis.py	##比较测试结果
        convert_imageset.py	##转换数据集
        eval_ccpd.py	##精度测量
        export_onnx.py	##导出onnx模型
        LPRNet.py

2. Running Steps

1. Python Example

1.1 Configure the Python Environment

bmcv environment (for running lprnet_bmcv.py)

Edit the .bashrc file to import the sophon Python environment:

sudo vim ~/.bashrc

Add the following line at the end of the file:

export PYTHONPATH=$PYTHONPATH:/opt/sophon/libsophon-current/lib:/opt/sophon/sophon-opencv-latest/opencv-python/

After :wq to save and exit, reload the terminal:

source ~/.bashrc

You can run echo $PYTHONPATH to verify the field.

opencv environment (for running lprnet_opencv.py)

pip install opencv-python-headless

1.2 Inference Test

File Parameter Description

lprnet_opencv.py and lprnet_bmcv.py take the same command-line arguments. Using lprnet_opencv.py as an example, the parameters are:

usage: lprnet_opencv.py [--input INPUT_PATH] [--bmodel BMODEL] [--dev_id DEV_ID]

--input: 测试数据路径，可输入整个图片文件夹的路径；
--bmodel: 用于推理的bmodel路径，默认使用stage 0的网络进行推理；
--dev_id: 用于推理的tpu设备id；

Image Test

The following is an image test example; testing an entire image folder is supported. Enter the corresponding script directory, e.g. /data/LPRNet/python/, to debug the files.

 python3 lprnet_bmcv.py --input ../datasets/test --bmodel ../models/BM1684X/lprnet_fp32_1b.bmodel --dev_id 0

After execution, the prediction results are saved under results/lprnet_fp32_1b.bmodel_test_bmcv_python_result.json, and the prediction results, inference time, and other information are printed. The output is as follows:

2. C++ Example

1. Cross-Compilation Environment Setup

1.1 Build Environment

C++ programs need their dependencies compiled to run on the board. To reduce load on the edge device, we use an x86 Linux environment for cross-compilation.

Two ways to set up the cross-compilation environment are provided:

(1) Install the cross-compilation toolchain via apt:

If your system's libc version matches the target SoC platform's libc version (check with ldd --version), you can install with:

sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu

To uninstall:

sudo apt remove cpp-*-aarch64-linux-gnu

If your environment does not meet the above requirements, method (2) is recommended.

(2) Set up the cross-compilation environment via Docker:

You can use the Docker image we provide — stream_dev.tar — as the cross-compilation environment.

If you are using Docker for the first time, run the following commands to install and configure it (only needed once):

sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker

Load the image from the downloaded image directory:

docker load -i stream_dev.tar

You can view the loaded image with docker images; it is named stream_dev:latest by default.

Create a container:

docker run --privileged --name stream_dev -v $PWD:/workspace  -it stream_dev:latest
# stream_dev只是举个名字的例子, 请指定成自己想要的容器的名字

The workspace directory inside the container is mounted to the host directory where you ran docker run; you can compile the project inside this container. The workspace directory is under the root directory, and changes in it are mapped to the corresponding files in the local directory.

Note: When creating the container, you must be in the parent directory of soc-sdk (the dependency build environment) or above.

1.2 Package Dependency Files

1. Package libsophon

   For libsophon_soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 创建依赖文件的根目录
   mkdir -p soc-sdk
   # 解压libsophon_soc_x.y.z_aarch64.tar.gz
   tar -zxf libsophon_soc_${x.y.z}_aarch64.tar.gz
   # 将相关的库目录和头文件目录拷贝到依赖文件根目录下
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/lib soc-sdk
   cp -rf libsophon_soc_${x.y.z}_aarch64/opt/sophon/libsophon-${x.y.z}/include soc-sdk

2. Package sophon-ffmpeg and sophon-opencv

   For sophon-mw-soc_x.y.z_aarch64.tar.gz (where x.y.z is the version number), decompress it.

   # 解压sophon-mw-soc_x.y.z_aarch64.tar.gz
   tar -zxf sophon-mw-soc_${x.y.z}_aarch64.tar.gz
   # 将ffmpeg和opencv的库目录和头文件目录拷贝到soc-sdk目录下
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-ffmpeg_${x.y.z}/include soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/lib soc-sdk
   cp -rf sophon-mw-soc_${x.y.z}_aarch64/opt/sophon/sophon-opencv_${x.y.z}/include soc-sdk

1.3 Perform Cross Compilation

After the cross-compilation environment is set up, use the cross-compilation toolchain to build the executable. lprnet_opencv and lprnet_bmcv use the same build method; here we use lprnet_opencv as an example:

cd cpp/lprnet_opencv
mkdir build && cd build
#请根据实际情况修改-DSDK的路径，需使用绝对路径。
cmake -DTARGET_ARCH=soc -DSDK=/workspace/soc-sdk/ ..
make

After the build, a .soc file is generated in the corresponding directory, e.g. cpp/lprnet_opencv/lprnet_opencv.soc:

2. Image Test

Copy the cross-compiled executable and the required model and test data to the SoC platform (i.e. the BM1684X development board) for testing.

Parameter Description

Usage: lprnet_opencv.soc [params]

        --bmodel (value:../../models/BM1684/lprnet_fp32_1b.bmodel)
                bmodel file path
        --dev_id (value:0)
                TPU device id
        --help (value:true)
                print help information.
        --input (value:../../datasets/test)
                input path, images directory

Image Test

The following is an image test example; testing an entire image folder is supported.

##先对文件加上可执行权限
chmod +x lprnet_opencv.soc
./lprnet_opencv.soc --input=../../datasets/test --bmodel=../../models/BM1684X/lprnet_fp32_1b.bmodel --dev_id=0

After execution, the prediction results are saved under results/lprnet_fp32_1b.bmodel_test_opencv_cpp_result.json, and the prediction results, inference time, and other information are printed. The output is as follows: