Sophgo Demo Introduction

1. SOPHON-DEMO Introduction

SOPHON-DEMO is developed based on the SOPHON SDK interface and provides a series of porting examples for mainstream algorithms. It includes model compilation and quantization based on TPU-NNTC and TPU-MLIR, inference engine porting based on BMRuntime, and pre/post-processing algorithm porting based on BMCV/OpenCV.

The SOPHON SDK is a deep-learning SDK customized by Sophgo for its self-developed deep-learning processor. It covers model optimization and efficient runtime support required for the neural-network inference phase, and provides an easy-to-use, efficient, full-stack solution for deep-learning application development and deployment. It is currently compatible with BM1684/BM1684X/BM1688 (CV186X). Below are some related term definitions:

TermNotes
BM1688/CV186AH, BM1684XSophgo's fifth-generation tensor processor for deep learning; Sophgo's fourth-generation tensor processor for deep learning
BM1684Sophgo's third-generation tensor processor for deep learning
Intelligent-vision deep-learning processorThe neural-network compute unit inside BM1688/CV186AH and BM1684/BM1684X
VPUThe codec unit inside BM1688/CV186AH and BM1684/BM1684X
VPPThe graphics compute acceleration unit inside BM1684/BM1684X
VPSSThe video processing subsystem inside BM1688/CV186AH, including the graphics compute acceleration unit and decode unit; also called VPP
JPUThe image JPEG codec unit inside BM1688/CV186AH and BM1684/BM1684X
SOPHON SDKSophgo's original deep-learning development kit based on BM1688/CV186AH and BM1684/BM1684X
PCIe ModeA working mode of BM1688/CV186AH and BM1684/BM1684X, used as an acceleration device
SoC ModeA working mode of BM1688/CV186AH and BM1684/BM1684X, running standalone as the host; customer algorithms can run directly on it
arm_pcie ModeA working mode of BM1684/BM1684X: a board carrying BM1684/BM1684X is plugged into an ARM-processor server as a PCIe slave device; customer algorithms run on the ARM-processor host
BMCompilerAn optimizing deep neural-network compiler developed for the intelligent-vision deep-learning processor; converts deep neural networks from deep-learning frameworks into instruction streams that run on the processor
BMRuntimeThe intelligent-vision deep-learning processor inference interface library
BMCVThe graphics-compute hardware-acceleration interface library
BMLibA low-level software library layered above the kernel driver: device management, memory management, data transfer, API send, A53 enable, power control
mlirAn intermediate model format generated by TPU-MLIR, used to port or quantize models
BModelA deep neural-network model file format for the intelligent-vision deep-learning processor; contains the target network's weights and instruction streams
BMLangA high-level programming model for the intelligent-vision deep-learning processor; users do not need to know low-level hardware details during development
TPUKernelA development library based on the atomic operations of the intelligent-vision deep-learning processor (an interface set wrapped around the BM1688/CV186AH and BM1684/BM1684X instruction sets)
SAILThe SOPHON Inference inference library with Python/C++ interfaces; further wraps BMCV, sophon-media, BMLib, BMRuntime, etc.
TPU-MLIRThe intelligent-vision deep-learning processor compiler project; converts pretrained neural networks from different frameworks into bmodels that run efficiently on the Sophgo intelligent-vision deep-learning processor

1.1 BModel

BModel: A deep neural-network model file format for the Sophgo intelligent-vision deep-learning processor; contains the target network's weights (weight), instruction streams, etc.

Stage: Supports combining models of the same network with different batch sizes into one BModel; different batch-size inputs of the same network correspond to different stages. At inference time, BMRuntime automatically selects the model of the corresponding stage based on the input shape. It also supports combining different networks into one BModel and retrieving different networks by network name.

Dynamic compilation and static compilation: Supports both dynamic and static compilation of models, configurable via parameters at conversion time. A dynamically compiled BModel supports, at runtime, any input shape not larger than the shape set at compile time; a statically compiled BModel supports, at runtime, only the shape set at compile time.

Note

Prefer statically compiled models: A dynamically compiled model requires the BM168X microcontroller ARM9 at runtime to dynamically generate intelligent-vision deep-learning processor instructions based on the actual input shape. Therefore, dynamically compiled models execute less efficiently than statically compiled models. Where possible, prefer statically compiled models, or statically compiled models that support multiple input shapes.

1.2 bm_image

BMCV: BMCV provides a machine-vision library optimized for the SOPHON deep-learning processor. By leveraging the processor's Tensor Computing Processor and VPP module, it can perform color space conversion, scaling, affine transformation, projection transformation, linear transformation, drawing boxes, JPEG codec, BASE64 codec, NMS, sorting, feature matching, and more.

bm_image: BMCV APIs all revolve around bm_image; one bm_image object corresponds to one image. The user constructs a bm_image object via bm_image_create, passes it to each BMCV function, and calls bm_image_destroy to destroy it after use.

BMImage: The SAIL library wraps bm_image as BMImage; for details, see the SOPHON-SAIL User Manual.

The bm_image struct and related data format definitions are as follows:

typedef enum bm_image_format_ext_{
    FORMAT_YUV420P,
    FORMAT_YUV422P,
    FORMAT_YUV444P,
    FORMAT_NV12,
    FORMAT_NV21,
    FORMAT_NV16,
    FORMAT_NV61,
    FORMAT_RGB_PLANAR,
    FORMAT_BGR_PLANAR,
    FORMAT_RGB_PACKED,
    FORMAT_BGR_PACKED,
    PORMAT_RGBP_SEPARATE,
    PORMAT_BGRP_SEPARATE,
    FORMAT_GRAY,
    FORMAT_COMPRESSED
} bm_image_format_ext;

typedef enum bm_image_data_format_ext_{
    DATA_TYPE_EXT_FLOAT32,
    DATA_TYPE_EXT_1N_BYTE,
    DATA_TYPE_EXT_4N_BYTE,
    DATA_TYPE_EXT_1N_BYTE_SIGNED,
    DATA_TYPE_EXT_4N_BYTE_SIGNED,
}bm_image_data_format_ext;

// bm_image结构体定义如下
struct bm_image {
    int width;
    int height;
    bm_image_format_ext image_format;
    bm_data_format_ext data_type;
    bm_image_private* image_private;
};

2. Directory Structure and Notes

SOPHON-DEMO examples are organized, from easy to advanced, into three modules: tutorial, sample, and application:

Warning

  • The tutorial module holds usage examples of basic interfaces.
  • The sample module holds sequential examples of classic algorithms on the SOPHON SDK.
  • The application module holds typical applications for typical scenarios.
ModuleLink
tutorialLINK1
sampleLINK2
applicationLINK3

3. Version Notes

VersionNotes
0.2.1Improved and fixed documentation and code issues; some examples added CV186X support; YOLOv5 adapted to SG2042; the sample module added GroundingDINO and Qwen1_5 examples; StableDiffusionV1_5 newly supports multiple resolutions; Qwen, Llama2, and ChatGLM3 added web and multi-session modes. The tutorial module added blend and stitch examples.
0.2.0Improved and fixed documentation and code issues; added application and tutorial modules; added ChatGLM3 and Qwen examples; SAM added a web UI; BERT, ByteTrack, and C3D adapted to BM1688; the former YOLOv8 was renamed YOLOv8_det and added a cpp post-processing acceleration method; optimized auto_test for common examples; updated the TPU-MLIR installation method to pip.
0.1.10Fixed documentation and code issues; added ppYoloe, YOLOv8_seg, StableDiffusionV1.5, and SAM; refactored yolact; CenterNet, YOLOX, and YOLOv8 adapted to BM1688; YOLOv5, ResNet, PP-OCR, and DeepSORT added BM1688 performance data; WeNet provided a C++ cross-compilation method.
0.1.9Fixed documentation and code issues; added segformer, YOLOv7, and Llama2 examples; refactored YOLOv34; YOLOv5, ResNet, PP-OCR, DeepSORT, LPRNet, RetinaFace, YOLOv34, and WeNet adapted to BM1688; OpenPose post-processing acceleration; chatglm2 added compilation methods and int8/int4 quantization.
0.1.8Improved and fixed documentation and code issues; added BERT, ppYOLOv3, and ChatGLM2; refactored YOLOX; PP-OCR added beam search; OpenPose added tpu-kernel post-processing acceleration; updated the SFTP download method.
0.1.7Fixed documentation and other issues; some examples support BM1684 mlir; refactored PP-OCR and CenterNet examples; YOLOv5 added sail support.
0.1.6Fixed documentation and other issues; added ByteTrack, YOLOv5_opt, and WeNet examples.
0.1.5Fixed documentation and other issues; added the DeepSORT example; refactored the ResNet and LPRNet examples.
0.1.4Fixed documentation and other issues; added C3D and YOLOv8 examples.
0.1.3Added the OpenPose example; refactored the YOLOv5 example (including adapting to arm PCIe, supporting TPU-MLIR compilation of BM1684X models, and replacing opencv decoding with the ffmpeg component, etc.).
0.1.2Fixed documentation and other issues; refactored SSD-related examples; LPRNet/cpp/lprnet_bmcv replaced opencv decoding with the ffmpeg component.
0.1.1Fixed documentation and other issues; refactored LPRNet/cpp/lprnet_bmcv using BMNN-related classes.
0.1.0Provided 10 examples including LPRNet; adapted to BM1684X (x86 PCIe, SoC) and BM1684 (x86 PCIe, SoC).

4. Environment Dependencies

SOPHON-DEMO mainly depends on TPU-MLIR, TPU-NNTC, LIBSOPHON, SOPHON-FFMPEG, SOPHON-OPENCV, and SOPHON-SAIL, with version requirements as follows:

SOPHON-DEMOTPU-MLIRTPU-NNTCLIBSOPHONSOPHON-FFMPEGSOPHON-OPENCVSOPHON-SAILRelease date
0.2.0>=1.6>=3.1.7>=0.5.0>=0.7.3>=0.7.3>=3.7.0>=23.10.01
0.1.10>=1.2.2>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.7.0>=23.07.01
0.1.9>=1.2.2>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.7.0>=23.07.01
0.1.8>=1.2.2>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.6.0>=23.07.01
0.1.7>=1.2.2>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.6.0>=23.07.01
0.1.6>=0.9.9>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.4.0>=23.05.01
0.1.5>=0.9.9>=3.1.7>=0.4.6>=0.6.0>=0.6.0>=3.4.0>=23.03.01
0.1.4>=0.7.1>=3.1.5>=0.4.4>=0.5.1>=0.5.1>=3.3.0>=22.12.01
0.1.3>=0.7.1>=3.1.5>=0.4.4>=0.5.1>=0.5.1>=3.3.0-
0.1.2Not support>=3.1.4>=0.4.3>=0.5.0>=0.5.0>=3.2.0-
0.1.1Not support>=3.1.3>=0.4.2>=0.4.0>=0.4.0>=3.1.0-
0.1.0Not support>=3.1.3>=0.3.0>=0.2.4>=0.2.4>=3.1.0-

Warning

  1. Different examples may have different version requirements; refer to each example's README. Other third-party libraries may need to be installed.

  2. The SDK for BM1688/CV186X differs from that for BM1684X/BM1684 and is not yet published on the official website; contact technical staff to obtain it.

5. Technical Materials

Tips

Please obtain relevant documents, materials, and video tutorials from the Sophgo official site Technical Materials.

6. Community

Tips

The Sophgo community encourages developers to communicate and learn together. Developers can communicate and learn through the following channels.

Sophgo community website: https://www.sophgo.com/

Sophgo developer forum: https://developer.sophgo.com/forum/index.html