Sophgo SDK Development

1. SDK Introduction

SOPHONSDK is a deep learning SDK customized by Sophgo Technologies based on BM1684 and BM1684X. It covers the capabilities required for neural network inference, including model optimization and efficient runtime support, providing an easy-to-use and efficient full-stack solution for deep learning application development and deployment.

1. Basic Tool Package

The basic tool package includes:

tpu-nntc: Responsible for offline compilation and optimization of neural network models trained in third-party deep learning frameworks, generating BModel files required for final runtime. Currently supports Caffe, Darknet, MXNet, ONNX, PyTorch, PaddlePaddle, TensorFlow, etc.
libsophon: Provides libraries such as BMCV, BMRuntime, and BMLib, used to drive VPP, intelligent vision deep learning processor modules, and other hardware, completing image processing, tensor operations, model inference, and other operations for deep learning application development.
sophon-mw: Encapsulates libraries such as SOPHON-OpenCV and SOPHON-FFmpeg, used to drive VPU, JPU, and other hardware, supporting RTSP stream and GB28181 stream parsing, video image encoding and decoding acceleration, for deep learning application development.
sophon-sail: Provides advanced Python/C++ interfaces, encapsulating underlying library interfaces such as BMRuntime, BMCV, sophon-mw, and BMLib, for deep learning application development.

2. Advanced Tool Package

The advanced tool package includes:

tpu-mlir: Provides a complete toolchain for the Tensor Processing Unit compiler project, converting pre-trained neural networks from different frameworks into BModel binary files that can run efficiently on Sophgo's intelligent vision deep learning processors. Currently directly supported frameworks include tflite, onnx, and Caffe.
tpu-perf: Provides a complete tool package for model performance and accuracy verification.
tpu-kernel: Tensor Processing Unit underlying development interface, which can not only call dedicated instructions to accelerate deep learning business logic but also call general instructions to implement various algorithm accelerations.

For details, refer to Sophgo Documentation

Remarks

sophon-demo: Provides comprehensive examples for single models or scenarios in both x86 and SoC environments, for reference during deep learning application development.
sophon-pipeline: Provides a simple high-performance inference framework based on pipeline, which can run preprocessing/inference/postprocessing on 3 threads respectively, maximizing parallel execution, for reference during deep learning application development.

3. SOPHONSDK Folder Directory

Folder	Remarks
libsophon	Libraries for image processing, tensor operations, model inference, etc. with SOPHON device hardware acceleration
sophon-mw	Multimedia library with SOPHON device hardware acceleration
tpu-kernel	Underlying development interface
tpu-mlir	Tensor Processing Unit compiler toolchain
tpu-nntc	Tensor Processing Unit compiler toolchain
tpu-perf	Model performance and accuracy verification tool package
sophon-pipeline	High-performance inference framework based on pipeline
sophon-img	SoC mode installation package
sophon-demo	Comprehensive examples for single models or scenarios
sophon-sail	Interface library encapsulating underlying interfaces in C++/Python API
sophon-rpc	Offloads some computing tasks to the ARM on the card in PCIe card products
docker-image	Docker image

4. SOPHONSDK Extracted Folder Directory

As follows:

    SOPHONSDK
    ├── docker-image
    │   ├── md5sum.txt
    │   ├── sophgo-tpuc_dev-v2.1-82d75f5c633d.tar.bz2
    │   └── sophgo-tpuc_dev-v2.2-f72913f3a83d.tar.bz2
    ├── libsophon_20240108_210425
    │   ├── BMCV Development Reference Manual.pdf
    │   ├── BMCV Technical Reference Manual.pdf
    │   ├── BMLIB Development Reference Manual.pdf
    │   ├── BMLib Technical Reference Manual.pdf
    │   ├── BMRUNTIME Development Reference Manual.pdf
    │   ├── BMRuntime Technical Reference Manual.pdf
    │   ├── centos
    │   ├── libsophon_0.4.9-LTS_aarch64.tar.gz
    │   ├── libsophon_0.4.9-LTS_loongarch64.tar.gz
    │   ├── libsophon_0.4.9-LTS_x86_64.tar.gz
    │   ├── LIBSOPHON User Manual.pdf
    │   ├── libsophon_dockerfile
    │   ├── libsophon.MD5
    │   ├── LIBSOPHON_User_Guide.pdf
    │   ├── release_version.txt
    │   ├── sophon-driver_0.4.9-LTS_amd64.deb
    │   ├── sophon-driver_0.4.9-LTS_arm64.deb
    │   ├── sophon-libsophon_0.4.9-LTS_amd64.deb
    │   ├── sophon-libsophon_0.4.9-LTS_arm64.deb
    │   ├── sophon-libsophon-dev_0.4.9-LTS_amd64.deb
    │   └── sophon-libsophon-dev_0.4.9-LTS_arm64.deb
    ├── sophon-demo_20231116_085900
    │   ├── release_version.txt
    │   ├── sophon-demo.MD5
    │   ├── sophon-demo_v0.1.8_dbb4632_20231116
    │   └── sophon-demo_v0.1.8_dbb4632_20231116.tar.gz
    ├── sophon-img_20240116_212937
    │   ├── bsp-debs
    │   ├── bsp_update.tgz
    │   ├── libsophon_soc_0.4.9-LTS_aarch64.tar.gz
    │   ├── release_version.txt
    │   ├── sdcard.tgz
    │   ├── SOPHON BSP Development Reference Manual.pdf
    │   ├── SOPHON_BSP_Technical_Reference_Manual.pdf
    │   ├── sophon-img.MD5
    │   ├── system.tgz
    │   └── tftp.tgz
    ├── sophon-mw_20240116_152830
    │   ├── MULTIMEDIA FAQ Manual.pdf
    │   ├── MULTIMEDIA Development Reference Manual.pdf
    │   ├── MULTIMEDIA User Manual.pdf
    │   ├── Multimedia FAQ.pdf
    │   ├── Multimedia Technical Reference Manual.pdf
    │   ├── Multimedia User Guide.pdf
    │   ├── release_version.txt
    │   ├── sophon-mw_0.8.0_aarch64.tar.gz
    │   ├── sophon-mw_0.8.0_loongarch64.tar.gz
    │   ├── sophon-mw_0.8.0_x86_64.tar.gz
    │   ├── sophon-mw.MD5
    │   ├── sophon-mw-soc_0.8.0_aarch64.tar.gz
    │   ├── sophon-mw-soc-sophon-ffmpeg_0.8.0_arm64.deb
    │   ├── sophon-mw-soc-sophon-ffmpeg-dev_0.8.0_arm64.deb
    │   ├── sophon-mw-soc-sophon-opencv_0.8.0_arm64.deb
    │   ├── sophon-mw-soc-sophon-opencv-dev_0.8.0_arm64.deb
    │   ├── sophon-mw-soc-sophon-sample_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-ffmpeg_0.8.0_amd64.deb
    │   ├── sophon-mw-sophon-ffmpeg_0.8.0_amd64.rpm
    │   ├── sophon-mw-sophon-ffmpeg_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-ffmpeg_0.8.0_arm64.rpm
    │   ├── sophon-mw-sophon-ffmpeg_0.8.0_loongarch64.deb
    │   ├── sophon-mw-sophon-ffmpeg-dev_0.8.0_amd64.deb
    │   ├── sophon-mw-sophon-ffmpeg-dev_0.8.0_amd64.rpm
    │   ├── sophon-mw-sophon-ffmpeg-dev_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-ffmpeg-dev_0.8.0_arm64.rpm
    │   ├── sophon-mw-sophon-ffmpeg-dev_0.8.0_loongarch64.deb
    │   ├── sophon-mw-sophon-opencv_0.8.0_amd64.deb
    │   ├── sophon-mw-sophon-opencv_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-opencv_0.8.0_loongarch64.deb
    │   ├── sophon-mw-sophon-opencv-abi0_0.8.0_amd64.rpm
    │   ├── sophon-mw-sophon-opencv-abi0_0.8.0_arm64.rpm
    │   ├── sophon-mw-sophon-opencv-abi0-dev_0.8.0_amd64.rpm
    │   ├── sophon-mw-sophon-opencv-abi0-dev_0.8.0_arm64.rpm
    │   ├── sophon-mw-sophon-opencv-dev_0.8.0_amd64.deb
    │   ├── sophon-mw-sophon-opencv-dev_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-opencv-dev_0.8.0_loongarch64.deb
    │   ├── sophon-mw-sophon-sample_0.8.0_amd64.deb
    │   ├── sophon-mw-sophon-sample_0.8.0_amd64.rpm
    │   ├── sophon-mw-sophon-sample_0.8.0_arm64.deb
    │   ├── sophon-mw-sophon-sample_0.8.0_arm64.rpm
    │   └── sophon-mw-sophon-sample_0.8.0_loongarch64.deb
    ├── sophon-rpc_20231208_174527
    │   ├── release_version.txt
    │   ├── sophon-rpc_3.2.0-LTS_amd64.deb
    │   ├── sophon-rpc_3.2.0-LTS_amd64.rpm
    │   ├── sophon-rpc_3.2.0-LTS_arm64.deb
    │   ├── sophon-rpc_3.2.0-LTS_arm64.rpm
    │   ├── sophon-rpc_3.2.0-LTS.tar.gz
    │   ├── sophon-rpc User Guide.pdf
    │   └── sophon-rpc.MD5
    ├── sophon-sail_20231116_085400
    │   ├── release_version.txt
    │   ├── sophon-sail_3.7.0.tar.gz
    │   ├── sophon-sail_en.pdf
    │   ├── sophon-sail.MD5
    │   └── sophon-sail_zh.pdf
    ├── sophon-stream_20231116_011200
    │   ├── release_version.txt
    │   ├── sophon-stream.MD5
    │   └── sophon-stream_v0.0.4-rc4_10a8ed8_20231115.tar.gz
    ├── tpu-kernel_20231130_055600
    │   ├── release_version.txt
    │   ├── tpu-kernel-1684x_v3.1.7-520261d8-231130.tar.gz
    │   └── tpu-kernel.MD5
    ├── tpu-mlir_20231116_054500
    │   ├── release_version.txt
    │   ├── tpu-mlir.MD5
    │   └── tpu-mlir_v1.3.140-g3180ff37-20231116.tar.gz
    ├── tpu-nntc_20231130_054100
    │   ├── release_version.txt
    │   ├── tpu-nntc.MD5
    │   └── tpu-nntc_v3.1.9-29fa956b-231130.tar.gz
    └── tpu-perf_v1.2.37
        ├── tpu_perf-1.2.37-py3-none-manylinux2014_aarch64.whl
        ├── tpu_perf-1.2.37-py3-none-manylinux2014_x86_64.whl
        ├── tpu-perf-v1.2.37.tar.gz
        └── tpu-perf-v1.2.37.zip

5. SDK Main Modules

Hardware Driver and Runtime Library LIBSOPHON: Contains libraries such as BMCV, BMRuntime, BMLib, used to drive VPP, intelligent vision deep learning processors, and other hardware, completing image processing, tensor operations, model inference, and other operations.
Multimedia Library SOPHON-MW: SOPHON-OpenCV and SOPHON-FFmpeg with SOPHON device hardware acceleration, supporting RTSP stream and GB28181 stream parsing, video and image encoding and decoding.
Model Compilation and Quantization Tool Chain TPU-MLIR: Supports model conversion for Caffe, TFLite, ONNX and other frameworks; supports model quantization: original model -> MLIR Model -> FP32 BModel or original model -> MLIR Model -> cali_table -> INT8 BModel, while providing model_deploy.py quantization script.
Model Compilation and Quantization Tool Chain TPU-NNTC: Supports model conversion for Caffe, Tensorflow, Pytorch, MXNet, Darknet, Paddle Paddle, ONNX and other frameworks; supports model quantization: original model -> FP32 UModel -> INT8 UModel -> INT8 BModel, while providing auto-cali automatic quantization tool.
Tensor Operations and Image Processing Library BMCV: Color space conversion, scale transformation, affine transformation, projection transformation, linear transformation, drawing boxes, JPEG encoding, BASE64 encoding, NMS, sorting, feature matching.
Device Management BMLib: Basic Interface: Device handle management, memory management, data transfer, API sending and synchronization, A53 enable, power management, etc.
Sophgo Deep Learning Processing Acceleration Library SAIL: Advanced Python/C++ interface, encapsulating underlying library interfaces such as BMRuntime, BMCV, sophon-mw.
Custom Operator Advanced Programming Library BMLang: C++ based advanced programming library for SOPHON intelligent vision deep learning processors, decoupled from hardware information, no need to understand hardware architecture, using tensor data (bmlang::Tensor) and computing operations (bmlang::Operator) to write code, finally using bmlang::compile or bmlang::compile_with_check to generate BModels that can run on intelligent vision deep learning processors; also supports using the ARM processor in BM168X to implement operators not yet supported by intelligent vision deep learning processors.
Algorithm Parallel Acceleration Programming Library TPUKernel: Underlying programming interface based on SOPHON BM1684, BM1684X underlying atomic operation interfaces, requires users to be familiar with device hardware architecture and instruction set.
Model Performance and Accuracy Verification Tool TPUPerf: Can perform performance analysis and accuracy verification on models.

2. SDK Acquisition and Installation

For details

Obtaining v24.04.01 SDK

Installing v24.04.01 SDK

3. More Information

For details

Sophgo Documentation