RK3568 NPU Overview

1 What is an NPU

1.1 Basic NPU Concepts

An NPU (Neural Processing Unit) is a dedicated processor designed specifically for artificial intelligence and machine learning tasks. Compared with traditional CPUs and GPUs, NPUs have higher energy efficiency and lower power consumption when performing neural network inference tasks.

1.2 Technical Features of NPU

Dedicated Architecture Design

• Parallel computing capability: Optimized for the matrix operations used in neural networks
• Low-power design: Lower power consumption than GPUs for inference tasks
• Efficient memory access: Optimized memory hierarchy reduces data movement
• Fixed-point arithmetic support: Supports low-precision operations such as INT8/INT16 to speed up inference

Application Scenarios

• Computer vision: image classification, object detection, face recognition
• Natural language processing: speech recognition, text analysis
• Intelligent control: industrial automation, robot control
• Edge computing: IoT devices, smart surveillance

1.3 NPU vs CPU/GPU Comparison

Feature	CPU	GPU	NPU
Architecture	General-purpose computing	Parallel computing	AI-dedicated computing
Inference performance	Low	Medium	High
Power efficiency	Low	Medium	High
Programming complexity	Simple	Medium	Simple (framework-supported)
Applicable scenarios	General tasks	Graphics/parallel computing	AI inference

2 RK3568 NPU Specifications

2.1 Hardware Specifications

Basic Parameters

• NPU model: Rockchip self-developed NPU
• Compute performance: 0.8 TOPS (INT8)
• Supported precision: INT8, INT16, FP16, BFP16
• Memory bandwidth: Shared system memory
• Operating frequency: Up to 600MHz

Architecture Features

RK3568 NPU 架构
├── 计算单元
│   ├── 矩阵乘法单元 (MAC Array)
│   ├── 激活函数单元 (Activation)
│   └── 池化单元 (Pooling)
├── 内存子系统
│   ├── 片上缓存 (On-chip Cache)
│   ├── DMA 控制器
│   └── 内存管理单元 (MMU)
└── 控制单元
    ├── 指令解码器
    ├── 调度器
    └── 中断控制器

2.2 Performance Benchmarks

Typical model performance (INT8)

Model	Input size	Inference time	FPS	Accuracy loss
MobileNetV2	224x224x3	~15ms	~66	< 1%
YOLOv5s	640x640x3	~180ms	~5.5	< 2%
ResNet50	224x224x3	~45ms	~22	< 1%
EfficientNet-B0	224x224x3	~25ms	~40	< 1%

Power characteristics

• Peak power: approximately 1.2W
• Average power: approximately 0.8W (typical inference task)
• Standby power: < 10mW
• Power efficiency: approximately 667 GOPS/W

2.3 Supported Operators

① Convolution operators

These operators are the core of deep learning, especially computer vision tasks, used to extract features from input data (such as images).

- Conv2D             (标准卷积)
- DepthwiseConv2D    (深度可分离卷积)
- TransposeConv2D    (转置卷积)
- DilatedConv2D      (空洞卷积)

• Conv2D: Standard convolution operation that uses a learnable convolution kernel (filter) to perform a sliding-window computation over the input feature map, extracting local features such as edges and textures.
• DepthwiseConv2D: Decomposes a standard convolution into two steps: depthwise convolution (an independent convolution per input channel) and pointwise convolution (a 1x1 convolution used to combine channel information). This structure can greatly reduce computation and the number of parameters.
• TransposeConv2D: Can be regarded as the "inverse" of standard convolution. It can upsample (enlarge) a small feature map into a larger one.
• DilatedConv2D: Inserts "holes" (zeros) between the elements of a standard convolution kernel, thereby enlarging the receptive field of the kernel without increasing the number of parameters or computation, capturing broader contextual information.

② Pooling and normalization

These operators are mainly used for dimensionality reduction, preserving translation invariance (such as in image classification tasks), and stabilizing the training process.

- MaxPool2D / AvgPool2D          (最大池化 / 平均池化)
- GlobalMaxPool / GlobalAvgPool  (全局最大池化 / 全局平均池化)
- BatchNormalization             (批量归一化)
- LayerNormalization             (层归一化)

• MaxPool2D / AvgPool2D: Takes the maximum (MaxPool) or average (AvgPool) value within a local region (such as a 2x2 window). Mainly used to reduce the spatial dimensions (width and height) of the feature map, reducing computation while enhancing the positional invariance of features.
• GlobalMaxPool / GlobalAvgPool: Takes the maximum or average value of each channel across the entire feature map, directly converting an HxWxC feature map into a 1x1xC vector, used for extracting global features.
• BatchNormalization: Normalizes data within a batch (subtracting the mean and dividing by the standard deviation) so its mean is 0 and variance is 1. Then it scales and shifts via learnable parameters. This accelerates model training convergence, mitigates vanishing/exploding gradient problems, and has a certain regularization effect. Used to stabilize the training process and accelerate convergence.
• LayerNormalization: Similar to batch normalization, but its normalization dimension differs. It normalizes all channels and spatial positions within a single sample. It performs better in sequence models (such as Transformer) and small-batch training, used to normalize the features of each sample.

③ Activation functions

Activation functions introduce non-linearity into neural networks, helping models learn complex feature representations.

- ReLU / ReLU6 / LeakyReLU    (修正线性单元)
- Sigmoid / Tanh              (S 型函数 / 双曲正切函数)
- Swish / Mish                (平滑 ReLU / Mish 激活函数)
- Softmax                     (软最大函数)

• ReLU: Rectified Linear Unit, a max-activation function that sets all negative values to 0 and keeps positive values unchanged. Commonly used in hidden layers and solves the vanishing-gradient problem.
• ReLU6: Similar to ReLU, but limits the output to the [0, 6] range; used for mobile-side deployment.
• LeakyReLU: An improvement on ReLU that solves the "dying ReLU" problem — when the input is negative, it outputs a small non-zero value, preventing neurons from "dying".
• Sigmoid: Maps the input to the (0, 1) range; commonly used in the output layer of binary classification problems.
• Tanh: Maps the input to the (-1, 1) range; similar to Sigmoid but with a wider output range.
• Swish: A smooth activation function defined as f(x) = x * sigmoid(x); performs well in some models.
• Mish: A newer activation function defined as f(x) = x * tanh(softplus(x)); also shows good performance in some models.
• Softmax: Maps an input vector to a probability distribution; commonly used in the output layer of multi-class classification problems. It converts each element into a probability value between 0 and 1, and the sum of all elements is 1.

④ Other operators

The following are the basic operations and connection operators necessary for building complex network structures.

- Add / Sub / Mul / Div    (基本算术运算)
- Concat / Split           (拼接与分割)
- Reshape / Transpose      (形状变换)
- MatMul / FullyConnected  (矩阵乘法 / 全连接层)

• Add / Sub / Mul / Div: Correspond to addition, subtraction, multiplication, and division respectively.
• Concat: Used to merge multiple tensors by concatenating along a specified dimension.
• Split: Used to split a tensor into multiple sub-tensors along a specified dimension.
• Reshape: Used to change the shape of a tensor without changing the number of elements.
• Transpose: Used to swap the order of tensor dimensions.
• MatMul: Used for matrix multiplication; mainly used to build linear-transformation layers in neural networks.
• FullyConnected: Used for fully connected layers; multiplies the input tensor by a weight matrix and then adds a bias term.

3 RKNN Software Stack Ecosystem Overview

3.1 RKNN Software Stack Architecture

应用层
├── Python 应用   (rknn-toolkit2)
├── C/C++ 应用    (rknnrt)
└── Android 应用  (RKNN API)
    │
框架层
├── RKNN-Toolkit2 (模型转换)
├── RKNN Runtime  (推理引擎)
└── RKNN API      (编程接口)
    │
驱动层
├── NPU 驱动   (Kernel Driver)
├── 内存管理   (Memory Manager)
└── 电源管理   (Power Manager)
    │
硬件层
└── RK3568 NPU 硬件

3.2 Core Components

1) RKNN-Toolkit2

The core function of RKNN-Toolkit2 is to serve as a bridge for model conversion and deployment. It supports converting models trained in multiple mainstream frameworks (such as TensorFlow, PyTorch, ONNX, etc.) into the dedicated RKNN format. During conversion, the tool automatically performs quantization and graph optimization, significantly improving the model's inference efficiency on the NPU. It also provides model simulation and performance-analysis functions, making it easy for developers to verify model correctness and execution speed before deployment. The tool runs on Windows, Linux, and macOS, providing good cross-platform compatibility.

Supported frameworks:

# 支持的输入格式
- TensorFlow / TensorFlow Lite
- PyTorch / ONNX
- Caffe / Caffe2
- MXNet
- Darknet

2) RKNN Runtime

Core functions:

• Efficient model inference engine
• Memory management and optimization
• Multi-threaded support
• Hardware resource scheduling

To meet the needs of different development scenarios, RKNN Runtime provides multi-level API language support. For embedded deployment scenarios that demand extreme performance and low latency, the C/C++ API is the best choice, offering the most direct and efficient low-level control. During the rapid prototyping of algorithms, scientific research, and script development, the Python API is favored for its concise syntax and fast iteration, greatly improving development convenience. In addition, for application development on the Android platform, it provides a Java API, making it convenient for developers to integrate AI capabilities into existing Android applications.

3.3 Development Toolchain

PC-side tools

# RKNN-Toolkit2 安装
pip install rknn-toolkit2

# 模型转换工具
rknn-toolkit2-convert

# 性能分析工具
rknn-toolkit2-profiler

Board-side runtime

# RKNN Runtime 库
librknnrt.so

# Python 绑定
rknnlite

# 示例程序
rknn_demo

3.4 Ecosystem Support

• GitHub repository: https://github.com/rockchip-linux/rknn-toolkit2
• Development documentation: Complete API documentation and user guides
• Sample code: Demos covering a variety of application scenarios
• Model library: Pre-trained models and conversion scripts

The RK3568 NPU has a mature development ecosystem led officially with an active community. Its core resources are concentrated in the official GitHub repository (rockchip-linux/rknn-toolkit2), which provides a complete software development kit, including detailed API documentation, user guides, sample code covering image classification, object detection, semantic segmentation, and other applications, as well as a continuously updated model library containing a large number of pre-trained RKNN models and conversion scripts to help developers get started quickly.

4 Development Workflow Overview

The first phase is "Development Environment Preparation". Here two foundational tasks must be completed: first, prepare the original model file (such as .pt or .onnx format) trained and exported by mainstream frameworks such as PyTorch or TensorFlow; second, configure the core model-conversion toolchain, i.e., install the RKNN-Toolkit2 SDK and its related dependency environment.

Next comes the second phase, "Model Verification", a key step in ensuring the model can run correctly and efficiently. First use RKNN-Toolkit2 to convert the original model into the RKNN format dedicated to the NPU. This process typically includes the critical quantization step, intended to optimize the model size and inference speed. Then perform simulation verification on the PC side — you can quickly check the functional correctness and basic performance of the converted model without connecting actual hardware, greatly improving development and debugging efficiency.

The final phase is "Deployment and Integration", deploying the verified RKNN model onto the target hardware. In this phase, board-side deployment is performed first to ensure the model is correctly loaded in the real environment; then through performance analysis and optimization, parameters are fine-tuned to fully release the NPU's computing power; finally, the optimized model is integrated into the final application, completing the implementation of the entire AI solution.

5 Summary

The RK3568's NPU provides powerful computing capability and complete software ecosystem support for edge AI applications. Through the RKNN software stack, developers can easily deploy a variety of deep learning models onto the MB-E30P development board to implement efficient AI inference applications.

The following chapters describe in detail the specific operational steps for setting up the development environment, running official examples, model conversion, and custom-model deployment, helping you quickly get started with RK3568 NPU development.