Model Conversion

This chapter clarifies the PC-side vs on-board toolchains and walks through both LLM and CNN conversion.

Overall block diagram

PC (x86_64 Ubuntu)                       Board (aarch64)
──────────────────                       ─────────────
PyTorch / ONNX / HuggingFace              rknn3-toolkit-lite
   │                                       │
   ▼                                       │
rknn-toolkit3 (pip)                       │
rknn.config + rknn.build + rknn.export_rknn
   │                                       │
   ▼                                       │
.rknn + .weight + .tokenizer.gguf + .embed.bin
   │                                       │
   ▼                                       ▼
scp ─────────────────────────────────────► rknn3-toolkit-lite.load_rknn()
                                          rkllm3-server / rknn3_cnn_demo

1. Toolchain Distinction

Important distinction:
PC side: rknn-toolkit (pip package rknn-toolkit3, Python module rknn)
Board side: rknn3-toolkit-lite (Python module rknn3lite)
The board side can only run inference, not conversion! Conversion must be done on the PC.

Location	Package name	Python module	Usage
PC (x86_64)	`rknn-toolkit3`	`rknn`	Model conversion, quantization, simulation
Board (aarch64)	`rknn3-toolkit-lite`	`rknn3lite`	Inference (cannot convert)

Python version requirement: only 3.10 and 3.12 are supported; 3.11 is not. The current system Python is 3.11.2, so Python 3.10 / 3.12 must be installed additionally to run the toolkit.

2. Using Official Pre-converted Models (Recommended)

The RKNN3 Model Zoo provides officially pre-converted models.

2.1 Check the SDK Model Zoo

ls /userdata/RK1820_RK1828_AI_SDK/rknn/

Output:

rknn3-model-zoo
rknn3-runtime
rknn3-toolkit
rknn-gstreamer-plugins

ls /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-model-zoo/examples/

Output:

FastVLM         glm_edge        GME-Qwen2-VL    HY_MT_1_5
InternVLM       Janus_Pro       MiniCPM_V_4     mobilenet_v1
mobilenet_v2    Qwen2_5         Qwen2_5_Omni    Qwen2_5_VL
Qwen3           Qwen3_Embedding Qwen3_VL        resnet
SmolVLM         yolov5          yolov6          yolov8

2.2 Clone the GitHub Repository

git clone https://github.com/airockchip/rknn3-model-zoo.git

2.3 Transfer to the Device

scp -r model/ root@<device IP>:/userdata/models/Qwen3-1.7B/

2.4 View Deployed Models

ls -lh /userdata/models/Qwen3-1.7B/

Output (measured):

总计 1.7G
-rw-r--r-- 1 linaro linaro 594M  8月20日 11:33 Qwen3-1.7B.embed.bin
-rw-r--r-- 1 linaro linaro  24M  8月20日 11:34 Qwen3-1.7B.rknn
-rw-r--r-- 1 linaro linaro 5.9M  8月20日 11:33 Qwen3-1.7B.tokenizer.gguf
-rw-r--r-- 1 linaro linaro 1.1G  8月20日 11:33 Qwen3-1.7B.weight

Total size:

du -sh /userdata/models/Qwen3-1.7B/

Output:

1.7G	/userdata/models/Qwen3-1.7B/

3. Converting LLM Models Yourself (Qwen3-1.7B)

3.1 Prerequisites

PC (GRQ quantization requires an NVIDIA GPU with ≥ 16 GB VRAM)
Python 3.10 or 3.12 (not 3.11)
pip install rknn-toolkit3 (full PC-side package)

Verify the PC-side tool:

# On the PC
python -c "from rknn.api import RKNN; print('RKNN Toolkit installed')"

Output:

RKNN Toolkit installed

3.2 Method A: Model Zoo Conversion Scripts (Recommended)

git clone https://github.com/airockchip/rknn3-model-zoo.git
cd rknn3-model-zoo
pip install -r requirements.txt
export PYTHONPATH=./
cd examples/Qwen3/python/

# Step 1: export ONNX (GRQ quantization)
python export_llm.py --quant --model_path Qwen/Qwen3-1.7B

# Step 2: convert to RKNN
python export_rknn.py

Even with --quant enabled, GRQ quantization still requires the dataset ../../../datasets/CMMLU/dataset.json.

Actual RKNN configuration (from export_rknn.py):

rknn.config(target_platform='rk1820',     # ← not 'rk1828'
            quantized_dtype='w4a16',
            quantized_algorithm='grq',
            quantized_method='group32')
rknn.load_llm(model=onnx_path, config=config_path)
rknn.build(do_quantization=True, dataset=dataset_path)
rknn.export_rknn(rknn_path)

3.3 Method B: Python API (PC side)

from rknn.api import RKNN      # ← the real PC-side API

rknn = RKNN(verbose=True)

# Configure (mind the parameter names)
rknn.config(target_platform='rk1820',     # not 'rk1828'
            quantized_dtype='w4a16',
            quantized_algorithm='grq',
            quantized_method='group32')

# Load the LLM (ONNX + config)
rknn.load_llm(model='Qwen3-1.7B.onnx', config='Qwen3-1.7B.config.pkl')

# Build
rknn.build(do_quantization=True, dataset='./dataset.txt')

# Export
rknn.export_rknn('./Qwen3-1.7B.rknn')

3.4 Conversion Output

Qwen3-1.7B/
├── Qwen3-1.7B.rknn            # Model structure
├── Qwen3-1.7B.weight          # Model weights (~1.1 GB)
├── Qwen3-1.7B.tokenizer.gguf  # Tokenizer
└── Qwen3-1.7B.embed.bin       # Embedding (~594 MB)

4. Converting CNN Models Yourself (MobileNet V2)

import torch
import torchvision.models as models

# Step 1: export ONNX
model = models.mobilenet_v2(pretrained=True)
model.eval()
dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, dummy_input, "mobilenet_v2.onnx",
                  input_names=["input"], output_names=["output"])

# Step 2: convert to RKNN
from rknn.api import RKNN
rknn = RKNN(verbose=True)
rknn.config(target_platform='rk1820',       # not 'rk1828'
            mean_values=[[0, 0, 0]],
            std_values=[[255, 255, 255]])
rknn.load_onnx(model='./mobilenet_v2.onnx')
rknn.build(do_quantization=True, dataset='./dataset.txt')
rknn.export_rknn('./mobilenet_v2.rknn')

4.1 Conversion Output

mobilenet_v2/
├── mobilenet_v2.rknn       # Model structure (~60KB)
└── mobilenet_v2.weight     # Model weights (~6.8MB)

5. Key Parameters

Parameter	Description	Recommendation
`target_platform`	Target platform	`'rk1820'` (not `'rk1828'`)
`quantized_dtype`	Quantization type	`'w4a16'` (`w4a8` / `w8a8` also available)
`quantized_algorithm`	Quantization algorithm	`'grq'` (GRQ) or `'mmse'` (traditional)
`quantized_method`	Quantization method	`'group32'` / `'channel'`
`do_quantization`	Whether to quantize	Enabled to reduce model size
`max_context_len`	LLM context (hard-coded into the model)	≥ 8192 recommended for 1.7B

max_context_len is hard-coded into the model at conversion time; at runtime rkllm3-server -c cannot exceed it (-c is actually the core mask, not the context; see ch03 LLM Inference).

6. Deployment Verification

6.1 File Integrity

# LLM model verification
file /userdata/models/Qwen3-1.7B/Qwen3-1.7B.rknn
md5sum /userdata/models/Qwen3-1.7B/Qwen3-1.7B.rknn

Output:

Qwen3-1.7B.rknn: data
a1b2c3d4e5f6...  Qwen3-1.7B.rknn

6.2 Loadability (on-board)

python3 -c "
from rknn3lite.api import RKNN3Lite
rknn = RKNN3Lite()
rknn.load_rknn(
    '/userdata/models/Qwen3-1.7B/Qwen3-1.7B.rknn',
    '/userdata/models/Qwen3-1.7B/Qwen3-1.7B.weight'
)
print('Model loaded successfully')
"

Output:

Model loaded successfully

weight_path is a required parameter; omitting it reports TypeError: missing 1 required positional argument: 'weight_path'.

7. FAQ

Symptom	Cause	Fix
`ModuleNotFoundError: No module named 'rknn'`	Installed on the board (aarch64 unsupported)	Install on the PC (x86_64) instead
Python 3.11 cannot install the toolkit	Only 3.10 / 3.12 supported	Install Python 3.10 or 3.12
`rknn.config` rejects `target_platform`	Written as `'rk1828'`	Change to `'rk1820'`
GRQ quantization complains about dataset	Dataset not provided	Download `rknn3-model-zoo/datasets/CMMLU/dataset.json`
On-board `load_rknn` complains about weight_path	Second argument not passed	`rknn.load_rknn(rknn_path, weight_path)`
`max_context_len` cannot be changed	Hard-coded in the model	Re-convert on the PC

8. Next Steps

LLM Inference — run the converted Qwen3-1.7B
CNN Inference — run MobileNet V2
Model conversion → performance data — performance benchmarks