INT8 Quantized Inference Deployment

This chapter covers deploying INT8 quantized models on the RK182X with optimized inference: RKNN3 C API + Python API + performance tuning + accuracy verification.

Overall block diagram

Application process
    ↓
C API / Python API (librknn3_api.so / rknn3lite)
    ↓
rknn3_transfer_proxy (PCIe proxy)
    ↓
RK1828 NPU (8 cores)

The RK182X uses the RKNN3 Toolkit. RKNN-Toolkit2 is for host SoCs such as the RK3588. All code examples are based on RKNN3.

To generate an INT8 model, first refer to Model Conversion.

1. C API Inference

A CNN inference skeleton that is verified to compile and run (compiled with aarch64-linux-gnu-gcc, tested on the board):

#include "rknn3_api.h"   // request the header file from Rockchip

int main(int argc, char* argv[]) {
    rknn3_context ctx = 0;
    rknn3_init_extend ext = {0};
    int ret = rknn3_init(&ctx, &ext);              // must pass context pointer + init_extend

    ret = rknn3_load_model_from_path(ctx, argv[1], argv[2]);
    // Only 3 arguments (context, model_path, weight_path); there is no core_mask argument

    rknn3_config cfg = {0};
    cfg.run_core_mask = 0x01;                      // core_mask is passed here
                                                  // CNN (e.g. MobilenetV2) uses 0x01; LLM/VLM use 0xff
    ret = rknn3_model_init(ctx, &cfg);

    ret = rknn3_profile_mem(ctx);                  // profile_mem exists only in the C API

    rknn3_destroy(ctx);                            // destroy the context
    return 0;
}

Differences between the C API and the LLM API (verified against header-file signatures):

  • CNN inference goes through rknn3_run(context, inputs, n_inputs, outputs, n_outputs); rknn3_session_run(session, ...) is LLM-only — do not mix the two
  • rknn3_load_model_from_path(context, model_path, weight_path) takes only 3 arguments; core_mask is not passed here — it is passed via rknn3_config.run_core_mask inside rknn3_model_init()
  • rknn3_destroy(context) destroys the context; rknn3_session_destroy(session) is the destruction interface for LLM sessions
  • For complete signatures, refer to rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h in the SDK

Measured output of rknn3_profile_mem (8 independent NPU nodes, ~620–640 MB each):

=========================== Memory Usage Information ===========================
Device Memory:
  System  : 19.12 MB total,       6.12 MB free,      13.00 MB used ( 68.0%)
  Node 0  : 639.50 MB total,     418.29 MB free,     221.21 MB used ( 34.6%)
  Node 1  : 639.50 MB total,     523.37 MB free,     116.13 MB used ( 18.2%)
  Node 2  : 619.90 MB total,     503.98 MB free,     115.91 MB used ( 18.7%)
  Node 3  : 639.50 MB total,     523.36 MB free,     116.14 MB used ( 18.2%)
  Node 4  : 619.90 MB total,     504.06 MB free,     115.84 MB used ( 18.7%)
  Node 5  : 619.90 MB total,     504.14 MB free,     115.76 MB used ( 18.7%)
  Node 6  : 639.50 MB total,     523.97 MB free,     115.53 MB used ( 18.1%)
  Node 7  : 639.50 MB total,     524.95 MB free,     114.55 MB used ( 17.9%)

2. Compilation (compile directly on the RK3588)

# The librknn3_api library is at /usr/lib/librknn3_api.so
# But the rknn3_api.h header is not on the standard include path; the SDK path must be given
aarch64-linux-gnu-gcc int8_inference.c -o int8_inference \
    -lrknn3_api \
    -I/userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include

Compilation on the RK3588 succeeds: librknn3_api is at /usr/lib/librknn3_api.so, and the header path must be specified explicitly.

Verifying the build environment:

which aarch64-linux-gnu-gcc
ls /usr/lib/librknn3_api.so
ls /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

/usr/bin/aarch64-linux-gnu-gcc
/usr/lib/librknn3_api.so
/userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

3. Python API Inference

python3 -c "from rknn3lite.api import RKNN3Lite; print('RKNN3Lite API available')"

Output:

RKNN3Lite API available
from rknn3lite.api import RKNN3Lite

rknn_lite = RKNN3Lite()
rknn_lite.load_rknn('model.rknn', 'model.weight')
rknn_lite.init_runtime(target='rk1820', core_mask=0x01)   # CNN: 1 core; LLM/VLM use 0xff

# Prepare inputs (NHWC, uint8)
outputs = rknn_lite.inference(inputs=[input_data])

rknn3lite has no profile_mem / profile_ops methods (verified with hasattr, both False) — memory profiling is only available via the C API's rknn3_profile_mem. Verified available methods (dir(), 19 total): load_rknn / init_runtime / inference / session_run / llm / get_devices_id / get_inputs_tensor_attr / get_outputs_tensor_attr / get_sdk_version / set_chat_template / release, etc.

4. Performance Tuning

ParameterRecommended value
Input formatNHWC (RKNN3 default)
Input typeuint8 (convert.py uses dtype='uint8')
Batch size1 (same as during training)
Core frequencyMaximum frequency (via /rockchip-test/npu2/npu_freq_scaling.sh or sysfs)
Memory allocation620–640 MB per independent NPU node (inspect with C API rknn3_profile_mem)

5. Performance Test Tools

NPU tool directory:

ls /rockchip-test/npu2/

Output:

model
npu_freq_scaling.sh
npu_stress_test.sh
npu_test.sh

rknn3_model_test help:

/usr/bin/rknn3_model_test --help

Output:

Usage: /usr/bin/rknn3_model_test <model_path> <weight_path> <input_npy_paths>
       <golden_output_npy_paths> <core_mask> [loop_count]
  - If input_npy_paths is not provided, random input will be generated
  - If golden_output_npy_paths is not provided, cosine similarity will be skipped
  - If core_mask is not provided, it will be auto-generated based on core_number
  - If you want to use both random input and set core_mask or loop_count, you can set both input_npy_paths and golden_output_npy_paths to empty strings.

6. INT8 Inference Accuracy Optimization

6.1 Troubleshooting Flow

  1. Check the calibration dataset size (20 representative samples recommended)
  2. Choose the dataset by model type:
    • LLM → datasets/CMMLU/dataset.json
    • CNN → datasets/imagenet/.../dataset_20.txt
  3. Verify accuracy with /usr/bin/rknn3_model_test (compares cosine similarity against the golden output; rknn3lite has no profile_ops method)

6.2 Key Configuration

rknn.config(
    target_platform='rk1820',
    mean_values=[[255*0.485, 255*0.456, 255*0.406]],   # ImageNet normalization
    std_values=[[255*0.229, 255*0.224, 255*0.225]],
    input_attrs={'input': {'dtype': 'uint8', 'layout': 'NHWC'}},
    quantized_dtype='w8a8',
)

7. Deployed INT8 Models

ls -lh /userdata/models/Qwen3-1.7B/

Output:

总计 1.7G
-rw-r--r-- 1 linaro linaro 594M  8月20日 11:33 Qwen3-1.7B.embed.bin
-rw-r--r-- 1 linaro linaro  24M  8月20日 11:34 Qwen3-1.7B.rknn
-rw-r--r-- 1 linaro linaro 5.9M  8月20日 11:33 Qwen3-1.7B.tokenizer.gguf
-rw-r--r-- 1 linaro linaro 1.1G  8月20日 11:33 Qwen3-1.7B.weight

8. Measured RK1828 NPU Performance

ModelInference timeFPS / TPS
MobileNet V2 FP16 (rknn3lite)5.40 ms185.1 FPS (average of 10 runs)
MobileNet V2 FP16 (rknn3_cnn_demo)4.54 ms220 FPS (see CNN Inference)
Qwen3-1.7B (TTFT)70 ms—
Qwen3-1.7B (TPS)—133 tok/s

9. FAQ

SymptomCauseResolution
rknn3_api.h: No such fileSDK include path not specifiedAdd -I/userdata/RK1820_RK1828_AI_SDK/.../include
cannot find -lrknn3_apiLibrary path not in /usr/libVerify /usr/lib/librknn3_api.so exists
profile_mem AttributeErrorCalling memory profiling via rknn3liteUse the C API instead
core_mask X is not match ...-c mismatched with the model's core countCNN uses 0x01, LLM uses 0xff
Severe accuracy dropInsufficient calibration set / wrong typeUse 20+ samples + w8a8 dtype

10. Next Steps

11. References

  • RK1820_RK1828_AI_Release-Note_CN.md (in the SDK) — RKNN3 V1.0.0 changelog
  • Rockchip_RK1820_RK1828_AI_SDK_RELEASE_CN.pdf (in the SDK) — Release Note
  • Rockchip_RK1820_RK1828_AI_SDK_Quick_Start_CN.pdf (in the SDK) — Quick Start