INT8 Quantized Inference

This chapter walks through the complete workflow for configuring INT8 quantized inference on the RK182X NPU: calibration dataset preparation, quantization accuracy verification, mixed-precision strategy, and performance comparison.

Overall block diagram

Training-framework model (FP32 / FP16)
       ↓
Prepare the calibration set (50-200 images)
       ↓
RKNN Toolkit INT8 configuration + build
       ↓
On-board INT8 inference (C API / Python API)
       ↓
Accuracy verification (accuracy_analysis)

1. How INT8 Quantization Accelerates Inference

The NPU's INT8 multiply-accumulate units (MACs) have far lower area and power than FP16. The RK182X NPU architecture accelerates INT8 at the hardware level.

PrecisionRelative speedMemory usageApplicable scenarios
FP321x100%Training, high-accuracy inference
FP162x50%General inference
INT83-4x25%First choice for production deployment
INT46x12.5%Extreme compression scenarios

Quantization formula:

Q = round(F / S)        F ≈ Q × S

where S (scale) is determined per layer from the min/max of activation values collected over the calibration dataset.

2. Complete INT8 Quantization Workflow

Four steps: Step 1 prepare the calibration set (50-200 images) → Step 2 configure INT8 in RKNN Toolkit + build → Step 3 on-board INT8 inference → Step 4 accuracy verification.

2.1 Step 1: Prepare the Calibration Dataset

import os, random

image_dir = '/path/to/dataset/train/images'
all_images = [os.path.join(image_dir, f) for f in os.listdir(image_dir)
              if f.endswith(('.jpg', '.png'))]

random.seed(42)
calib_images = random.sample(all_images, 100)

with open('calib_list.txt', 'w') as f:
    for img in calib_images:
        f.write(img + '\n')
print(f'Calibration set: {len(calib_images)} images')

2.2 Step 2: Run the Quantized Conversion

from rknn.api import RKNN

rknn = RKNN(verbose=True)

# Actual legal values (rknn.py:177-178):
#   target_platform = rv1103 / rv1103b / rv1106 / rv1106b / rk2118 /
#                     rk3562 / rk3566 / rk3568 / rk3576 / rk3588 / rk1820
# Actual legal values (rknn.py:173):
#   quantized_dtype = w8a8 / w4a16 (default w16a16)
rknn.config(
    target_platform='rk1820',
    mean_values=[[123.675, 116.28, 103.53]],
    std_values=[[58.395, 57.12, 57.375]],
    quantized_dtype='w8a8',
)

rknn.load_onnx('model.onnx')

# Actual signature (rknn.py:291): build(do_quantization, dataset, rknn_batch_size, auto_hybrid)
rknn.build(
    do_quantization=True,
    dataset='calib_list.txt',
)

rknn.export_rknn('model_int8.rknn')

# Actual signature (rknn.py:417): accuracy_analysis(inputs, output_dir, core_mask, target, device_id)
rknn.accuracy_analysis(
    inputs='./test_set/',
    output_dir='./quant_report/',
)
rknn.release()

2.3 Step 3: On-Board INT8 Inference

#include <rknn3_api.h>   /* the actual header is not rknn_api.h */

rknn3_context ctx = 0;
rknn3_init_extend init_extend = {0};
int ret = rknn3_init(&ctx, &init_extend);
if (ret != RKNN3_SUCCESS) return -1;

/* rknn3_api.h:328 rknn3_tensor_attr */
rknn3_tensor_attr input_attr = {0};
input_attr.index = 0;
rknn3_query(ctx, RKNN3_QUERY_INPUT_ATTR, &input_attr, sizeof(input_attr));

rknn3_tensor_attr output_attr = {0};
output_attr.index = 0;
rknn3_query(ctx, RKNN3_QUERY_OUTPUT_ATTR, &output_attr, sizeof(output_attr));

/* For INT8 quantized models the input/output tensor dtype is given by
 * rknn3_tensor_attr.dtype; see the rknn3_tensor_type enum at rknn3_api.h:114
 * (UINT8 / INT8 / FLOAT16 / ...) */

3. API Verification

Header file paths:

find /userdata/RK1820_RK1828_AI_SDK -name "rknn3_api.h" -type f

Output:

/userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h
/userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-model-zoo/3rdparty/rknpu3/include/rknn3_api.h

Header path confirmed: rknn3_api.h (not rknn_api.h); the main path is rknn3-runtime/rknn3-api/include/.

The rknn3_tensor_type enum definition:

sed -n '114,128p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

typedef enum _rknn3_tensor_type
{
  RKNN3_TENSOR_FLOAT32 = 0, /** < data type is float32. */
  RKNN3_TENSOR_FLOAT16,     /** < data type is float16. */
  RKNN3_TENSOR_INT8,        /** < data type is int8. */
  RKNN3_TENSOR_UINT8,       /** < data type is uint8. */
  RKNN3_TENSOR_INT16,       /** < data type is int16. */
  RKNN3_TENSOR_UINT16,      /** < data type is uint16. */
  RKNN3_TENSOR_INT32,       /** < data type is int32. */
  RKNN3_TENSOR_UINT32,      /** < data type is uint32. */
  RKNN3_TENSOR_INT64,       /** < data type is int64. */
  RKNN3_TENSOR_UINT64,      /** < data type is uint64. */
  RKNN3_TENSOR_BOOL,        /** < data type is boolean. */
  RKNN3_TENSOR_INT4,

  RKNN3_TENSOR_TYPE_MAX
} rknn3_tensor_type;

The rknn3_tensor_attr struct definition:

sed -n '328,340p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

typedef struct _rknn3_tensor_attr
{
  uint32_t index;                /** < input parameter, the index of input/output tensor,
                                    need set before call rknn3_query. */
  char name[RKNN3_MAX_NAME_LEN]; /** < the name of tensor. */

  uint32_t n_dims;                        /** < the number of dimensions. */
  uint32_t shape[RKNN3_MAX_DIMS];         /** < the valid dimensions array. */

RKNN3_QUERY constants:

grep -n "RKNN3_QUERY_INPUT_ATTR\|RKNN3_QUERY_OUTPUT_ATTR\|RKNN3_SUCCESS" /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h | head -5

Output:

28:#define RKNN3_SUCCESS 0                       /** < execute succeed. */
81:  RKNN3_QUERY_INPUT_ATTR  = 1, /** < query the attribute of input tensor. */
82:  RKNN3_QUERY_OUTPUT_ATTR = 2, /** < query the attribute of output tensor. */
326: * @brief The information for RKNN3_QUERY_INPUT_ATTR / RKNN3_QUERY_OUTPUT_ATTR.
1259: * @return Return RKNN3_SUCCESS on success, return error code on failure

4. Mixed-Precision Strategy

When full INT8 quantization falls short on accuracy, use mixed precision: keep sensitive layers in FP16 and quantize the rest to INT8.

Switches provided by the RKNN3 toolchain (rknn.api.rknn:127 RKNN.config):

grep -n "auto_hybrid" /tmp/rknn/api/rknn.py

Output:

146:               auto_hybrid_cos_thresh=0.98,
147:               auto_hybrid_euc_thresh=None,
192:        :param auto_hybrid_cos_thresh: The thresholds of cosine distance in auto hybrid when model is quantizate. default is 0.98
193:        :param auto_hybrid_euc_thresh: The thresholds of euclidean distance in auto hybrid when model is quantizate. default is None
291:    def build(self, do_quantization=True, dataset=None, rknn_batch_size=None, auto_hybrid=False):
297:        :param auto_hybrid: Whether to enable automatic hybrid quantization to adjust accuracy or overflow. default is False.
rknn.config(
    target_platform='rk1820',
    quantized_dtype='w8a8',
    # Quantization algorithm (rknn.py:174): normal / mmse / kl_divergence / gdq
    quantized_algorithm='mmse',
    # Quantization method (rknn.py:175): layer / channel / group{32..256}
    quantized_method='channel',
    # Mixed-precision thresholds (rknn.py:146-147)
    auto_hybrid_cos_thresh=0.98,
    auto_hybrid_euc_thresh=None,
)

# auto_hybrid in build() is the real automatic mixed-precision switch (rknn.py:291)
rknn.build(
    do_quantization=True,
    dataset='calib_list.txt',
    auto_hybrid=True,   # the toolchain detects mixed precision automatically
)

4.1 Identifying Sensitive Layers

rknn.accuracy_analysis(
    inputs='./test_set/',
    output_dir='./layer_wise_report/',
)

5. FAQ

SymptomCauseFix
Accuracy drops with full INT8 quantizationSensitive layers got quantizedSwitch to quantized_method='channel' + auto_hybrid=True
Top-1 drops >1%Insufficient calibration coverageExpand to 100+ samples
quantized_dtype='w4a8' reported as unsupportedRKNN3 doesn't support that combinationChange to 'w8a8' or 'w4a16'
accuracy_analysis can't find layersWrong test-set pathUse absolute paths
auto_hybrid=True reports OOMModel too largeReduce with rknn_batch_size=1

6. Next Steps