Multi-Model Parallel Inference

This chapter covers the parallel scheduling strategies and the multi-context API for running multiple AI models simultaneously on the RK182X NPU.

Overall block diagram

Camera frame (1080p@30fps)
       ↓
┌─────────────────────────────────┐
│  Model A: person detection      │  → full-frame inference
│  (YOLOv8n)                      │
│  Output: list of bboxes         │
└────────┬────────────────────────┘
         │ crop the person region
         ↓
┌─────────────────────────────────┐
│  Model B: attribute recognition │  → 1 inference per person
│  (ResNet50)                     │
│  Output: gender / age / clothing│
└────────┬────────────────────────┘
         │ attributes + tracks
         ↓
┌─────────────────────────────────┐
│  Model C: behavior analysis     │  → temporal inference
│  (SlowFast)                     │
│  Output: stand / walk / run /   │
│          fall                   │
└─────────────────────────────────┘

Real products often need to run several models at once: face detection + attribute recognition + behavior analysis + OCR.

1. Typical Application Scenario

Real products commonly require a three-stage pipeline: Model A (YOLOv8n full-frame person detection) → Model B (ResNet50 attribute recognition on the crops) → Model C (SlowFast temporal behavior analysis), with each model's output cascaded to the next.

2. NPU Multi-Instance Support

The RK182X NPU supports multi-context parallelism: each rknn3_init creates an independent context, and the hardware scheduler automatically time-shares the NPU cores.

#include <rknn3_api.h>

rknn3_context ctx_det  = 0;
rknn3_context ctx_attr = 0;
rknn3_context ctx_act  = 0;

rknn3_init_extend ext = {0};

rknn3_init(&ctx_det,  &ext);
rknn3_init(&ctx_attr, &ext);
rknn3_init(&ctx_act,  &ext);

/* the three contexts can call rknn3_run() alternately */

rknn3_destroy(ctx_det);
rknn3_destroy(ctx_attr);
rknn3_destroy(ctx_act);

3. Compilation (Directly on the RK3588)

aarch64-linux-gnu-gcc multi_model.c -o multi_model \
    -lrknn3_api \
    -I/userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include \
    -lpthread

Compiles successfully on the RK3588: the multi-context API uses pthread, so -lpthread is required.

3.1 Key API Points

API	Description
`rknn3_init(ctx, ext)`	2-argument version
`rknn3_destroy(ctx)`	Releases the context
`rknn3_run(ctx, inputs, n_in, outputs, n_out)`	5-argument inference
`rknn3_init_extend`	Contains only `device_id` + `reserved`

4. RKNN3 Multi-Context API Verification

grep -n "rknn3_init\|rknn3_destroy\|rknn3_run\|rknn3_init_extend" /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h | head -10

Output:

569: * @struct _rknn3_init_extend
571:typedef struct _rknn3_init_extend
576:} rknn3_init_extend;
1046: * @note The context must be released using rknn3_destroy when no longer needed
1048:int rknn3_init(rknn3_context* context, rknn3_init_extend* init_extend);
1104:int rknn3_destroy(rknn3_context context);
1124: * @param context The RKNN3 context handle obtained from rknn3_init
1135:int rknn3_run(rknn3_context context, const rknn3_tensor inputs[], uint32_t n_inputs, rknn3_tensor outputs[], uint32_t n_outputs);
1140: * @param context The RKNN3 context handle obtained from rknn3_init
1151:int rknn3_run_async(rknn3_context context, const rknn3_tensor inputs[], uint32_t n_inputs, rknn3_tensor outputs[], uint32_t n_outputs);

The rknn3_init_extend struct definition:

sed -n '571,576p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

typedef struct _rknn3_init_extend
{
  char* device_id;       /** < input parameter, indicate which device selected. if only one
                            device connected, can set nullptr. */
  uint8_t reserved[128]; /** < reserved */
} rknn3_init_extend;

API signature verification:
rknn3_init(): 2 arguments (rknn3_context*, rknn3_init_extend*)
rknn3_destroy(): 1 argument (rknn3_context)
rknn3_run(): 5 arguments (rknn3_context, inputs[], n_inputs, outputs[], n_outputs)
rknn3_run_async(): 5 arguments (asynchronous version)

Full declaration of rknn3_init (including the return-value comment):

sed -n '1042,1050p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

 * @param[out] context Pointer to the RKNN3 context handle that will be initialized
 * @param[in] init_extend Pointer to the device-specific initialization information
 * @return int Return status code:
 *         - 0: Success
 *         - <0: Error code
 *
 * @note The context must be released using rknn3_destroy when no longer needed
 */
int rknn3_init(rknn3_context* context, rknn3_init_extend* init_extend);

Full declaration of rknn3_destroy:

sed -n '1098,1104p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:

 * @param context The RKNN3 context handle to be destroyed.
 *
 * @return int Return 0 if the operation is successful, otherwise return error code.
 */
int rknn3_destroy(rknn3_context context);

Full declaration of rknn3_run:

sed -n '1125,1135p' /userdata/RK1820_RK1828_AI_SDK/rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h

Output:


 * @brief Execute the RKNN3 model inference.
 *
 * @param context The RKNN3 context handle obtained from rknn3_init
 * @param inputs Array of input tensors containing the input data
 * @param n_inputs Number of input tensors
 * @param outputs Array of output tensors to store the inference results
 * @param n_outputs Number of output tensors
 * @return int Return 0 if successful, otherwise return error code

5. Recommended Three-Stage Pipeline Architecture

Implemented with Python threading + Queue: detector_thread (full-frame detection) → attribute_thread (crop recognition) → action_thread (temporal analysis), with frames and detection results passed between threads via Queues.

import threading
import queue
from rknn3lite.api import RKNN3Lite

# three queues: raw frames / detection results / attribute results
frame_q = queue.Queue(maxsize=10)
det_q   = queue.Queue(maxsize=10)
attr_q  = queue.Queue(maxsize=10)

det_model = RKNN3Lite();  det_model.load_rknn('yolov8n.rknn', 'yolov8n.weight');  det_model.init_runtime()
attr_model = RKNN3Lite(); attr_model.load_rknn('resnet50.rknn', 'resnet50.weight'); attr_model.init_runtime()
act_model = RKNN3Lite();  act_model.load_rknn('slowfast.rknn', 'slowfast.weight');  act_model.init_runtime()

def detector_thread():
    while True:
        frame = frame_q.get()
        bbox = det_model.inference(inputs=[frame])[0]
        det_q.put((frame, bbox))

def attribute_thread():
    while True:
        frame, bbox = det_q.get()
        # crop the person region
        attr = attr_model.inference(inputs=[crop])[0]
        attr_q.put((frame, bbox, attr))

def action_thread():
    while True:
        frame, bbox, attr = attr_q.get()
        act = act_model.inference(inputs=[frame])[0]
        # output (frame, bbox, attr, act)

6. FAQ

Symptom	Cause	Fix
Compile error: undefined reference to `rknn3_*`	Missing `-lrknn3_api`	Add `-lrknn3_api`
Compile error about pthread	Multithreading	Add `-lpthread`
`rknn3_init` returns a negative value	Device busy / permissions	`lsof /dev/pcie-rkep-*`
Multiple contexts preempt each other on the NPU	Scheduling is time-shared by default	Accept it, or use `-c` to pin a single model
Wrong `device_id` passed	Multi-device setup	Leave nullptr (single device) or check `lspci`

7. Next Steps

RKNN Model Conversion — model conversion and quantization
INT8 Quantized Inference — quantization and performance basics
RGA in Detail — preprocessing acceleration