RK182X Series LLM Inference (RK1828 Model)

Overall block diagram

Host (RK3588)                            RK1828 (PCIe AI coprocessor)
──────────────────                       ─────────────────────
rknn3_llm_demo / rkllm3-server / Web     8-core NPU
   │                                          │
   ▼                                          │
PCIe Gen3 x4 ──────────────────────────────►  NPU inference (8 cores)
                                              Qwen3-1.7B / 4B / ...

1. C Inference Test (rknn3_llm_demo)

A pure CLI benchmark, not a service.

rknn3_llm_demo \
  -m  <model dir>/Qwen3-1.7B/Qwen3-1.7B.rknn \
  -w  <model dir>/Qwen3-1.7B/Qwen3-1.7B.weight \
  -tk <model dir>/Qwen3-1.7B/Qwen3-1.7B.tokenizer.gguf \
  -em <model dir>/Qwen3-1.7B/Qwen3-1.7B.embed.bin \
  -c  0xff \
  -ctx 16384 \
  -no 4096

Parameter quick reference:

Parameter	Description
`-m`	RKNN model path
`-w`	Model weight path
`-tk`	Tokenizer path
`-em`	Embedding path
`-c`	NPU core mask, must match the core count declared at model compile time (Qwen3-1.7B is 8-core → `0xff`)
`-ctx`	Maximum context length (do not exceed the model's `max_context_len`)
`-no`	Maximum number of generated tokens
`-d`	Device ID (for multiple NPUs; use 0 for a single device)

Passing a wrong value to -c reports core_mask X is not match with npu core number 8!: using 1 reports "core_mask 1 is not match with npu core number 8!"; a context-like value such as 8192 also reports "core_mask 8192 is not match with npu core number 8". CNN models (e.g. MobileNet V2) declare 1 core at compile time → -c 1; see ch04 CNN Inference.

Help:

rknn3_llm_demo --help

Output:

Usage:
rknn3_llm_demo [options]

Options:
  -m, --model_path
  -w, --model_weight
  -tk, --tokenizer
  -em, --embedding
  -c, --core_mask
  -ctx,--max_context_len
  -ni,--n_input_tokens
  -no,--max_new_tokens
  -cl, --cnt_loop  default:1
  -tl, --time_loop(hours)
  -d, --device_id       specify device ID
  -mm, --enable_memory_monitor  enable memory monitoring (default: disabled)

Expected output:

===== RKNN3_LLM_TEST Results Summary =====

Model: Qwen3-1.7B
DeviceID: 0004:41:00.0
TTFT: 63.208 ms
TPOT: 8.636 ms
Prefill: 253.132 tokens/s
TPS: 115.798 tokens/s
TotalTime: 607.262 ms
TotalRuns: 1
AvgMemoryUsage: unavailable
AvgNpuUtilization: unavailable

Qwen3-1.7B chat demo

2. HTTP Inference Service (rkllm3-server)

rkllm3-server provides an OpenAI-compatible API; any application using the OpenAI SDK can connect directly.

2.1 Starting the Service

rkllm3-server \
  -m <model dir>/Qwen3-1.7B/Qwen3-1.7B.rknn \
  --weight <model dir>/Qwen3-1.7B/Qwen3-1.7B.weight \
  --vocab <model dir>/Qwen3-1.7B/Qwen3-1.7B.tokenizer.gguf \
  --embed <model dir>/Qwen3-1.7B/Qwen3-1.7B.embed.bin \
  --host 0.0.0.0 --port 8080 \
  -c 0xff

Parameter quick reference:

Parameter	Description
`-m`	RKNN model path
`--weight`	Model weight path
`--vocab`	Tokenizer path
`--embed`	Embedding path
`--host`	Listen address (default 127.0.0.1)
`--port`	Listen port (default 8080)
`-c`	NPU core mask (8-core → `0xff`)

Help:

rkllm3-server --help

Output:

version: 1.0.0 (b4bd144@2026-01-22T21:27:27)

----- common params -----

-c,    --ctx-size N                     size of the prompt context (default: 4096, 0 = loaded from model)
-n,    --predict, --n-predict N         number of tokens to predict (default: -1, -1 = context size)
-a,    --alias STRING                   set alias for model name (to be used by REST API)
-m,    --model FNAME                    rknn llm model path
--weight FNAME                          rknn llm model weight path
--vocab FNAME                           vocab path
--embed FNAME                           embed path

Measured behavior: -c is treated as the NPU core mask (Qwen3-1.7B requires -c 0xff), which does not match the -c, --ctx-size description in --help. The context length is determined by max_context_len set at model compile time; passing a context value (e.g. 8192) makes startup fail with core_mask 8192 is not match with npu core number 8.

2.2 Model Files

ls /userdata/models/Qwen3-1.7B/ | grep -E "\.(rknn|weight|gguf|bin)$"

Output:

Qwen3-1.7B.embed.bin
Qwen3-1.7B.rknn
Qwen3-1.7B.tokenizer.gguf
Qwen3-1.7B.weight

2.3 API Endpoints

# List models
curl http://localhost:8080/v1/models

# Chat inference
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen","messages":[{"role":"user","content":"你好"}]}'

2.4 Python SDK Integration

from openai import OpenAI

client = OpenAI(base_url="http://<device IP>:8080/v1", api_key="none")

response = client.chat.completions.create(
    model="qwen",
    messages=[
        {"role": "system", "content": "你是一个助手"},
        {"role": "user", "content": "你好"},
    ],
    max_tokens=100,
)
print(response.choices[0].message.content)

3. Web Chat Interface

A lightweight Flask + vanilla-frontend web chat interface that talks directly to the rkllm3-server API, supporting natural-language conversation and simulated device control.

cd <project dir>
bash start.sh
# Visit http://<device IP>:7080

Web initial interface

Web demo: device control

4. Performance Reference

4.1 rknn3_llm_demo Performance

Metric	Value
Time to first token (TTFT)	~63 ms
Time per output token (TPOT)	~8.6 ms
Prefill speed	~253 tok/s (measured)
Generation speed (TPS)	~116 tok/s (measured)

4.2 rkllm3-server vs rknn3-toolkit-lite

Dimension	rknn3-toolkit-lite (Python API)	rkllm3-server (HTTP)
Stability	High (in-process calls)	Occasional hangs
VLM support	Native	Limited
Deployment	pip install wheel	deb package
Best suited for	VLM/CNN/multimodal	Pure LLM chat

5. FAQ

Symptom	Cause	Fix
`core_mask X is not match with npu core number 8!`	`-c` is not a core_mask, or mismatched with the model's core count	LLM uses `-c 0xff`, CNN uses `-c 1`
`core_mask 8192 is not match with npu core number 8`	A context value was passed to `-c`	Context is fixed at compile time; don't pass `-c`
`rkllm3-server` fails with `port already in use`	Port occupied	Switch with `--port 8081`
Python SDK reports `Connection refused`	Service not running / wrong port	Verify with `ss -tlnp \| grep 8080`

6. Next Steps

CNN Inference — -c 1 single-core models
Model Conversion — ONNX → RKNN
AI Agent Applications — integrate into Agent frameworks