RKLLM On-Device LLM Inference

RKLLM is Rockchip's on-device LLM inference runtime; it loads RKNN-format large language models on the RK182X and runs inference. Both rkllm3-server and the C API inference demos are built on the RKLLM runtime.

Overall block diagram

Application process
    ↓ OpenAI-compatible API / C API
rkllm3-server / rknn3_llm_demo
    ↓
librknn3_api.so (RKNN3 runtime)
    ↓
rknn3_transfer_proxy (PCIe proxy)
    ↓
PCIe Gen3 x4 → RK1828 NPU

1. Actual Components

Executables under /usr/bin/:

ls -lh /usr/bin/rkllm* /usr/bin/rknn3_* /usr/bin/rknn*

Output:

-rwxr-xr-x 1 root root 4.4M  8月19日 17:59 /usr/bin/rkllm3-server
-rwxr-xr-x 1 root root 1.1M  4月 9日 10:33 /usr/bin/rknn3_cnn_demo
-rwxr-xr-x 1 root root 921K  4月 9日 10:33 /usr/bin/rknn3_llm_demo
-rwxr-xr-x 1 root root  47K  4月 9日 10:33 /usr/bin/rknn3_model_test
-rwxr-xr-x 1 root root 735K  4月 9日 10:33 /usr/bin/rknn3_session_test
-rwxr-xr-x 1 root root 1.5K  4月 9日 10:33 /usr/bin/rknn3_startup
-rwxr-xr-x 1 root root 396K  8月19日 17:59 /usr/bin/rknn3_transfer_proxy
-rwxr-xr-x 1 root root  672  4月 9日 10:33 /usr/bin/rknn3_usb_startup.sh
-rwxr-xr-x 1 root root 1.8M  4月 9日 10:33 /usr/bin/rknn3_vlm_demo
-rwxr-xr-x 1 root root 6.0M  4月 9日 10:33 /usr/bin/rknn-console
-rwxr-xr-x 1 root root 6.2M  4月 9日 10:33 /usr/bin/rknn-smi

Libraries and firmware:

ls -lh /usr/lib/librknn*.so /lib/firmware/rknn3_rk1820.img

Output:

-rwxr-xr-x 1 root root 2.2M  8月19日 17:59 /lib/firmware/rknn3_rk1820.img
-rwxr-xr-x 1 root root 8.4M  8月19日 17:59 /usr/lib/librknn3_api.so
-rw-r--r-- 1 root root 7.4M  2025年 6月10日 /usr/lib/librknnrt.so
-rwxr-xr-x 1 root root 210K  8月19日 17:59 /usr/lib/librknnsmi.so

Key file sizes:

du -h /usr/bin/rkllm3-server /usr/lib/librknn3_api.so /lib/firmware/rknn3_rk1820.img

Output:

4.4M	/usr/bin/rkllm3-server
8.4M	/usr/lib/librknn3_api.so
2.2M	/lib/firmware/rknn3_rk1820.img

2. Installation

Pre-installed out of the factory. Manual installation:

# Files are copied directly into /usr/bin/ and /usr/lib/ (no .deb packages)
sudo cp librknn*.so /usr/lib/
sudo cp rkllm3-server rknn3_* /usr/bin/
sudo cp rknn3_rk1820.img /lib/firmware/

3. Querying NPU Status

Use the following 4 methods to check NPU status:

3.1 Process Check

ps aux | grep rknn3_transfer_proxy | grep -v grep

It returns two records: the main process + a random-suffix process; the hash suffix is randomly generated at every startup, and PID / CPU / memory are dynamic fields — do not hard-code a query like ps -C rknn3_transfer_proxy_<hash>.

3.2 Service Check

systemctl is-active rknn3.service

Output:

active

3.3 Device Check

ls -l /dev/pcie-rkep-*

Output:

crw------- 1 root root 10, 124  8月20日 16:56 /dev/pcie-rkep-0004:41:00.0

On the board side, the RK1828 PCIe card has two equivalent device nodes:
/dev/pcie-rkep-0004:41:00.0: PCIe EP (Endpoint) character device, created by the PCIe driver
/dev/dri/renderD128: DRM render node, used by rknn3_transfer_proxy to communicate with the NPU
Both actually point to the same RK1828 PCIe device; either one works.

3.4 PCIe BDF

lspci | grep -i 182a

Output:

0004:41:00.0 Processing accelerators: Rockchip Electronics Co., Ltd Device 182a (rev 01)

4. C API Function List

#include "rknn3_api.h"  // ← request the header file from Rockchip

Library file type:

file /usr/lib/librknn3_api.so

Output:

/usr/lib/librknn3_api.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=c202427e4b11543df2fd52dcb1b8dea46e668946, stripped

Function	Description
`rknn3_init(&ctx, &ext)`	Global initialization (context pointer + init_extend)
`rknn3_load_model_from_path(...)`	Load model (3 args: ctx, model, weight)
`rknn3_load_model_from_data(...)`	Load model (from memory data)
`rknn3_model_init(ctx, &cfg)`	Model initialization (core_mask passed via `rknn3_config.run_core_mask`)
`rknn3_run(...)`	CNN inference (inputs/outputs tensor arrays)
`rknn3_profile_mem(ctx)`	Inspect memory usage of each NPU node
`rknn3_session_destroy(...)`	Destroy session (LLM only)
`rknn3_session_run(...)`	Synchronous LLM inference (LLM only)
`rknn3_session_run_async(...)`	Asynchronous inference (LLM only)
`rknn3_session_clear_kvcache(...)`	Clear KV cache (LLM only)
`rknn3_session_enable_lora(...)`	Enable LoRA (LLM only)
`rknn3_session_disable_lora(...)`	Disable LoRA (LLM only)
`rknn3_destroy(ctx)`	Destroy context

CNN goes through rknn3_run(), LLM goes through rknn3_session_*(); do not mix them. For verified signatures, refer to rknn/rknn3-runtime/rknn3-api/include/rknn3_api.h in the SDK (for a compiled, runnable CNN example see INT8 Quantized Inference).

5. OpenAI-Compatible API

Endpoint	Method	Description
`/v1/models`	GET	List models
`/v1/chat/completions`	POST	Chat inference
`--model2` startup arg	—	Multimodal VLM

Start command:

rkllm3-server \
  -m /userdata/models/Qwen3-1.7B/Qwen3-1.7B.rknn \
  --weight /userdata/models/Qwen3-1.7B/Qwen3-1.7B.weight \
  --vocab /userdata/models/Qwen3-1.7B/Qwen3-1.7B.tokenizer.gguf \
  --embed /userdata/models/Qwen3-1.7B/Qwen3-1.7B.embed.bin \
  --embed-mmap \
  -a Qwen3-1.7B \
  --host 127.0.0.1 \
  --port 7878 \
  -c 0xff \
  -n 512

-c actually takes effect as the NPU core mask — Qwen3-1.7B is compiled with 8 cores declared, so you must pass -c 0xff; passing a number such as 8192 produces core_mask 8192 is not match with npu core number 8. The context length is determined by max_context_len at model compile time; see LLM Inference for details.

Testing the API endpoint:

curl -s http://127.0.0.1:7878/v1/models

Output:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen3-1.7B",
      "object": "model",
      "created": 1787214642,
      "owned_by": "rknn",
      "meta": {
        "vocab_type": 2,
        "n_vocab": 151936,
        "n_ctx_train": -1,
        "n_embd": 2048,
        "n_params": 0,
        "size": 0
      }
    }
  ]
}

6. FAQ

Symptom	Cause	Resolution
`rknn-smi` prints usage and exits	RK1828 PCIe not supported	Use `ps` / `systemctl` / `lspci` instead
`librknn3_api.so` not found	Runtime deb not installed	Install the `rknn3-runtime` deb
`rknn3_init` header file missing	`rknn3_api.h` not installed	Request it from Rockchip
`core_mask 8192 is not match ...`	Context number passed to `-c`	Use `-c 0xff` for LLM
`port 7878 already in use`	Service already running	Check with `ss -tlnp \| grep 7878`
`/v1/models` no response	Service not started	Run the start command in the background

7. Next Steps

INT8 Quantized Inference — quantization + CNN inference
MPP Multimedia Framework — SoC-side multimedia
NPU Overview — NPU hardware architecture

8. References

RKNN3 Model Zoo
Full documentation: docs/RK1820_RK1828_AI_Release-Note_CN.md in the SDK