Qwen3-1.7B LLM Text Chat

This example provides an OpenAI-compatible REST API (POST /v1/chat/completions) running Qwen3-1.7B plain-text chat on the RK3588 + RK1828.

Overall block diagram:

Client curl /v1/chat/completions
    │
    ▼
rkllm3-server (RK1828 NPU via PCIe)
    ├─→ Tokenize (gguf SentencePiece)
    ├─→ Embed lookup (mmap)
    ├─→ LLM decode (Qwen3-1.7B, 103 tokens/s)
    └─→ response (OpenAI ChatCompletion JSON)

Two processing domains:

RK3588: HTTP service process, tokenizer, JSON encode/decode
RK1828 (PCIe co-processor): rkllm3-server performs LLM decode via /dev/dri/renderD128 (PCIe address 0004:41:00.0)

1. Install dependencies

# 1. rkllm3-server binary
which rkllm3-server

# 2. RK1828 NPU node
ls -l /dev/dri/renderD128

# 3. rknn3 systemd service
systemctl is-active rknn3

# 4. Model files
ls -lh /userdata/models/Qwen3-1.7B/

Measured output (excerpt):

/usr/bin/rkllm3-server
crw-rw---- 1 root video 226, 128  8月27日 11:11 /dev/dri/renderD128
active
total 1.7G
-rw-r--r-- 1 linaro linaro  24M  Qwen3-1.7B.rknn
-rw-r--r-- 1 linaro linaro 1.1G  Qwen3-1.7B.weight
-rw-r--r-- 1 linaro linaro 5.7M  Qwen3-1.7B.tokenizer.gguf
-rw-r--r-- 1 linaro linaro 594M  Qwen3-1.7B.embed.bin

Binary / NPU node / service / model all present, ~1.7 GB total (runtime RSS ~2.1 GB; with 4 GB swap it runs stably).

2. Model files

/userdata/models/Qwen3-1.7B/
    Qwen3-1.7B.rknn              24M     LLM structure
    Qwen3-1.7B.weight            1.1G    LLM weights
    Qwen3-1.7B.tokenizer.gguf    5.7M    tokenizer (SentencePiece BPE)
    Qwen3-1.7B.embed.bin         594M    word embeddings (mmap-able)

~1.7 GB total (runtime RSS ~2.1 GB; with 4 GB swap it runs stably).

3. One-click deployment

cd /home/linaro/qwen3-llm
bash install.sh           # interactive (precheck → y/n → verify)
bash install.sh --yes     # skip confirmation

install.sh prechecks 3 things: the 5 files (rkllm3-server + 4 model files) / rknn3.service / /dev/dri/renderD128.

4. Daily usage

4.1 `make info` — inspect the environment

make info

Measured output (excerpt):

--- rkllm3-server ---
/usr/bin/rkllm3-server
--- 模型 ---
-rw-r--r-- 1 linaro linaro  24M  Qwen3-1.7B.rknn
-rw-r--r-- 1 linaro linaro 1.1G  Qwen3-1.7B.weight
-rw-r--r-- 1 linaro linaro 5.7M  Qwen3-1.7B.tokenizer.gguf
-rw-r--r-- 1 linaro linaro 594M  Qwen3-1.7B.embed.bin
--- rknn3 service ---
active

4.2 `make run` — start the server in the background

make run

Measured output:

[server] starting rkllm3-server on 127.0.0.1:7878...
[server] PID 12345, log=/tmp/qwen3_llm.log

Background start command:

nohup rkllm3-server \
  -m /userdata/models/Qwen3-1.7B/Qwen3-1.7B.rknn \
  --weight /userdata/models/Qwen3-1.7B/Qwen3-1.7B.weight \
  --vocab /userdata/models/Qwen3-1.7B/Qwen3-1.7B.tokenizer.gguf \
  --embed /userdata/models/Qwen3-1.7B/Qwen3-1.7B.embed.bin \
  --embed-mmap \
  -a Qwen3-1.7B \
  --host 127.0.0.1 --port 7878 \
  -c 8192 -n 256 \
  > /tmp/qwen3_llm.log 2>&1 &

4.3 `make health` — health check

make health

Measured output:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen3-1.7B",
      "object": "model",
      "created": 1787815326,
      "owned_by": "rknn",
      "meta": {
        "vocab_type": 2,
        "n_vocab": 151936,
        "n_ctx_train": -1,
        "n_embd": 2048,
        "n_params": 0,
        "size": 0
      }
    }
  ]
}

4.4 `make chat PROMPT="..."` — run one chat turn

make chat PROMPT="用一句话介绍深圳视美泰有限公司"

Measured output:

Qwen3-1.7B chat measured results — two make chat runs asking for a company intro / self-intro, with the thinking process and [usage] stats

Performance: ~103 tokens/s (measured timings.predicted_per_second).

Qwen3-1.7B LLM verified end to end: rkllm3-server → OpenAI-compatible API → Chinese understanding + thinking inference + streaming token output.

4.5 `make stop` / `make clean`

make stop     # kill PID, kill the background server
make clean    # stop + rm /tmp/qwen3_llm.log

5. REST API (OpenAI-compatible)

5.1 `GET /v1/models`

List loaded models:

curl -s http://127.0.0.1:7878/v1/models

5.2 `POST /v1/chat/completions`

OpenAI ChatCompletion API compatible.

Request:

{
  "model": "Qwen3-1.7B",
  "messages": [{ "role": "user", "content": "你好,介绍下你自己" }],
  "max_tokens": 128,
  "stream": false
}

Response:

{
  "choices": [
    {
      "finish_reason": "length",
      "index": 0,
      "message": { "role": "assistant", "content": "..." }
    }
  ],
  "model": "Qwen3-1.7B",
  "usage": {
    "prompt_tokens": 16,
    "completion_tokens": 128,
    "total_tokens": 144
  },
  "timings": { "predicted_per_second": 103.5 }
}

6. Client examples

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:7878/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="Qwen3-1.7B",
    messages=[{"role": "user", "content": "你好"}],
    max_tokens=256,
)
print(resp.choices[0].message.content)

curl

curl -s -X POST http://127.0.0.1:7878/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-1.7B",
       "messages":[{"role":"user","content":"你好"}],
       "max_tokens":256,"stream":false}' | jq .choices[0].message.content

Streaming (stream=true)

curl -s -X POST http://127.0.0.1:7878/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"Qwen3-1.7B",
       "messages":[{"role":"user","content":"你好"}],
       "max_tokens":256,"stream":true}'

7. Performance

Metric	Measured
Cold start → port ready	~5 s (LLM load + embed mmap)
Text generation speed	103.5 tokens/s
Time for 128 tokens	1236 ms
prompt_tokens	16 (Chinese system prompt + user question)
Model parameters	1.7 B
Vocab size	151,936
Embed dimension	2048
Context size	8192 (change via `-c`)
Runtime RSS	~2.1 GB

8. Key facts quick reference

Item	Value
Model path	`/userdata/models/Qwen3-1.7B/`
Binary path	`/usr/bin/rkllm3-server`
HTTP port	7878
Service API	`GET /v1/models` / `POST /v1/chat/completions`
Compatibility	OpenAI ChatCompletion (partial fields; `timings` is an rkllm extension)
PID file	`/tmp/.qwen3_llm.pid`
Log file	`/tmp/qwen3_llm.log`
embed	mmap (speeds up cold start)
Thinking mode	On by default (can be bypassed via prompt)
install.sh installs packages	No packages installed

9. FAQ

Symptom	Cause	Fix
`which rkllm3-server` finds nothing	Binary not installed	Ask the SDK provider to install `/usr/bin/rkllm3-server`
`make health` unresponsive after `make run`	Still loading (~5 s)	Retry with `sleep 10 && make health`
`make chat` returns `connection refused`	Server not running	`make run && sleep 5 && make health`
Generated content contains a `<think>...` block	Qwen3 enables thinking by default	Add "answer directly without thinking" to the prompt, or add `{"role": "system", "content": "不要思考,直接回答"}` to messages
Want `/completion` (llama.cpp style)	rkllm3-server is OpenAI-style only	Use `/v1/chat/completions` instead
embed not mmap'd, slow start	Makefile already adds `--embed-mmap`	Check the Makefile `run` target
Token speed slower than expected	Possible concurrency / memory pressure	rk equivalent of `nvidia-smi`: `cat /sys/kernel/debug/rknpu/load` (if present), or `free -m` for memory
`install.sh` reports `/dev/dri/renderD128 missing`	NPU driver not up	`sudo systemctl restart rknn3`

10. Known limitations

/completion not supported: rkllm3-server is OpenAI-style only, not the native llama.cpp API
Thinking mode: Qwen3 emits <think>... blocks by default; first answers are verbose (bypass by asking for "answer directly without thinking" in the prompt)
Token speed: ~103 tokens/s for the 1.7B model; the 4B model will be slower
No vision: plain-text chat only

RTSP Streaming + AI Analysis — camera + NPU + RTSP end to end
PaddleOCR-VL OCR — text-recognition HTTP service