NPU Driver and Runtime Library Architecture

An NPU (Neural Processing Unit) is a processor dedicated to AI computation, optimized at the hardware level for neural networks and deep-learning algorithms. Compared with traditional CPUs and GPUs, an NPU has higher computational efficiency and lower power consumption for AI-related tasks such as matrix operations and convolutions. It is widely used in image recognition, speech recognition, object detection, smart security, and large-model inference.

The NPU software stack of the GK7206 platform uses a layered design to provide developers with an efficient and flexible deep neural network application development environment. This architecture covers the entire path from low-level hardware drivers to upper-level runtime libraries and supports the development of various AI applications such as object recognition and image classification.

1 Software Stack Layers

The NPU software stack mainly includes the following layers from top to bottom:

Application layer and algorithm model library (ALG SDK): provides ready-to-use algorithm applications such as object detection and image classification for specific service scenarios. Typical examples are in sample/npu/demo_ai/, covering 21 AI functions including person detection, vehicle detection, and face recognition.
Runtime layer (XMEDIA_CL): an heterogeneous programming framework designed for the NPU. It provides a C-language API library to upper-layer applications, responsible for model loading, resource management, and task scheduling, and hides underlying hardware differences. The core headers are xmedia_cl.h and xmedia_cl_common.h, and the shared library is libxmedia_npu.so.
User-mode driver (NPU UMD): encapsulates the concrete driver invocation logic and works with the runtime layer to dispatch commands and exchange data.
Kernel-mode driver (NPU KMD): runs in Linux kernel space, responsible for NPU hardware initialization, power management, interrupt handling, memory mapping, and hardware-level scheduling of task queues. The kernel module is xm_npu.ko, loaded with the command ./load xm7206v11a -i.
Hardware: the NPU compute unit that executes specific neural network operator instructions.

Software stack layers

2 Core Architecture Features

The XMEDIA_CL framework provides the following key features by design to ensure efficient NPU utilization:

Zero-copy and Ping-Pong Buffer: supports a zero-copy mode to reduce memory copy overhead, and a Ping-Pong Buffer mechanism to overlap data flow and computation, improving throughput.
Sync/async modes: supports both synchronous and asynchronous invocation modes. Asynchronous mode allows the CPU and NPU to work in parallel, improving overall efficiency.
JIT and AOT support: supports both Just-In-Time (JIT) and Ahead-Of-Time (AOT) compilation, balancing flexibility with maximum performance.
Heterogeneous device extension: supports unified management and scheduling of heterogeneous devices such as CPU, NPU, and DSP.
Low memory footprint: the internal bufferless design has no extra memory footprint, suitable for resource-constrained embedded scenarios.

3 NPU Hardware Specifications

3.1 Supported Data Types

The NPU compute unit supports the following data types (defined in xmedia_cl_common.h):

Data type	Enum value	Description
INT8	`XMEDIA_CL_INT8`	8-bit signed integer; the most common type for quantized inference
UINT8	`XMEDIA_CL_UINT8`	8-bit unsigned integer
INT16	`XMEDIA_CL_INT16`	16-bit signed integer
UINT16	`XMEDIA_CL_UINT16`	16-bit unsigned integer
FP16	`XMEDIA_CL_FP16`	Half-precision floating point; balances accuracy and performance
INT32	`XMEDIA_CL_INT32`	32-bit signed integer
FP32	`XMEDIA_CL_FP32`	Single-precision floating point; used for high-precision scenarios
INT4	`XMEDIA_CL_INT4`	4-bit signed integer; for maximum quantization compression
UINT10	`XMEDIA_CL_UINT10`	10-bit unsigned integer; suitable for RAW image data
UINT12	`XMEDIA_CL_UINT12`	12-bit unsigned integer; suitable for RAW image data

Tip

In real deployments, most models use INT8 quantization, achieving the best inference performance with minimal accuracy loss. FP16 is suitable for scenarios that require higher accuracy.

3.2 Supported Data Formats

Format	Enum value	Description
RGB	`XMEDIA_CL_FORMAT_RGB`	Standard 3-channel RGB
RGrGbB	`XMEDIA_CL_FORMAT_RGrGbB`	Bayer format
BGbGrR	`XMEDIA_CL_FORMAT_BGbGrR`	Bayer format
GrRBGb	`XMEDIA_CL_FORMAT_GrRBGb`	Bayer format
GbBRGr	`XMEDIA_CL_FORMAT_GbBRGr`	Bayer format
YUV	`XMEDIA_CL_FORMAT_YUV`	YUV color space
YVU	`XMEDIA_CL_FORMAT_YVU`	YVU color space

3.3 NPU Management Interface

The NPU provides the following management interfaces (defined in xmedia_npu.h):

API	Description
`xmedia_npu_set_quick_start_flag(flag)`	Set the quick-start flag; when enabled, it speeds up NPU initialization
`xmedia_npu_get_quick_start_flag(&flag)`	Get the current quick-start flag
`xmedia_npu_get_proc_info(&proc)`	Get NPU memory mapping information (physical address and buffer length)
`xmedia_npu_get_usage_rate(&usage)`	Get the NPU utilization (percent) for performance monitoring

The following example shows how to query NPU utilization:

#include "xmedia_npu.h"

xmedia_float usage = 0.0f;
xmedia_s32 ret = xmedia_npu_get_usage_rate(&usage);
if (ret == XMEDIA_SUCCESS) {
    printf("NPU usage rate: %.2f%%\n", usage);
}

4 Core Data Types

The XMEDIA_CL framework defines a set of core data structures to describe the input/output tensor information of a model.

4.1 Tensor Shape (tensor_shape)

Describes the dimension information of a tensor:

typedef struct _xmedia_cl_tensor_shape {
    xmedia_cl_u32 ndims;                      // Number of dimensions (max XMEDIA_CL_MAX_DIMS_NUM=8)
    xmedia_cl_u32 dims[XMEDIA_CL_MAX_DIMS_NUM]; // Size of each dimension
    xmedia_cl_u32 pch[XMEDIA_CL_MAX_DIMS_NUM]; // Stride (pitch) of each dimension
    xmedia_cl_data_type type;                  // Data type
} xmedia_cl_tensor_shape;

For example, for an NHWC input tensor [1, 640, 640, 3], ndims=4, dims={1,640,640,3}.

4.2 Tensor Quantization Parameters (tensor_quant)

Describes the scale factor and zero point required for quantized inference:

typedef struct _xmedia_cl_tensor_quant {
    xmedia_cl_float scale;  // Scale factor
    xmedia_cl_s32 zp;       // Zero point
} xmedia_cl_tensor_quant;

The dequantization formula is: real_value = (int8_value - zp) * scale

4.3 Tensor (tensor)

The complete tensor description structure:

typedef struct _xmedia_cl_tensor {
    xmedia_cl_u32 tensor_id;       // Unique tensor identifier
    void *addr;                    // Data buffer address
    xmedia_cl_tensor_shape shape;  // Tensor shape
    xmedia_cl_tensor_quant quant;  // Quantization parameters
    xmedia_cl_u32 size;            // Total data size (bytes)
    xmedia_cl_s8 *name;            // Tensor name
} xmedia_cl_tensor;

4.4 Input/Output Tensor Info (tensor_info_inout)

Used by the xmedia_cl_graph_get_input/get_output interface to return tensor information:

typedef struct _xmedia_cl_tensor_info_inout {
    xmedia_cl_u32 num;                        // Number of tensors
    xmedia_cl_tensor *tensor;                  // Tensor array
    xmedia_cl_tensor_batch *tensor_batch;      // Batch information (used in dynamic batching)
    xmedia_cl_u32 *current_batch;             // Current batch size
} xmedia_cl_tensor_info_inout;

4.5 Model Memory Information (mem_info)

Describes the various memory sizes required to run a model, obtained through xmedia_cl_graph_query_model_info_from_file:

typedef struct _xmedia_cl_mem_info {
    xmedia_cl_u32 worksize;           // Workspace size
    xmedia_cl_u32 weightsize;         // Model weight size
    xmedia_cl_u32 inputsize;          // Input buffer size
    xmedia_cl_u32 outputsize;         // Output buffer size
    xmedia_cl_u32 codesize;           // Model code segment size
    xmedia_cl_u32 memory_reuse_type;  // Memory reuse mode (see Section 3.6)
    xmedia_cl_u32 private_data_size;  // Private data size
} xmedia_cl_mem_info;

Usage example:

#include "xmedia_cl.h"

xmedia_cl_mem_info mem_info;
xmedia_cl_s32 ret = xmedia_cl_graph_query_model_info_from_file(
    "model.xmm", &mem_info, XMEDIA_CL_MEM_INFO);
if (ret == XMEDIA_CL_SUCCESS) {
    printf("workspace: %u bytes\n", mem_info.worksize);
    printf("weight:    %u bytes\n", mem_info.weightsize);
    printf("input:     %u bytes\n", mem_info.inputsize);
    printf("output:    %u bytes\n", mem_info.outputsize);
}

4.6 Memory Reuse Modes

The NPU supports four memory reuse modes, which effectively reduce memory usage in multi-model scenarios:

Mode	Enum value	Reused content
Workspace only	`XMEDIA_CL_WORKSPACE`	workspace
Workspace + input	`XMEDIA_CL_WORKSPACE_INPUT`	workspace + input
Workspace + output	`XMEDIA_CL_WORKSPACE_OUTPUT`	workspace + output
All	`XMEDIA_CL_WORKSPACE_INPUT_OUTPUT`	workspace + input + output

You can query the reuse mode supported by a model via xmedia_cl_graph_get_memory_reuse_type(). For details, see XMM Model Loading.

5 Device and Context Management

5.1 Device Types

The XMEDIA_CL framework supports the following heterogeneous device types:

Device type	Enum value	Description
CPU	`XMEDIA_CL_DEVICE_CPU`	Use the CPU for inference
NPU	`XMEDIA_CL_DEVICE_NPU`	Use the NPU for accelerated inference
ALL	`XMEDIA_CL_DEVICE_ALL`	Query all available devices

5.2 Context Lifecycle

The Context is the core object that manages all resources in the XMEDIA_CL framework. A typical lifecycle is as follows:

#include "xmedia_cl.h"

// 1. Initialize the CL framework
xmedia_cl_s32 ret = xmedia_cl_init();
if (ret != XMEDIA_CL_SUCCESS) {
    printf("CL init failed: %d\n", ret);
    return -1;
}

// 2. Get NPU device IDs
xmedia_cl_device_id devices = NULL;
xmedia_cl_u32 num_devices = 0;
ret = xmedia_cl_get_device_ids(XMEDIA_CL_DEVICE_NPU, &devices, &num_devices);

// 3. Create a context
xmedia_cl_s32 err_code = 0;
xmedia_cl_context context = xmedia_cl_create_context(
    num_devices, &devices, &err_code);
if (context == NULL) {
    printf("Create context failed: %d\n", err_code);
    return -1;
}

// 4. Use the context for model loading and inference...
// (See ch03-xmm-model-loading.md)

// 5. Release resources
xmedia_cl_release_context(context);
xmedia_cl_release_device_ids(&devices, &num_devices);
xmedia_cl_uninit();

Note

Before destroying a context, ensure that all model graphs (Graphs) using that context have been unloaded via xmedia_cl_graph_unload().

6 Error Code Reference

The XMEDIA_CL framework defines detailed error codes (in xmedia_cl_common.h) to help quickly locate problems during development.

6.1 General Errors

Error code	Value	Description
`XMEDIA_CL_SUCCESS`	0	Operation succeeded
`XMEDIA_CL_OUT_OF_HOST_MEMORY`	-6	Insufficient host memory
`XMEDIA_CL_INVALID_VALUE`	-30	Invalid parameter value
`XMEDIA_CL_INVALID_BUFFER_SIZE`	-61	Invalid buffer size

6.2 Device and Context Errors

Error code	Value	Description
`XMEDIA_CL_INVALID_DEVICE_TYPE`	-31	Invalid device type
`XMEDIA_CL_INVALID_PLATFORM`	-32	Invalid platform
`XMEDIA_CL_INVALID_DEVICE`	-33	Invalid device
`XMEDIA_CL_INVALID_CONTEXT`	-34	Invalid context
`XMEDIA_CL_INVALID_COMMAND_QUEUE`	-36	Invalid command queue

6.3 Model Loading Errors

Error code	Value	Description
`XMEDIA_CL_INVALID_MODEL`	-64	Invalid model file or unsupported format
`XMEDIA_CL_READ_MODEL_FAIL`	-65	Failed to read the model file
`XMEDIA_CL_INVALID_BINARY`	-42	Invalid binary data
`XMEDIA_CL_INVALID_PROGRAM`	-44	Invalid program object
`XMEDIA_CL_ERROR_MODEL_TYPE`	-68	Model type error
`XMEDIA_CL_MODEL_DECOMPRESS_FAIL`	-71	Model decompression failed
`XMEDIA_CL_NOT_FIND_FILE`	-75	Model file not found

6.4 Memory and Address Errors

Error code	Value	Description
`XMEDIA_CL_INVALID_HOST_PTR`	-37	Invalid host pointer
`XMEDIA_CL_INVALID_MEM_OBJECT`	-38	Invalid memory object
`XMEDIA_CL_INSUFFICIENT_SIZE`	-66	Insufficient memory size
`XMEDIA_CL_ERROR_ADDR_ALIGN`	-69	Address not aligned (must be 8-byte aligned)

6.5 Runtime Errors

Error code	Value	Description
`XMEDIA_CL_INVALID_KERNEL`	-48	Invalid kernel function
`XMEDIA_CL_INVALID_KERNEL_ARGS`	-52	Invalid kernel arguments
`XMEDIA_CL_INVALID_OPERATION`	-59	Invalid operation
`XMEDIA_CL_INVALID_UNINIT`	-60	CL framework not initialized
`XMEDIA_CL_ALREADY_INIT`	-63	CL framework already initialized (duplicate initialization)
`XMEDIA_CL_OUT_OF_MAX_BATCH`	-74	Exceeded maximum batch size

6.6 Event Errors

Error code	Value	Description
`XMEDIA_CL_WAIT_EVENT_FAILED`	-56	Wait event failed
`XMEDIA_CL_INVALID_EVENT_WAIT_LIST`	-57	Invalid event wait list
`XMEDIA_CL_INVALID_EVENT`	-58	Invalid event

7 Core Concepts and Resource Management

The XMEDIA_CL architecture uses the "context" to uniformly manage all device resources. The core concepts are as follows:

Device: a hardware compute unit such as NPU, CPU, or DSP. Task queues queue commands onto a specific device for execution.
Context: the resource manager. It manages each device, the memory accessible to each device, the task queue for each device, the program, and each kernel function.
Model graph file: a binary file (.xmm format) generated by the compiler from an AI model, supporting heterogeneous instructions.
Task queue: used to queue kernel-function commands for execution.
Event: used to indicate the execution status of a task, and to explicitly establish dependency constraints between tasks.

8 Task Scheduling Flow

The context is responsible for unified device resource management. During execution, the user first loads the model file, and XMEDIA_CL automatically creates the program object and kernel functions based on the Graph structure. Once created, XMEDIA_CL places the kernel functions into the task queue and generates corresponding events.

Asynchronous execution mechanism: kernel-function execution in this architecture always uses asynchronous mode. After a user submits a command to the task queue, the CPU can do other work without waiting for the NPU command to complete; if you must wait for a command to complete, you can explicitly establish this constraint through an event, maximizing CPU and NPU parallelism.

8.1 Event Status

The execution status after a task is submitted is as follows:

Status	Enum value	Description
QUEUED	`XMEDIA_CL_QUEUED (0)`	Queued, waiting for execution
SUBMITTED	`XMEDIA_CL_SUBMITTED (1)`	Submitted to the device
RUNNING	`XMEDIA_CL_RUNNING (2)`	Running
COMPLETED	`XMEDIA_CL_COMPLETED (3)`	Execution completed
FAILED	`XMEDIA_CL_FAILED (4)`	Execution failed

You can query the current status via xmedia_cl_query_event_status() or block-wait for completion via xmedia_cl_wait_for_events().

8.2 Task Priority

XMEDIA_CL supports 4 levels of task priority, set via xmedia_cl_graph_set_schedule_prio():

Priority	Macro	Value
Lowest	`XMEDIA_CL_JOB_SCHEDULE_PRIO_MIN`	0
Medium	`XMEDIA_CL_JOB_SCHEDULE_PRIO_MEDIUM`	1
High	`XMEDIA_CL_JOB_SCHEDULE_PRIO_HIGH`	2
Highest	`XMEDIA_CL_JOB_SCHEDULE_PRIO_MAX`	3

// Set high priority
xmedia_cl_graph_set_schedule_prio(graph, XMEDIA_CL_JOB_SCHEDULE_PRIO_HIGH);