MPP Media Processing Software

1 Overview

The Media Processing Platform (MPP) supports rapid development of application software. This platform hides chip-specific low-level processing from the application software and provides the MPI (MPP Program Interface) directly to the application to complete the corresponding functions. The platform enables rapid development of: input video capture, H.265/JPEG encoding, H.265 decoding, video output display, video image preprocessing (including denoising, enhancement, sharpening), image geometric correction, image stitching, black-and-white image fusion, image stabilization, audio capture and output, audio encoding/decoding, and more.

1.1 MPP Architecture

The MPP architecture is divided, from bottom to top, into the hardware layer, the operating system layer, the OS adaptation layer, the media processing platform, other drivers, and the application layer, as shown below.

• Hardware layer: consists of the chip and necessary peripheral components, including Flash, DDR, video Sensor or AD, audio AD, etc.
• Operating system layer: an OS based on Linux.
• OS adaptation layer: provides the underlying system-call functions, abstracts OS differences, and supports running the media processing platform on different operating systems or versions.
• Media Processing Platform (MPP): built on the OS adaptation layer, it controls the chip to complete the corresponding media processing functions. It hides hardware processing details from the application layer and provides an API to the application layer.
• Other drivers: drivers for hardware processing units outside the media processing platform, such as CIPHER and RTC drivers.
• Application layer: application software systems developed by the user, built on the media processing platform and other drivers.

1.2 MPP Video Stream

Note

In this document, "VI online/offline" refers to the online/offline mode between VI_CAP and VI_PROC. In this document, "VPSS online/offline" refers to the online/offline mode between VI_PROC and VPSS. In this document, "GDC online/offline" refers to the online/offline mode between GDC and VPSS. VI_PROC and GDC are always in offline mode.

The MPP video stream is shown below:

• VI_CAP module: obtains raw image data from the sensor via the MIPI protocol.
• VI_PROC module: crops, denoises, and otherwise processes the raw image data, and outputs multiple image streams at different resolutions.
• GDC module: performs geometric transforms on the image, such as fisheye correction, lens distortion correction, rotation, and perspective transform (optional).
• VPSS module: receives images sent from VI, GDC, and VDEC, can perform image enhancement, sharpening, stitching, and other processing, and can output multiple image streams of different resolutions from the same source.
• Encoding module: receives the image data output by VPSS, encodes it according to different protocols, and outputs the corresponding bitstream.
• VO module: receives the images output by VPSS, performs playback control and other processing, and outputs them to peripheral video devices according to the user-configured output protocol.

pipe mode configuration notes

1. For any pipe, GDC and VPSS cannot both be in online mode at the same time.
2. Only pipe0 supports VI online mode.
3. When pipe0 is in VI offline mode, the GDC mode and VPSS mode of all other pipes must match those of pipe0.

---

2 System Control Module

Based on chip characteristics, system control performs the basic system initialization, and also manages abnormal exits of each MPP (Media Processing Platform) service module, provides version information for the current MPP system, provides binding functions, provides large-block physical memory management, and provides OST and QOS value configuration for modules.

2.1 SYS Module Initialization and De-initialization

SYS module initialization specifies the working mode between modules in a pipe. Any MPP service must perform system initialization before running; after the service ends, you must perform system de-initialization to release resources. Example (required header: xmedia_sys.h):

xmedia_s32 mpp_sys_init()
{
    xmedia_s32 ret = XMEDIA_SUCCESS;
    ret=xmedia_sys_exit();
    if(ret!=XMEDIA_SUCCESS)
    {
        printf("xmedia_sys_exit failed !\n");
        return ret;
    }
    xmedia_sys_config sys_config={0};
    /*
     |Note:
     |  1. For any pipe, GDC and VPSS cannot both be in online mode at the same time.
     |  2. Only pipe0 supports VI online mode.
     |  3. When pipe0 is in VI offline mode, the GDC mode and VPSS mode of all other pipes must match those of pipe0.
     */
    sys_config.pipe_mode[0].vicap_viproc_mode = XMEDIA_WORK_MODE_ONLINE;
    sys_config.pipe_mode[0].viproc_vpss_mode = XMEDIA_WORK_MODE_ONLINE;
    sys_config.pipe_mode[0].gdc_vpss_mode =XMEDIA_WORK_MODE_OFFLINE;
    ret=xmedia_sys_init(&sys_config);
    if(ret!=XMEDIA_SUCCESS)
    {
        printf("xmedia_sys_init failed !\n");
        return ret;
    }
    else
    {
        printf("xmedia_sys_init success !\n");
        return XMEDIA_SUCCESS;
    }

}

Usage notes

1. In a single process, repeated initialization still returns success, but it has no actual effect on the running state of MPP; the first valid configuration is the effective result. To change the initialization configuration, call xmedia_sys_exit first.

2. In multi-process, each process must perform initialization. Only one valid configuration (non-null input parameter) is supported. If process a has been initialized with a non-null input parameter, the input parameter of other processes' initialization must be the same as that of process a or null; if process a has been initialized with a null input parameter, it has no effect on other processes' initialization.

3. The pipe numbers of VI and VPSS on the same data path must be identical.

2.2 Video Buffer (VB) Management

The video buffer pool mainly provides large-block physical memory management for media services. It is responsible for memory allocation and reclamation, makes full use of the buffer pool, and allows physical memory resources to be used reasonably across media processing modules. A video buffer pool consists of a set of buffer blocks of the same size with contiguous physical addresses. A common video buffer pool must be configured before any MPP service. Depending on the service, the number of common buffer pools, the block size, and the number of blocks differ.

• Common buffer pool configuration:

  The system supports configuring multiple common buffer pools, each containing several blocks of the same size. During configuration, use the commont_pool member of the xmedia_vb_cfg structure to specify the number of pools and the block size and count of each pool.

  • Block size: must be calculated based on the size and pixel format of the largest-resolution image in the service; can be obtained through xmedia_vb_get_buffer_config.
  • Block count: must be evaluated based on the number of concurrent service streams; too few causes block acquisition failures, while too many wastes memory.

• Module-private buffer pool: In addition to common buffer pools, some modules (such as ISP and VDEC) also support configuring module-private buffer pools for optimization of specific service scenarios. This is configured through the supplement_config member of the xmedia_vb_cfg structure.

• Buffer block acquisition and release:

  • Acquire: xmedia_mpi_vb_get_block, gets a free block from the specified common buffer pool.
  • Release: xmedia_mpi_vb_release_block, a block must be released after use, otherwise the buffer pool will be exhausted.

Note

Because other modules depend on the common buffer pool, vb initialization must be performed before other modules are initialized (called after sys initialization). It can be called multiple times and returns success, but only the first configuration takes effect.

xmedia_s32 mpp_vb_init()
{
    xmedia_s32 ret = XMEDIA_SUCCESS;

    ret = xmedia_vb_exit();
    if(ret!=XMEDIA_SUCCESS)
    {
        printf("xmedia_vb_exit failed !\n");
        return ret;
    }

    xmedia_vb_config vb_config={0};// The configuration must be zeroed first; otherwise random non-zero values may appear, causing an illegal configuration and vb pool initialization failure.
    /*
    #define XMEDIA_VB_SUPPLEMENT_ISP_INFO_ENABLE     1
    #define XMEDIA_VB_SUPPLEMENT_VDEC_INFO_ENABLE    (1 << 1)
    supplement_config: supplemental memory pool configuration; bit[0] and bit[1] are valid, the rest are illegal.
    bit[0]: 1 enables ISP memory allocation, 0 disables it;
    bit[1]: 1 enables VDEC memory allocation, 0 disables it.
    */
    vb_config.supplement_config=XMEDIA_VB_SUPPLEMENT_ISP_INFO_ENABLE|XMEDIA_VB_SUPPLEMENT_VDEC_INFO_ENABLE;
    vb_config.max_pool_cnt = 25;//Maximum number of memory pools


    xmedia_vb_base_info vb_base_info = {0};
    vb_base_info.video_fmt = XMEDIA_VIDEO_FMT_LINEAR;//Video storage format
    vb_base_info.pixel_fmt = XMEDIA_VIDEO_PIXEL_FMT_YVU_SEMIPLANAR_422;//Pixel format
    vb_base_info.cmp_mode = XMEDIA_VIDEO_COMPRESS_MODE_NONE;//Image compression mode
    vb_base_info.bit_width = XMEDIA_VIDEO_DATA_WIDTH_8;//Data width
    vb_base_info.width = 1920;//Width
    vb_base_info.height = 1080;//Height
    vb_base_info.align = DEFAULT_ALIGN;//Alignment
    vb_base_info.ainr_attr.ainr_en = XMEDIA_FALSE;//AINR disabled

    xmedia_vb_cal_cfg vb_cal_cfg = {0};
    xmedia_vb_get_buffer_config(&vb_base_info,&vb_cal_cfg);//Calculate the block size

    vb_config.common_pool[0].block_size = vb_cal_cfg.vb_size;//Block size - total size of a video image frame
    vb_config.common_pool[0].block_cnt = 3;//Number of blocks
    vb_config.common_pool[0].map_mode = XMEDIA_VB_MAP_MODE_NONE;//Kernel virtual address mapping mode of the VB
    /* When mmz_name is all zero, the pool is created from the anonymous mmz zone by default */

    ret = xmedia_vb_init(&vb_config);

    if(ret!=XMEDIA_SUCCESS)
    {
        printf("xmedia_vb_init failed, ret = 0x%x !\n", ret);
        return ret;
    }
    else
    {
        printf("xmedia_vb_init success !\n");
        return XMEDIA_SUCCESS;
    }
}

2.3 Binding Mechanism

MPP supports a module binding mechanism. Through the binding API xmedia_sys_bind, a data receiver binds a data source to establish the relationship between the two (only a data receiver is allowed to bind a data source). After binding, data generated by the data source is automatically sent to the receiver, eliminating the need for frequent application-layer intervention. The binding relationships supported by MPP are shown in the table below:

Data source	Data receiver
VI	VPSS / MCF
MCF	VPSS
VPSS	VO / VENC
VDEC	VPSS / VO

Required Headers and Libraries

Headers: common.h, xmedia_sys.h

Library: libxmedia_common.a

• Binding API:

//Establish a binding:
xmedia_s32 xmedia_sys_bind(const xmedia_chn_info *src_chn,const xmedia_chn_info *dest_chn);
//Unbind:
xmedia_s32 xmedia_sys_unbind(const xmedia_chn_info *src_chn,const xmedia_chn_info *dest_chn);

Note

1. A single data receiver can only be bound to one data source.

2. VI and VDEC, as data sources, send data to other modules by channel; the user should set the device number to 0, and the SDK does not check the input device number. In all other cases, both the device number and channel number must be specified.