Capture APS Image

Task goal

Our cameras are HVS hybrid vision — one chip outputs both events (EVS) and images (APS). The previous guides covered the event stream; this one covers how to capture the image stream (APS), and how to start EVS and APS at the same time for the minimal HVS dual-stream demo. For the HVS concept see HVS Hybrid Vision.

Classes and concepts involved

  • Shimeta::Frame::aps (a BufferView): raw bytes of the APS frame; the format is determined by Frame.format (usually NV12 on USB/S100, Gray8 on X5).
  • Shimeta::hv::Camera::GetFrame / SetImageCallback: synchronous / asynchronous APS retrieval.
  • Shimeta::hv::ImageData: the image packet in the async callback (pixels + width/height + format + ts).

Key changes in v2.0

v2.0 no longer hands you a cv::Mat directly. APS arrives as raw bytes (BufferView) in Frame.aps / ImageData.pixels, and the application must decode it according to Frame.format. The old startImageCapture(cv::Mat) and getLatestImage() no longer exist.

Synchronous: GetFrame + decode APS

#include <shimetapi/hv/camera.h>
#include <shimetapi/hv/device_config.h>
#include <opencv2/opencv.hpp>

int main() {
    Shimeta::hv::Camera cam;
    Shimeta::hv::DeviceConfig cfg;
    cfg.backend   = Shimeta::hv::Backend::Usb;
    cfg.vendor_id = 0x1d6b;
    cfg.product_id = 0x0105;
    cam.Init(cfg);
    if (!cam.StartStream()) return 1;

    Shimeta::Frame f;
    while (cam.GetFrame(f, 1000)) {
        if (f.aps.data && f.format == Shimeta::PixelFormat::NV12) {
            // NV12 = YUV420sp, total bytes = w*h*3/2
            cv::Mat nv12(f.height * 3 / 2, f.width, CV_8UC1,
                         const_cast<uint8_t*>(f.aps.data));
            cv::Mat bgr;
            cv::cvtColor(nv12, bgr, cv::COLOR_YUV420sp2BGR);   // NV12 → BGR
            cv::imshow("APS", bgr);
            if (cv::waitKey(1) == 27) break;   // ESC to quit
        }
    }
    cam.StopStream();
    cam.Destroy();
    return 0;
}

Asynchronous: SetImageCallback

cam.SetImageCallback([](const Shimeta::hv::ImageData& img) {
    if (img.pixels.data && img.format == Shimeta::PixelFormat::NV12) {
        cv::Mat nv12(img.height * 3 / 2, img.width, CV_8UC1,
                     const_cast<uint8_t*>(img.pixels.data));
        cv::Mat bgr;
        cv::cvtColor(nv12, bgr, cv::COLOR_YUV420sp2BGR);
        // Hand bgr off to the main thread for display; do not imshow inside the callback
    } else if (img.pixels.data && img.format == Shimeta::PixelFormat::Gray8) {
        cv::Mat gray(img.height, img.width, CV_8UC1,
                     const_cast<uint8_t*>(img.pixels.data));
        // X5: copy gray to the main thread for display
    }
});

Gray8 APS on X5:

if (f.aps.data && f.format == Shimeta::PixelFormat::Gray8) {
    cv::Mat gray(f.height, f.width, CV_8UC1,
                 const_cast<uint8_t*>(f.aps.data));
    // If color display is needed, cvtColor(gray, bgr, cv::COLOR_GRAY2BGR)
}

Callback-thread note

Image callbacks fire serially on the dispatch thread; cv::imshow belongs on the main thread — in the callback, just decode + copy and leave display to the main loop.

HVS dual stream: capture APS + EVS simultaneously

The value of HVS is capturing image + event together. In v2.0, a single GetFrame returns a Frame containing both .aps and .evs (the two streams are synchronized at pixel level inside the chip):

#include <shimetapi/codec/evt2_codec.h>
#include <vector>

Shimeta::codec::Evt2Decoder edec;   // stateful; construct outside the loop
Shimeta::Frame f;
while (cam.GetFrame(f, 1000)) {
    // EVS: decode events
    std::vector<Shimeta::EventCD> events;
    edec.Decode(f.evs.data, f.evs.size, events);

    // APS: decode NV12 on S100/USB; on X5 build the cv::Mat directly as Gray8
    // ...
}

You can also register SetFrameCallback (combined frame), SetEventCallback (event packet), and SetImageCallback (image) separately.

MIPI HVS

Backend::MipiHvs uses dual VCs: VC0 carries EVS (a RAW8 subframe stream, decoded with MipiRaw8Decoder), VC1 carries APS. On S100 the APS goes through ISP→PYM and outputs NV12; on X5 it is read directly through VIN and outputs Gray8. Both use the same GetFrame / SetImageCallback interfaces, but image conversion must branch on Frame.format. MIPI is ARM-only; see Toolkit — Quick Start for building.

Build and run

Dependencies are OpenCV, libusb, and the Hybrid Vision Toolkit; for USB permissions and troubleshooting see Your First C++ Program. RDK ARM64 MIPI uses neither VID/PID nor libusb.

Further reading