Hybrid Vision SDK Overview

📋 Overview

HVAlgo is a high-performance algorithm library designed by ShiMetaPi for event cameras, providing a range of advanced event denoising, computer-vision, and image-restoration algorithms. Built on the ShiMetaPi Hybrid Vision SDK, it supports real-time event-stream processing and is suited to robotic vision, autonomous driving, high-speed motion analysis, and similar applications.

The ShiMetaPi Hybrid Vision SDK is composed of two independent SDKs — hybrid_vision_toolkit and hybrid_vision_algo — which implement interface control and algorithm processing for the hybrid vision camera respectively.

🚀 Where to start

• Want it running immediately: Your First C++ Program (open the camera and capture events in 5 minutes)
• Want to learn by task: Programming Guides (open camera / read events / record / denoise / display / tune)
• Want a no-code preview: MultiVision Studio
• Want ready-made examples: Samples Overview

📋 SDK overall architecture

shimetapi_Hybrid_vision_algo (algorithm-layer SDK)

• Role: This is the core algorithm-processing layer of the SDK, the middle layer of the architecture (the yellow part).
• Core function: It focuses on processing the raw data stream from the event camera and executing advanced computer-vision algorithms to improve data quality, extract useful information, or perform 3D understanding.
• Included modules:
  • Denoise: removes noise from the event stream to improve signal quality.
  • Interpolation: used to generate intermediate data points between events, or for spatiotemporal alignment/fusion with standard frame images.
  • Restoration: used to handle missing data or anomalies.
  • Computer Vision / 3D Vision (CV/CV3D): algorithms specific to event cameras or fusion applications — object detection, tracking, 3D reconstruction, pose estimation, etc.
  • Samples & data: sample code, models, or required datasets.
  • External dependencies: integrates third-party libraries to give users more device choices, e.g. shimetapi Hybrid vision toolkit, OpenEB (installed via command), etc.
• Effect: It provides processed, enhanced, or understood event data to upper-layer applications (APP). Application developers mainly call the interfaces of this layer to implement the advanced features of the event camera (playback, recording, analysis).

shimetapi_Hybrid_vision_toolkit (HAL-layer toolkit)

• Role: This is the Hardware Abstraction Layer (HAL) and base tool layer of the SDK, at the very bottom of the architecture (the dark-blue part labeled HAL).
• Core function: Provides hardware interaction and basic data-stream operations. It reads raw data from the physical interface, does preliminary processing (e.g. encode/decode), passes the data to the upper algorithm layer through its callback and buffer mechanisms, and also provides camera-device abstraction and control.
• Key modules / functions:
  • Read: gets the raw event data stream from the hardware interface.
  • Decoder: if event data is transmitted in a specific encoded format, decodes it into a processable format.
  • Callback: asynchronously pushes captured/decoded data to the upper layer via callbacks, so the application does not have to poll.
  • Camera: abstraction and control of camera devices (device discovery, open/close, parameter configuration, etc.).
  • Write: sends processed data or control commands back to the hardware (if supported).
  • Encoder: encodes processed data or recorded streams into a specific format.
  • Buffer: data-buffer management, responsible for data movement and memory reuse, smoothing the data flow between capture and processing.
• Supported hardware interfaces: the interfaces layer at the bottom of the block diagram lists USB, MIPI, etc. (... means more); the toolkit abstracts different physical interfaces into unified backends, hiding transport-layer differences.
• Effect: It abstracts the details of the underlying hardware and transport interfaces and provides a unified, relatively hardware-independent interface for the upper algorithm layer (shimetapi_Hybrid_vision_algo) to access and control the event-camera data stream. It handles the underlying communication protocol, data movement, and basic format conversion.

Summary of how the two relate and cooperate:

1. shimetapi_Hybrid_vision_toolkit (toolkit / HAL layer):
   • Deals directly with the hardware and the underlying system SDK.
   • Acquires raw data (Read), does basic format conversion (Decoder), delivers data via callbacks (Callback), manages the data flow with buffers (Buffer), and outputs data (Write, Encoder), etc.
   • Provides a standardized data-access interface to the upper (algo) layer.
2. shimetapi_Hybrid_vision_algo (algorithm layer):
   • Built on top of the toolkit layer.
   • Receives the standardized event data stream from the toolkit layer.
   • Uses its advanced algorithm modules (Denoise, Interpolation, CV/CV3D, etc.) to process, enhance, analyze, and understand the data.
   • Provides the processed, more valuable information to the topmost application (APP) — e.g. display (UI), Player playback, Recorder recording, Analyzer analysis.
   • Compatible with different vendor interfaces (currently supports ShiMetaPi HV Toolkit and OpenEB).