AI Noise Reduction (AI_NR)

AI NR is a denoising technology that uses deep learning in the Bayer Raw domain. Compared with traditional denoising algorithms, AI denoising removes noise more cleanly with finer-grained noise patterns, providing a higher signal-to-noise ratio in dark environments or at very high gain, so the image can be seen more clearly.

1 Overview and Working Principle

The AI NR tool's logical architecture follows the "large-model distillation + small-model fine-tuning" approach. First, using the FPN (fixed-pattern noise) and noise parameters obtained through calibration, together with a general-purpose denoising large model, the captured Noisy Raw data is processed to generate Noisy-Clean Raw data pairs. The small model is then trained with this data and finally quantized and deployed to the device side.

The complete usage flow has six stages:

1. Preprocess: data preprocessing; generate training data pairs.
2. Float Finetune: floating-point model fine-tuning training.
3. Float Inference: floating-point model inference verification.
4. Quant Finetune: quantized model fine-tuning training.
5. Quant Inference: quantized model inference verification.
6. Quant Deploy: quantized model deployment export.

2 Environment Setup and Data Preparation

2.1 Environment Setup

• Hardware environment: confirm the basic information of the current sensor, including the data bit-width, Bayer pattern, resolution, and the ISO value at the maximum Again.
• Software environment: install miniconda and configure the environment required to run ai_nr.

2.2 Data Preparation

Before training, you must prepare the following data:

• Noise parameters: confirm the noise parameters (five in total) of the sensor to be fine-tuned and the FPN (must include black-level information).

• FPN data: capture 100~200 black frames. During capture, the lens must be fully black, the exposure time must match the actual darkest scene, again set to maximum, ispdgain=1, and sensor dgian=1.

  Note

  The shorter the exposure time, the fewer bad pixels the FPN exposes, and the weaker the label's ability to learn bad-pixel removal. When the sensor is changed, the FPN data must also be replaced.

• Training data: capture the Raw sequence of real scenes along with the corresponding meta information and AWB and CCM information. The data storage format is: each group of data is stored in a folder named by the ISO value, containing ***.raw, ***.txt, awb.txt, and ccm.txt.

3 Model Training, Quantization, and Deployment Flow

The tool controls each stage by modifying the config.yml file.

Stage 1: Preprocess (Data Preprocessing)

Generate the Noise-Clean Raw data pairs for the current distillation through the large model.

1. Configure YML: state: "preprocess", substate: "preprocess_sm".
2. Fill in parameters: enter bit, max_dgain, max_iso, bayer_pattern, and sensor_id in the data field.
3. Fill in noise parameters: enter the sensor's Gaussian-Poisson calibration parameters (KA, KB, BA, BB, BC) at the end of the YML.
4. Fill in paths: configure fpn_dir (FPN path), data_dir (distillation data path), dump_pre_root (output path), and the teacher model path.
5. Run and check: after running, view the generated input-label visualization result under dump_pre_root to confirm that preprocessing is normal.

Stage 2: Float Finetune (Floating-Point Fine-Tuning)

Train the floating-point small model in two stages.

1. Configure YML: state: "float", substate: "float_finetune".

2. Training settings: in the train field, set stage (1 or 2), learning_rate, batch_size, etc.

3. Model configuration: in the model field, choose net_name (the nearest match by sensor resolution, e.g. 3M picks "4M", 6M picks "8M").

4. Check the result: after training, input, label, and predict images are generated under the train_dump path; check whether the color and brightness correspond.

   Suggestion

   Training is recommended to exceed 60 epochs. If the loss is NaN, check whether the FPN bit / sensor bit is correct, or try lowering the learning rate and increasing the batch_size.

Stage 3: Float Inference (Floating-Point Inference)

Visually verify the effect of the floating-point model.

1. Configure YML: state: "float", substate: "float_inference".
2. Input model: use the higher-PSNR pth model produced by float_finetune.
3. Inference data: fill in the benchmark path; you can select a specific ISO for inference via select_iso.

Stage 4: Quant Finetune (Quantization Fine-Tuning)

Convert the floating-point model into a quantized model, in two stages.

1. Configure YML: state: "quant", substate: "quant_finetune".

2. Stage 1: fill in the best_float_model path in the model field. After running, quant_finetune_stage1.pth is generated.

3. Stage 2: point best_quant_model to the pth generated in Stage 1, and continue training to generate quant_finetune_stage2.pth.

   Suggestion

   Quant training is recommended to exceed 10 epochs. If new data is added, generate a new .h5 file and update the train_name list.

Stage 5: Quant Inference (Quantization Inference)

Verify the effect of the quantized model. The output must show no visible difference from the floating-point model output, with no obvious sharpness or color differences.

1. Configure YML: state: "quant", substate: "quant_inference".
2. Input model: use the pth model from quant_finetune_stage2.

Stage 6: Quant Deploy (Quantization Deployment)

Export the ONNX model and deployment files for TVM compilation.

1. Configure YML: state: "quant", substate: "quant_deploy".
2. Key parameters:
   • padding_w: fixed at 32 (16 for 7206_8M deployment).
   • padding_h: formula is $\text{padding_h} = \frac{\left(\left|\frac{h}{(\text{row_num} * 32)}\right| + p_s_ h\right) \times 16 - \frac{h}{(\text{row_num} * 2)}}{2}$
   • one_step_deploy: set to True to run all three stages at once.
3. Output: generate a _tvm.encrypted model, a weights txt, and the corresponding bin file.

Note

If deployment reports "The model has overflow!!!!", locate whether the overflow is in the Quant1 or Quant2 stage, and use the model from the non-overflow stage for the Deploy test.

4 Key Parameter Configuration

4.1 Base State Control

Parameter name	Value range	Description
state	"preprocess", "float", "quant"	Controls the tool's functional state. preprocess means data processing; float means floating-point model operations; quant means quantized model operations.
substate	"float_finetune", "float_inference", "quant_finetune", "quant_inference", "quant_deploy", "preprocess_sm"	Specific sub-state. When configuring float operations, state must be set to float; when configuring quant operations, state must be set to quant.

4.2 Model Configuration

Parameter name	Description	Value range
net_name	Select the model resolution; pick the nearest match by sensor resolution	"2M_3g", "2M_5g", "4M_8g", "4M_12g", "8M_19g"
best_float_model	Path to the pre-trained floating-point model / floating-point model used in quantization	User-defined path
best_quant_model	Path to the pre-trained quantized model / quantization Stage-2 initial model	User-defined path

4.3 Training Configuration

Parameter name	Description	Value range
stage	Stage of floating-point or quantized model training	1 / 2
learning_rate	Training learning rate	0.000001 ~ 0.01
batch_size	Training batch size	1 ~ 16

4.4 Deployment Configuration

Parameter name	Description	Value range
one_step_deploy	Quantized model deployment switch	True / False
deploy_version	Deployment version	"7206"`, `"7606"
layer_bit	Quantization bit-width	10 / 12

5 Model Tuning and Troubleshooting

5.1 Sensor Noise Model Calibration

Accurate noise calibration directly affects the pipeline's effectiveness. Calibration notes:

• Confirm that the sensor's bad-pixel removal mode is disabled (enabling DPC may affect the noise model and sharpness).
• Confirm that the AE SensorDgain is 1x (disabled by default in AI mode).
• Verify that the calibrated K/B curve matches expectations.

5.2 AI Pipeline Debugging

After completing the base traditional pipeline debugging, load the AI model for finetuning:

• Parameter switch: in BNR, set both ainrposswitch and coringposswitch to post.
• Threshold setting: enable the AINR switch, and set switchthlow and switchthhigh; it is recommended that thdhigh > thdlow >= ISO at againmax.

5.3 Common Issues Q&A

Symptom	Possible cause and troubleshooting
Float Finetune loss is NaN	1. Check whether the train_dump thumbnails are abnormal; 2. Check whether the FPN bit / Sensor bit in the YML is correct; 3. Check whether the Input and Label content, color, and brightness correspond; 4. Try lowering the learning rate and increasing the batch size (e.g. 24, 32).
Bleeding at extremely high ISO	Usually the temporal strength is too strong. It is recommended to first disable YUVTNR and observe whether the bleeding strongly correlates with BayerTNR; if so, fine-tune and lower the tnrmd threshold; if disabling YUVTNR helps, check the YUVTNR tnrmd and Mdwinsize.
Moving objects are blurry	1. Check whether the STNR YNR strength is too strong; 2. Adjust the motion re-overlay noise strength in BNR; 3. Adjust the K/B gain values in the AINR module to tune the AI denoising strength.
Quant Deploy reports overflow	Locate whether the overflow is in the Quant1 or Quant2 stage. Deploy with the Quant1 result; if there is no overflow, the overflow was caused by Quant2 training.
Bad pixels, jitter, pseudo-texture	These are overly-strong-detail issues; they can be mitigated by enabling smooth_on. Bad pixels can be tuned by adjusting the bpc strength.
Color cast	Check whether the training data pairs have a color cast; if not, check the difference between the test scene and the training scene, and if the difference is large, add data for the test scene.

---

6 sample_ainr Usage Steps

- Step 1: Build

# Run from the SDK root directory:
source build/env.sh
make clean
make build -j
make sample

- Step 2: Copy the executable and the neuron_network.xmm model file to the board

The neuron_network.xmm model file path: /tools/linux/pq_board/arm-gcc12.2.0-linux-uclibceabi/configs/sc465sl/neuron_network.xmm

Note

Note: the on-board storage paths of the model file and the executable must follow these rules: The executable accesses the model file at: ../../tools/linux/pq_board/arm-gcc12.2.0-linux-uclibceabi/configs/sc465sl/neuron_network.xmm.

You can modify the following three lines in sample_comm_isp.c(56-80).

- Run sample_ainr

Parameter description

---

Usage: ./sample_ainr [scene_mode]

scene_mode:

0:  Single-stream linear ainr 15fps

1:  Dual-stream linear ainr 7fps (switch)

2:  Dual-stream linear ainr 7fps

3:  Dual-stream linear ainr 7fps (external switch) stream_mode: VENC output

e.g: ./sample_vio 0

---