Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction

TL;DR: This paper addresses the problem of rolling shutter correction in complex nonlinear and dynamic scenes with extreme occlusion.

Overview

This paper addresses the problem of rolling shutter correction in complex nonlinear and dynamic scenes with extreme occlusion. Existing methods suffer from two main drawbacks. Firstly, they face challenges in estimating the accurate correction field due to the uniform velocity assumption, leading to significant image correction errors under complex motion. Secondly, the drastic occlusion in dynamic scenes prevents current solutions from achieving better image quality because of the inherent difficulties in aligning and aggregating multiple frames. To tackle these challenges, we model the curvilinear trajectory of pixels analytically and propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels. Besides, to reconstruct high-quality occlusion frames in dynamic scenes, we present a 3D video architecture that effectively Aligns and Aggregates multi-frame context, namely, RSA2-Net. We evaluate our method across a broad range of cameras and video sequences, demonstrating its significant superiority. Specifically, our method surpasses the state-of-the-art by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively.

Contributions:

We analytically model the trajectory in complex non-linear movements and present a novel geometry-based quadratic rolling shutter motion solver that precisely estimates the high-order correction field of individual pixels.
We propose a self-alignment 3D video architecture for high-quality frame aggregation and synthesis against extreme scene occlusion.
A broad range of evaluations demonstrates the significant superiority and generalization ability of our proposed method over state-of-the-art methods.

Motivation

Existing methods suffer from two main drawbacks. Firstly, they face challenges in estimating the accurate correction field due to the uniform velocity assumption, leading to significant image correction errors under complex motion. Secondly, the drastic occlusion in dynamic scenes prevents current solutions from achieving better image quality because of the inherent difficulties in aligning and aggregating multiple frames

To tackle these challenges, we model the curvilinear trajectory of pixels analytically and propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels. Besides, to reconstruct high-quality occlusion frames in dynamic scenes, we present a 3D video architecture that effectively Aligns and Aggregates multi-frame context.

Overview

Overview of the proposed method. We aim to estimate precise correction fields in nonlinear motion with the QRS motion solver and synthesize high-quality frames against dynamic scenes with extreme occlusion by a self-alignment 3D video architecture RSA2-Net.

The object moves at variable velocity and has a complex curvilinear trajectory. Note that the object has only 2 Dof in the image plane, and the time interval between frames is very small. Thus, we use a quadratic motion model to formalize the curvilinear trajectory of the pixel.

Quantitative Analysis and Generalization Ability

The results reported in table show that the proposed method outperforms the other eight RSC methods by large margins on Carla-RS, Fascte-RS, BS-RSC and ACC datasets. These superior performances significantly demonstrate the effectiveness of our model on highly dynamic scenes with occlusion and real-world curvilinear movements. We performed cross-tests on three datasets the proposed method demonstrates strong generalization performance, benefiting from the QRS motion solver.

Visual Comparisons

Visual comparisons on Carla-RS, Fastec-RS, BS-RSC and ACC datasets. The proposed method can effectively correct the rolling shutter distortion in complex nonlinear and dynamic scenes with extreme occlusion.

Citation

@InProceedings{Qu_2023_ICCV,
            author    = {Qu, Delin and Lao, Yizhen and Wang, Zhigang and Wang, Dong and Zhao, Bin and Li, Xuelong},
            title     = {Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction},
            booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
            month     = {October},
            year      = {2023},
            pages     = {10680-10688}
        }

For more information, check out the paper, code, and YouTube video:

Paper Code YouTube