Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution


Bangyan Liao1,2,*
Delin Qu1,3,*
Yifei Xue4
Huiqing Zhang1
Yizhen Lao1,†
1College of Computer Science and Electronic Engineering, Hunan University
2College of Electrical and Information Engineering, Hunan University
3Shanghai AI Laboratory
4Jiangxi Provincial Natural Resources Cause Development Center

We propose an accurate and fast bundle adjustment (BA) solution that estimates the 6-DoF pose with an independent RS model of the camera and the geometry of the environment based on measurements from a rolling shutter (RS) camera. This tackles the challenges in the existing works, namely, relying on high frame rate video as input, restrictive assumptions on camera motion and poor efficiency. To this end, we first verify the positive influence of the image point normalization to RSBA. Then we present a novel visual residual covariance model to standardize the reprojection error during RSBA, which consequently improves the overall accuracy. Besides, we demonstrate the combination of Normalization and covariance standardization Weighting in RSBA (NW-RSBA) can avoid common planar degeneracy without the need to constrain the filming manner. Finally, we propose an acceleration strategy for NW-RSBA based on the sparsity of its Jacobian matrix and Schur complement. The extensive synthetic and real data experiments verify the effectiveness and efficiency of the proposed solution over the state-of-the-art works.
Paper


CVPR 2023 (arXiv)
Citation
          @inproceedings{liao2023revisiting,
            title={Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution},
            author={Liao, Bangyan and Qu, Delin and Xue, Yifei and Zhang, Huiqing and Lao, Yizhen},
            booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
            pages={4863--4871},
            year={2023}
          }
Download

Code
Video

1-minute video
Results

Absolute trajectory error (ATE) of different RSBA methods after Sim(3) alignment to ground truth. The best results are shown in green. Since some methods will lose tracking without processing the whole sequence, thus we highlight the background of each cell with different colours depending on its corresponding DUR value. Specifically, DUR > 0.9, 0.5 < DUR ⩽ 0.9 and DUR ⩽ 0.5 are highlighted in light green , cyan , and orange .


Quantitative ablation study of RSSfM on TUM-RSVI [21] and WHU-RSVI [3] datasets. ATE: absolute trajectory error of estimated camera pose in meters (m), Runtime: time cost in seconds (s). Best and second best results are shown in green and blue respectively.


Three-view graph of reconstructions using SfM pipeline with GSBA [16], NM-RSBA [2] and proposed NW-RSBA.


Time cost of GSBA [4], DC-RSBA [3], NM-RSBA [1], NW-RSBA-0S (without Schur complement), NW-RSBA-1S (one-stage Schur complement to Jacobian matrices with series connection), and proposed NW-RSBA-2S (two-stage Schur complement to Jacobian matrices with parallel connection) with increasing camera number.


Camera pose (2 nd and 3 rd columns) and reconstruction (1 st column) errors of GSBA, DC-RSBA, DM-RSBA, NM-RSBA and NW- RSBA with increasing angular and linear velocity (1 st row) and noise levels in the image (2 nd row) in general scenes, also with increasing readout directions in degeneracy scene (3 rd row).


Ground truth and trajectories estimated by GSBA [4], NM-RSBA [1] and proposed NW-RSBA after Sim(3) alignment on 10 sequences from TUM-RSVI [6] and 2 sequences from WHU-RSVI [2] datasets.