Real time video stabilization with reduced temporal mismatch and low frame buffer

Abstract--This paper proposes a real time video stabilization

method which only requires a low number of buffered frames.

The effect of temporal mismatch between the original and

deshaked frames to the black border artifacts is also considered.

This mismatch effect is then removed by delaying the motion

compensation with a suitable number of frames. To minimize the

number of buffered frames, the deshaking filter is designed to

have smallest filter delay. Simulation shows that the proposed

stabilization filter effectively removes the shaky motions and has

reduced black border artifacts.

I. INTRODUCTION

For non-professional recorded sequences, such as those

done without using tripods or steadicams, it is hard to achieve

good video with smooth motion, even with the hardware-based

image stabilization. Software-based video stabilization is thus

required to remove shaky motion and to bring more pleasant

visual effects. A video stabilization procedure generally

includes three steps as in Fig. 1. In motion detection, different

models such as translational, geometric, affine or perspective

models [1] [2] are used to estimate the global motion.

Generally, the motion path is assumed to be smooth, so a low￾pass or adaptive temporal filter [3] or motion path fitting [4] is

then applied to remove shaky motions. Next, frames are

motion compensated according to the difference between the

original and filtered motion trajectory. Finally, video cropping

or other post processing techniques is applied to remove black

borders or other artifacts.

Fig. 1. Video Stabilization Process.

Although shown to be effective, previous methods have to

use a very high number of frames, such 12 frames in [5] and

[6] or 20 frames in [4]. [7] even requires two-round procedure

over the whole sequence. These methods are thus suitable for

off-line video deshaking where there is no limitation on the

number of accessible frames. In this paper, we propose a real

time video stabilization algorithm which only needs a very low

number of buffered frames. Another issue with video

stabilization is temporal mismatch, which happens if the

motion compensation is not done at the frame delay

determined by the filter delay. With the limitation on the

number of buffered frame, this delay has an upper boundary.

We design the target low-pass filter to have lowest cut-off

frequency under this condition.

II. REDUCED MISMATCH VIDEO STABILIZATION

At first, a block-based motion estimation is implemented to

formulate the motion vector field (MVF). Then a translational

global motion vector estimation and MVF correction steps are

used to improve the accuracy of the block-based motion vector

field. Finally, a translation-scaling-rotation (TSR) motion

model is applied to estimate the global motion parameters.

Assume that the filtering is applied in a similar way for each

transformation parameters: scaling factor S0(t), rotation angle

θ(t), global motion vector GMVm(t), and GMVn(t). Such as for

GMVm(t), the filtering is done based on the accumulated GMV

to avoid the drifting of GMV value over long temporal filtering

( ) .)'(

' 0

∑=

=

t

t

m m AGMV t GMV t

(1)

If the filter is a causal IIR with filter order of N, then the

filtered GMV value is calculated by

∑ ∑

−

= − = −

= − × + − ×

1

'

'

'

'

( ) ( )' )'( ( )' )'(

t

t t N

m m

t

t t N

m AGMV t B t t AGMV t A t t AGMV t

(2)

where [A, B] are the filter coefficients. This deshaked

parameter for motion compensation is calculated as

( ) ( ) ( ). ' GMV t AGMV t AGMV t ∆ m = m − m

(3)

Fig. 2. Temporal delay effect on GMV.

Because of the filter delay, there is a temporal mismatch

between the input GMV and the filtered GMV′. Fig. 2 shows an

example of the temporal mismatch when applying the

averaging causal filter with length of 5 on a pure panning

sequence with no shaking. As seen in this figure, there is a

mismatch of two frames between the input and filtered GMV

trajectory with non-zero deshaked parameter ∆GMV(t). If the

motion compensation is implemented on the current frame, the

Real Time Video Stabilization

with Reduced Temporal Mismatch and Low Frame Buffer

Dũng Trung Võ1

, Surapong Lertrattanapanich1

, Cao Bui-Thu2

, and Yeong-Hwa Kim3

1 Digital Media Solutions Lab, Samsung Electronics US R&D Center, Irvine, CA 92612, USA

2

Ho Chi Minh University of Industry, Ho Chi Minh City, Vietnam

3 Department of Applied Statistics, Chung-Ang University, Seoul Korea

Motion

Compensation

GMV

Filtering

TSR

Estimation ME

Post

Processing

i.e. Video

Completion

Zooming/

Cropping

Input

video

Output

video

2012 IEEE International Conference on Consumer Electronics (ICCE) 1569490555

978-1-4577-0231-0/12/$26.00©2012 IEEE 61