Temporal exposure inconsistency correction for CMOS captured video sequences

Abstract--The paper proposes a novel method to correct the

temporal inconsistency in exposure of video sequences captured

from CMOS cameras. The temporal exposure consistency is

improved by avoiding significant change in histogram over

frames. This is done by histogram equalization where the target

histogram is the temporal low-pass filtered result from the

previous frames' histogram. Simulations on different video

sequences verify the effectiveness of the algorithm.

I. INTRODUCTION

CMOS sensors are popularly implemented in hand-held cell

phone cameras and DSLR cameras due to their lower cost than

CCD sensors. Different than whole frame sampling in CCD

sensors, row by row sampling is utilized in CMOS sensors.

They are so called rolling shutter cameras. This sampling

scheme helps to reduce buffer memory on the sensor as well as

permitting on-chip auto-focus and white balance [1]. Due to

row-by-row sampling, there will be a time lapse between

different row samplings. If the motion is fast, straight lines and

structured objects located in different rows will be bent. This

causes shearing artifacts [2]. If the lighting changes during

multi-frame capture, camera exposure may be not the same

between rows. This causes the partial exposure artifacts for

still images and temporal exposure inconsistency over frames

in video sequences. For videos, the exposure changes lead to

tone changes which really annoy viewers.

Just a few works have addressed the artifacts from rolling

shutter cameras. Most of them focus on shearing artifact

removal such as [2] [3]. Software-based correction methods

for partial exposure and temporal exposure inconsistencies

have not been discussed. This paper proposes a novel method

to correct these tone changes in videos. The main idea is to

temporal low-pass filter the histograms of the neighboring

frames to avoid fast changes in histograms. The filtered

histogram is used as the target histogram for the current frame.

This is equivalent to histogram equalization or specification.

Histogram equalization has been widely studied and applied to

contrast enhancement [4], brightness preservation [5], image

sharpening [6] and tone reproduction [7]. Our work is similar

to tone correction, where the target tone specification is

specified by users.

II. TEMPORAL HISTOGRAM EQUALIZATION

The block diagram of the proposed temporal histogram

equalization is shown in Fig. 1. At first, the frame is converted

from RGB color space to Lab color space. This color space

approximates human vision and has perceptual uniformity

characteristics. Each color component is processed

independently. For example, L component’s histogram is

estimated based on pixels in current frame L(t, m, n) to form

the histogram h(t, k), where k is the bin location. k's value

range is [0, 255] for 8 bit pixels. To avoid fast change in tone

over frames, a temporal filter is applied in each bin over time

to determine the target histogram h'(t, k), such as for k

th bin

∑ ∑

−

= − = −

= − × + − ×

1

' '

,(' ) ( )' ,'('' ) ( )' ,'( )

t

t t N

t

t t N

h t k A t t h t k B t t h t k (1)

where [A, B] are the filter coefficients, N is the filter order and

h''(t, k) is the buffered processed histogram of previous frames.

It is then used as the target histogram in histogram equalization

step to formulate the enhanced output Leq(t, m, n). To avoid

artifacts of the histogram equalization, the process is only

applied to pixels having a value that falls within the range:





 ≤ ≤

=

L t m n otherwise

L t m n if L L t m n L

L t m n

eq

eq ,( , )

,( , ) ,( , )

' ,( , )

min max (2)

where Lmin and Lmax are pre-defined thresholds. The equalized

histogram heq(t, k) of L'eq(t, m, n) is buffered for future frame

processing. To avoid bins with zero appearance that may result

from the histogram equalization, an average spatial filter is

implemented on the equalized histogram

( ,( )1 ,( ) ,( )1 )

3

1

h t,('' k) = heq t k − + heq t k + heq t k + . (3)

Fig. 1. Block diagram of the temporal histogram equalization.

Fig. 2. An example of the temporal histogram equalization.

An example of the temporal histogram equalization is

shown in Fig. 2. In this example, the histogram of previous

frame t-1 is used as the target for the histogram equalization of

Temporal Exposure Inconsistency Correction

for CMOS Captured Video Sequences

Dũng Trung Võ1

, Surapong Lertrattanapanich1

, Cao Bui-Thu2

, and Yeong-Hwa Kim3

1 Digital Media Solutions Lab, Samsung Electronics US R&D Center, Irvine, CA 92612, USA

2 Ho Chi Minh University of Industry, Ho Chi Minh City, Vietnam

3 Department of Applied Statistics, Chung-Ang University, Seoul Korea

Input

video

Output

Histogram video

Estimation

Temporal

Filtering

Histogram

Equalization

Spatial

Filtering

Histogram

Buffering

Histogram

Histogram

t-1 frame

Temporal histogram

equalization

t frame Enhanced t frame

2012 IEEE International Conference on Consumer Electronics (ICCE) 1569490547

978-1-4577-0231-0/12/$26.00©2012 IEEE 7