Pixel Based Temporal Analysis Using Chromatic Property for Removing Rain from Videos

The raindrops degrade the performance of outdoor vision system, and it brings difficulties for objects detection and analysis in image sequence. In this paper, we propose an algorithm detect moving objects using chromatic based properties in rain-affected videos captured by outdoor vision systems. Thus the raindrop removal algorithm includes two parts that is removal raindrops in background and removal raindrops in moving objects. Since the degradation made by raindrops is complex and appears as various changes. The raindrops detection function considering the chromatic properties of image sequence is induced, which does not need the velocity and time information of raindrops. Therefore, it is suitable for all the blur effects caused by raindrops. The removal raindrops are able to distinguish accurately the raindrops-affected pixels from the immovable or movable objects. Although the objects are moving in the rain, the algorithm is also effectual. The experiment results show that the proposal algorithm is able to remove the raindrops and improve the quality of image sequence remarkable


Introduction
The rain-affected image sequences annoy human viewers and degrade image quality.The degraded images also decrease performance of computer vision algorithm in areas such as object detection, tracking, segmentation, video surveillance.Now many algorithms are proposed to remove obvious rain streaks.But there are still two difficult points needing further research.1.To remove light rain.Rain streaks are too small to be detected in single frame when the background is complex, but viewer can feel rain streaks in videos.2. To get accurate intensity value.Due to the imaging precision and video compression algorithms, it's difficult to get the true intensity value of a rain-affected pixel.The existing algorithms just have good effectiveness for detecting and removing obvious and clear rain streaks.So to resolve the two difficult points described above, in this paper, we propose a general pixel based algorithm using chromatic property for removing rain from an ordinary video.

Previous work
The methods of removing rain streaks can be classified into two types: pixel-based method and frequency-based method.The first pixels-based method used a temporal median filter in (Starik, 2003).This works during moderate rain in which the scenario is not seriously corrupted.But moving objects will cause blur effects.
For removing rain streaks during video acquisition, Garg and Nayar (2007, pp.3-27) proposed modifying the camera parameters by increasing exposure time or reducing the depth of field.But for some types of cameras their parameters are not adjustable.For removing rain streaks in video, Garg and Nayar (2004) supposed that there are few raindrops will cover three consecutive frames.So if a pixel is covered by one raindrop, the intensity change of this pixel between this frame and its previous frames is equal to the intensity change between this frame and its latter frame.Then they used a linearity constraint to reject improperly detected pixels.Finally they calculated the binary rain field to segment rain area.This method has two limits: 1.In heavy rain, raindrops could frequently affect the same position in three consecutive frames.2. Due to noise or video compress algorithm, the linearity constraint is not always valid.Zhang at al (2006, pp.461-464) proposed another pixel-based algorithm using the chromatic constraint.They found If a pixel is covered by raindrops, the varieties of the intensities of , , and , are approximately the same.This property is used to segment moving objects and rain area.But in some videos, the intensity changes of , , and in area where comprise object motion are far less than in rain-affected area.It is difficult to find an appropriate threshold that is suitable for both stationary and dynamic objects.Barnum at al (2007) used frequency information to detect and remove rain streaks.They used a blurred Gaussian to approximate the blurred effect caused by a moving raindrop.It works when the streaks are clear.But for light rain or heavy rain, the model of a blurred Gaussian is not always effective.

Our work
This paper focuses on rain removal in video.If a video is captured by a moving camera, the frames can be stabilized in advance.So to make the problem simpler, we mainly focus on scenario comprising a stationary background and some moving objects captured by a stationary camera.
First, we make a further study of the raindrop's model, and give a general detecting method using chromatic property which is able to avoid noise effect.Then we estimate the distribution of the detected variable to reject improperly detected pixels.In the removal step, we use the imaging model to restore the background information.In this paper, we don't use the information of raindrops' velocities or shapes.Therefore our method is suitable for various types of raindrops.The result shows that our method is effective and has a better performance.Garg and Nayar (2007, pp.3-27) got a conclusion that when a raindrop is passing through a pixel, the intensity of its image is brighter than the background.This imaging process is illustrated by the formula:

Where
is the intensity of this pixel affected by the raindrop.is the time during which the raindrop projects onto the pixel.is the irradiance caused by the raindrop during the time , is the average irradiance of the background.
is the exposure time of the camera.
If the background is stationary, or the motion of it is slow, we can use the average irradiance value to calculate the background intensity of the pixel over the time duration . (2) Also we use the time-averaged irradiance to compute the intensity of raindrops.

R G
r r

E x y E x y dt
(3) Use to denote the change in intensity at a pixel due to a raindrop, we obtain (4) Let , , rewrite Eq.( 3) and Eq.( 4) (5) Here means the equivalent ideal intensity caused by raindrops during the exposure time .
means the ratio of rain-affected time to the exposure time.
From Eq.( 6) we know that the intensity change of pixels along a rain streak is proportional to its background values.But usually the intensity values of pixels along a rain streak are similar. (7) proximate to the intensity of a little white sphere.Therefore the equivalent ideal intensity and the ratio are uniform respectively in , , channels.That is, although a falling raindrop appears as complicate shape and intensity change along the falling direction, the values of and at each pixel are equal respectively in three channels.But the Eq.( 6) shows the relation of rain-affected pixel and its background.When rain is heavy, raindrops usually cover the same position in consecutive frames.It's hard to get the accurate value of background.In the next part, we will derive the formula of the relation between rain-affected pixels in consecutive frames.

Relation of Background pixel and rain pixel
Consider pixels at the same position in two consecutive frames, one is background pixel, and the other is rain-affected pixel.The brighter one is rain-affected pixel.denotes the vector of intensity change between frames in , , channels.is the vector of background intensity in three channels.We rewrite the Eq.( 6) (8) Where .
From the discussion above, we get .We still use to denote this value.

Relation of Two Rain Affected Pixels
When rain is heavy, or the frames extracted from videos are not consecutive frames.Sometimes the pixels at position in both frames are covered by raindrops.We use denotes the intensity vector of the brighter one, use denotes the other one.is its background intensity vector.
and is the parameters according to the brighter pixel.and is the parameters of the other one.We get From Eq.( 10), we get , substitute it into Eq.(9), we get ( Where , is a threshold to judge if approaches to zero.For three channels we will get three values by using Eq.( 15).That is  The final value of is .To determine whether a pixel is covered by a raindrop by values of is very tedious.Here we use another method to segment moving object from rain field.
From Eq.( 11) and Eq.( 12), we know .Consider , because , and we get , that is .So ( 17) Where , and .So when the pixel at is covered by a raindrop, we have .
However when the pixel belongs to a moving object, the term of does not converge to zero.The detecting function is (18)

If
dose not converge to zero, the pixel is of a moving object.We obtain the edges of moving object by using this detecting function.

Detection and removal of rain
As mentioned in section 1.2, we suppose those frames abstracted from video are aligned in advance.Using the detecting function, a frame is divided into two parts: static background and foreground.So the removal algorithm is twofold: Many methods are suitable for removing raindrops in static background.We use Kalman Filter to suppress intensity increase caused by raindrops.To remove raindrops in foreground, we align the foregrounds respectively, and then use the method of removing raindrops in background like the first step.

Results and Analyses
Our experiments use a threshold of 3 gray levels to detect the intensity change of pixels.Use a threshold of 5 gray to judge if the subtraction between channels such as approaches to zero.In calculating of detecting function, the order of is 1, the threshold of the detecting function is 5.
Fig. 2 (a) is an image of static scene from the video captured by Zhang and Li (2006, pp.461-464).(b) is removal result using Garg and Nayar's method (2007, pp.3-27).The result shows that the method of Garg and Nayar is not effective when rain is heavy because their removal algorithm only calculates two consecutive frames.Image using method of Zhang and Li (2006, pp.461-464) has better quality.But their method uses K-means clustering to calculate the background color which is effective in static scene.But this method will damage the image quality in some areas of dynamic scene by improperly regarded some moving object pixels as rain-affected pixels.(k) and (l) are local areas of (j).Our results show a better performance in this conditon.

CONCLUSIONS
By further studying the model of raindrops, we obtain a detecting function using chromatic property which is suitable for a general rain condition.It can segment rain-affected pixels from moving objects between two frames.Then removing approach is classified into two steps: to remove raindrops in background and to remove raindrops covered moving objects.The removal method makes the pixels at the same position of backgrounds in each frame similar and shows a better visual quality.Another advantage is that our method does not use any information about the shape, the velocity of raindrops, neither uses the value of camera's exposure time.Therefore it is effective in various conditions, such as heavy rain, light rain, rain in focus, rain out of focus and so on.When the objects are moving in the rain, this is (a) To remove raindrops in background.(b) To remove raindrops in foreground.

Fig. 3
Fig. 3 shows results in dynamic scenes.Fig. 3 (b) and (c) are result of Zhang's method.The improperly detected pixels obviously damage the visual quality of derained image.(d) is our result.(e) and (f) are local areas of (d).Compared with (c), our results accurately detected rain-affected area.(g) is another frame in this video.(j) is derained frame.(h) and (i) are local areas of (g).(k) and (l) are local areas of (j).Our results show a better performance in this conditon.