An Improved Guidance Image Based Method to Remove Rain and Snow in a Single Image

Rain and snow bring poor visibility at outdoor vision systems. The common used image processing methods may be not suitable for a degraded image. In this paper, a guidance image method is proposed to remove rain and snow in a single image. To removal rain and snow only using one image, a guidance image is derived from the imaging model of a raindrop or a snowflake when it is passing through an element on the CCD of the camera. Since only using this guidance image may lose some detailed information, in this paper, a refined guidance image is proposed. This refined guidance image has similar contour with the un-degraded image and also maintains the detailed information which may be lost at the guidance image. Then a removal procedure is given by the use of the refined guidance image. Some comparison results are made between different methods using the guidance image and the refined guidance image. The refined guidance image can be used to get a better removal result. Our results show that this proposed method has both good performance in rain removal and snow removal.


Introduction
In videos and images taken from an outdoor condition, bad weather like rain and snow annoys human viewers, brings difficulty to image processing and decreases the performance of vision algorithms.Both rain and snow bring complex intensity changes.A region covered by a falling down snowflake or raindrop seems brighter than its original background.But it is hard to detect rain and snow only using the property of intensity changes.Because there exist so many objects which have similar linear edges with rain streaks and snow streaks.Most methods use multi-frames to estimate rain streaks.Due to the different speed between rain and snow, most of rain removal methods are not suitable for snow removal.Also removal methods from videos are not suitable for single image.However, in some cases, there is important application value to remove the rain or snow from only one outdoor image is used to get more information.In this paper, we mainly concern to remove rain and snow in a single image.

Previous Work
The existed methods to detect rain from videos could be classified into two types: (1) Detection using properties of pixel-based intensity changes; (2) Detection using the linear appearance property of raindrops.Gary and Nayar (2004;2007) used intensity changes between consecutive frames to detect raindrops, then compute binary rain field to distinguish rain regions from other moving objects.Zhang (2006) proposed a detection method using chromatic properties of each pixel.The two methods are pixel-based and need multi-frames to get a statistical result.They may fail and get a wrong detection when some exception exists such as some region with gray color.Barnum (2007) found the intensity changes caused by raindrops could be modeled using a blurred Gaussian.The appearance of rain or snow streaks looks like a linear streak caused by a moving blurred Gaussian.So they use the frequency-based method to removal those blurred Gaussian.This method could remove both rain and snow.But it also needs multi-frames to compute the information of streaks, such as the direction.Fu (2011) proposed rain removal method for single frame using morphological component analysis.The single image affected by raindrops could be decomposed by sparse coding and dictionary leaning.Since streaks caused by snow have more complicate appearances than raindrops.This method is not suitable for snow removal.

Our Work
This paper concerns on rain removal and snow removal in a single image.This method is also useful to video restoration.Because in videos, some scene is taken with a moving camera, or includes some moving objects, single image removal method could simplify the removal procedure and avoid the alignment between frames.
First, we analyze the imaging model of rain and snow formation to find a guidance image.Second, we propose a refined guidance image such that this novel image could keep detailed information and at the same time remove the linear edges caused by rain and snow.Thereby we use filtering method to remove rain and snow by this guidance image, for example, using guided filter (He, 2010).The experimental results show that our algorithm has good visibility and suitable for most images.

Guidance Image Based Model
Generally the appearance of rain streaks is similar with linear streaks.But the detection is complicate since lots of objects have a similar appearance with raindrops.It is difficult to distinguish raindrops from others by only using a linear determinant condition.The speed of a snowflake is slower than a raindrop.Its falling direction is changeable due to the influence of wind.So the linear rain detection methods can not be used to detect snow.Our methods focus on finding a way to keep the edges of objects and remove the edges caused by snow or rain.Bilateral filter also has this property.But bilateral filter could not directly be used to remove rain and snow because it can distinguish edges of rain streaks or snow streaks from edges of different objects.Bilateral filter could remove raindrops and at the same time some detailed information will be eliminated.In this paper, we use a guidance image to decide which edge should be remained and which region should be smoothed.Therefore bilateral filter or guided filter can be used to remove rain and snow.

Guidance Image
Guidance image is an image only keeping the information of contour of background and without remaining the edges caused by rain or snow.The guidance image could be obtained using imaging model of raindrops and snowflakes.
When a falling raindrop or a snowflake is captured by a camera, the intensity is a linear combination of irradiance of raindrops or snowflakes and the irradiance of background.Their intensity values can be both expressed as followed Where rs I is the intensity value of a pixel of an outdoor image captured in a rain day or a snow day, rs E is the time-averaged irradiance due to a raindrop or a snowflake, b E is the time-averaged irradiance of background, T is the exposure time and during the time  a raindrop or a snowflake is passing through a element on the CCD.
We (1 )( ) , the time duration  is far less than T .The value 1   .Equation ( 5) is only determined by background I b .Therefore if we use I f as a guidance image, I f is only related with the restored image and is not affected by rain or snow.So I f could be used as a guidance image.We will show how to remove rain and snow using I f and guided filter in the next section.

Removal Method Using Guided Filter
The guidance image I f is a gray image which has similar contour with the original image rs I .Also edges caused by rain or snow are not included in I f .We use I f as the guidance image p , and use each channel of rs I as the input image I, respectively.Obviously image I is a single channel image (gray image).We define J as the output chromatic image, and each channel of J is J corresponding to I .The output image J has two properties: (1) Image J has similar intensity values with rs I ; (2) If a region in I f has no edges, the corresponding region in J has no edges.Therefore J has similar contour with rs I and without degradation caused by raindrops or snowflakes.We could use guided filter to realize this procedure.The Equation of the output image J at each channel is as followed: Where i is the location of the pixel to calculate, k  is the region of the kernel, k is the location in the kernel,  is the number of pixels in k  , k  and 2 k  are the mean and variance of I respectively in k  , and k p is mean value of p in k  .To compute the output value in a local window, we use the average value of all the windows containing pixel i .So the image J is each channel of the restored image J .
Although image J could remove most edges caused by rain or snow, also it removes some detailed information such as tiny edges with similar color, because in Equation ( 5), some edges with similar color will be eliminated in f I .In the next part, we propose a refine guidance image to solve this problem.

Refined Guidance Image
If some detained information in a region is eliminated, there must be some edges which should not be smoothed.
Let x be the location of an edge, ( ) I x be its intensity value, and ( 1) I x  be the intensity values which ( 1) x  is near x but is not on the edge.The relationship of these two value satisfies ( ) ( 1) there exists a edge at the position x .The output ( ) J x and ( 1) J x  satisfies ( ) ( 1) J x J x   , and ( ) J x is an average value of a local region due to guided filter method.Therefore the subtraction image between ( ) J x and ( ) I x shows the wrong eliminated information.At the same time since the image ( ) J x approaches the removal image, the intensity of a rain-affected region and a snow-affected region is much brighter than the background.The region of a rain streak or snow streak at the subtraction image has a negative intensity value.This subtraction image has more detailed information and also it is not affected by rain or snow.But this image has information of local edges, but it is not sensitive to the color contour.We can use an average value of this subtraction image and the guidance image f I in Section 2.2 as a refined guidance image.The refined guidance image maintains both the detailed information and the contour of background.So if J is the output chromatic image using guidance image f I in the Section 2.2, let g J be the gray image of rs  J I , (here rs  J I only includes the positive values).The refined guidance image is Using the refined guidance image could get a better restored image which has more detailed information.In the next part, we describe the process to remove rain and snow in a single image using this refined guidance image.

Rain Removal and Snow Removal
Although using the refined guidance image g I as the guidance image, and using each channel of rs I as the input image, we get a restored image according to Equation (6).But since there is a big difference between the value of the refined guidance and the original image, the pixels which have similar value at a region of the result image have an unsmooth appearance.So we use two steps to get a better result: (1) Get a new image which has similar value with rs I , and also this guidance image has similar edges with g I ; (2) Use this new image as the input image to get a restored output image.
The removal procedure is as followed: (1) Use f I as the guidance image, and use each channel of rs I as a input image, we could get the output image J using Equation ( 6).
(2) Use Equation ( 9) to get the refined guidance image g I .
Use the refined guidance image g I as the guidance image, and use each channel of rs I as a input image, we could get the output image ref J as a reference image of each channel.
Use each channel of the original image rs I as the guidance image, and use its corresponding reference image at step (3) as the input image, we can get the resorted image of background.The result is shown in Figure 1.

Experimental Results
Figure 1 and Figure 2 are some experimental results of rain removal and snow removal.Figure 1 shows the removal procedure proposed in this paper.Image (a) is a rain-affected image.Image (b) is the guidance image I f in Equation (5).Image (c) is the refined guidance image I g in Equation ( 9).Image (c) has more information than image (b).Image (d) is the output image J using guided filter and I f is used as the guidance image and each channel of rs I is used as the input image.Image (e) is the result directly using guided filter and use the refined guidance image g I as a guidance image, use each channel of rs I as the input image.Image (f) is the removal result using the three steps in Section 3. The results of zoom-in region in (g-i) show that the procedure in Part 3 has better performance and more detailed information.
Figure 2 shows two results of rain removal and snow removal respectively.The top images are two image, one is taken in the rain and the other is taken in the snow.The middle images are two results using guidance image I f .And the bottom images are removal results using the refined guidance image and our methods proposed in Section 3. The results show that the refined guidance image is better than the guidance image I f in rain removal and snow removal.

Conclusions
In this paper, we propose a method to remove rain and snow using a single image.The removal procedure includes calculating a refined guidance image, and getting an output image by use of guided filter in two times.Unlike the existed methods which are only suitable for one type of weather condition, our method is suitable for both rain removal and snow removal.The restored results show that our method could reduce the degradation caused by dynamic weather and also could maintain some detailed information of local regions.
define b I as the background intensity, E I as the intensity of a stationary snowflake or raindrop keeping still at the time T .Because rs E is the time-averaged irradiance during  , E use a variable  to express the ratio between  and T , we use RGB space to compute Equation (2), we will get three equations at RGB space respectively.Generally rain or snow makes background look bright, and a stationary raindrop or snowflake taken at a very short exposure time looks like a white spot.Therefore if C indicates one coordinate of the RGB space and rs C I  is the maximum value at RGB space of rs I , b C I  must be the maximum value at RGB space of b I .The similar relation is true when we compute the minimum value of each vector at RGB space.The equation could be expressed

Figure 1 .Figure 2 .
Figure 1.The removal procedure (a) A rain affected image; (b) The guidance image in Equation (5); (c) The refined guidance image in Equation (9); (d) The result using the guidance image; (e) The result directly using the refined guidance image in one time; (f) The result using the refined guidance image in Section 3; (g) A zoom-in region of image (d); (h) A zoom-in region of image (e); (i) A zoom-in region of image (f)