Solder Joints Detection Method Based on Surface Recovery

Machine vision has been widely used in various industrial productions. However, the study for solder joints detection is not enough. This paper presents a solder joints detection method based on surface recovery. For a single gray-scale image, using shape-from-shading (SFS) technology, the surface of the solder joints is recovered. According to the shape distribution, the quality of solder joints is discriminated. In order to improve the accuracy of recovery for real images, hybrid illumination model is introduced and a reflection-component estimation method based on simulated annealing algorithm is designed. Then recovery process of the algorithm is improved. Compared to other detection methods based on two-dimensional images, this method provides more information about explicit physical meaning and make detailed quantitative analysis for solder joints easier. At the same time, even for defect that is difficult to detect, this method also has important research value.


Introduction
In industrial production, inspection is regarded as a key step.Especially for automatic optical inspection (AOI) in electronics manufacturing, machine vision plays an increasingly important role in the product assurance.Compared to other visual detection applications, solder joints detection is more difficult and more challenging for its complex process.Current detection methods are all directly or indirectly based on the solder joint's surface analysis.
Indirect analysis of solder joint's surface morphology mainly focuses on both research directions.One is the illumination device and image acquisition environment designing; the other is related to the detection algorithm.At present, research work in this area is relatively mature, and published articles are greater.Sanderson and Nayar in 1990 proposed a method to detect the defect of solder joints based on Extended Gaussian Image (referred to as EGI) (Shree K. Nayar, Arthur C. Sanderson, Lee E. Weiss, David A. Simon, 1990).This method used the special structural light source to find the curvature distribution characteristics of each solder joints to classify the different types of defects.Although the direction and size error caused by circuit board location can be avoided, but still can not prevent the long time lighting requirement with the EGI bad overall performance.Since then, Taiwan's National Tsinghua University, Zi-chao Chen achieved the detection of the location of surface-mount components by optimized the lighting way (Zichao Chen, 2004); Shao-nong Qiu do some research about the tiered ring lighting mode, and two-dimensional image features extraction and selection were discussed (Shaonong Qiu, 2003).Paper (Debao Peng, 2001) proposed a method by using image-based similarity comparison to achieve precise position of components and solder volume detection.Paper (Kim T-H, Cho T-H, Moon YS, Park SH, 1999)used three-tired structural color light source to collect the images of solder joints under different colors, after the region segmented, feature extracted and trained by BP neural network to judge the quality of the solder joints.Recently, some scholars introduced wavelet transform into the solder joints detection and achieved some results.T.Taniguchi et al presented a one-dimensional wavelet transform for detecting defects of solder joints (Z.Ibrahim, S.A.R. Al-Attas, Osamu Ono and M.M. Mokji, 2004); Z.Ibrahim proposed a solder joints location and defect detection method based on wavelet transform(Z.Ibrahim, S.A.R. Al-Attas, Osamu Ono and M.M. Mokji, 2002); In 2006,Giuseppe Acciani et al proposed two-dimensional discrete wavelet transform to extract features of solder joints and used artificial neural networks to detect the defects and obtained a higher efficiency of solder joint inspection(Kuk Won KO, Young Jun Roh, Hyung Suck Cho and Hyung Cheol Kim, 2000).Above methods are all indirect detection methods, and its essence is to use two-dimensional image of the statistical results to reflect the morphological structure of three-dimensional solder joints, the sample data is dependent on, once the size, type and direction of solder joint changed, then must be re-trained using solder joint images.In product quality high requirement's areas, usually further asked for marking the defect level, to discover the reasons leading to defects, allowing timely feedback to the front of the production control unit, to guide the optimal adjustment of the production process, which is two-dimensional detection methods can not complete in a detection task.
Directed surface analysis techniques based on the surface recovery of solder joints, that is no longer restricted to the analysis of the image itself as the detection of an object, but to use various means to obtain three-dimensional solder joint's surface data for test.Compared to the indirect detection methods, not only ambiguity of physical meaning brought by statistical model can be avoided and a more detailed quantitative analysis for the solder joint's quality can be achieved, but also for some special defect that can not be detected by indirect method, a possible solution may be found.At present, research in this area is still limited; some of the existing research is based on laser measuring technology, such as: Chun-Yue Huang of Guilin Institute of Electronic Engineering used laser triangulation to obtain solder joint's surface point cloud, and then reconstructed the solder joint's surface by the triangulation and obtained a key slice by a selected tangent Plane.Finally, shape parameters extraction carried out on key slice, including wetting angle, solder joints height and area (Chunyue Huang, 2007).Toshifumi Hond used confocal laser technology to measure surface morphology of solder joints and then extracted features, including the morphological characteristics as well as the position relationship between the solder joints and pads (Toshifumi Hond et al., 1996).Though these methods can obtain the solder joint's surface morphology, but it is not difficult to find that demands on the environment configuration are all very high, limit the practical application of industrial production.On the other side, system efficiency is so low that it can not meet the needs of the actual manufacturing and testing.The method proposed in this paper does not require anything almost on the system environment configuration, as long as have access to the image of the solder joints.This method is based on shape-from-shading technology.Solder joint's surface morphology can be recovered by the single gray-scale image.Because the solder joint's surface consisting with small surfaces, and existing more serious specular reflection, there will be some minor changes in different perspective, showing an unstable state.At the same time, shape-from-shading technology is not mature enough.Noise suppression and surface restoration accuracy are lower.So based on the above issues, we modified the shape-from-shading method in the hybrid reflection framework and derived and improved the corresponding calculation process.

Recovery of Solder Joint's Surface
In this section we discuss the method of recovery of solder joint's surface.In order to obtain the 3D shape, one or more two-dimensional images are required.Based on single image we can use texture or shading to restore the shape.Based on multiple images, the three-dimensional reconstruction methods include stereo vision method, dynamic image sequence method and photometric stereo learning methods.Considering the system operating cost and the characteristics of the surface mounting technology, this paper focuses on the research of shape from shading for the surface recovery of solder joints.

Shape-from-shading Algorithm Theory
Shading plays an important role in perception of surface shape.Researchers in human vision have attempted to understand and simulate the mechanisms by which our eyes and brains actually use the shading information to recover the 3D shapes.Artists have long exploited lighting and shading to convey vivid illusions of depth in paintings (B.Liu and J.T. Todd, 2004).To solve the SFS problem, it is important to study how the images are formed.A simple model of image formation is the Lambertian model, in which the gray level at a pixel in the image depends on the light source direction and the surface normal.In SFS, given a gray level image, the goal is to recover the surface shape at each pixel in the image.However, real images do not always keep to the Lambertian model (Hossein Ragheb, Edwin R. Hancock, 2003).Even if we assume that following Lambertian reflectance and known light source direction, the intensity of one pixel can be described as a function of surface shape and light source direction, the problem is still not simple.Because the surface shape can be described in terms of the surface normal, then we must solve a linear equation with three unknowns, and if the surface shape is described in terms of the surface gradient, we have a non-linear equation with two unknowns.Therefore, finding an unique solution for SFS is a very difficult task, it requires additional constraints (P.L. Worthington, E.R. Hancock, 1999).A great number of papers about the SFS problem have appeared since the publications of the original study by Horn and Brooks(B.K.P. Horn, M.J. Brooks, 1989).However, most of the algorithms are all based on the assumption of Lambertian that limited its applications.For real images, different reflectance components always exist simultaneously.So using a more complex reflectance model would be a better way to solve this problem.But it could increase the computation cost greatly; moreover a number of algorithms will not be tractable easily.However the linear approximation algorithm of Tsai and Shah (P.S. Tsai and M. Shah., 1994) has obvious advantages that have the same computational complexity and good controllability under various reflectance models.For these reasons, TS algorithm is considered firstly.In this paper, two different reflectance components are taken including the Lambertian and specularity.Although Tsai and Shah have paid attention to the situation of specularity and designed a scheme for that, they deal with the images alternatively Lambertian or pure specularity(P.S. Tsai and M. Shah., 1994).The combination of these two reflectance components was not considered.Follow this paper (P.S. Tsai and M. Shah., 1994), no papers focused on this issue.This paper will modify this algorithm by taking hybrid reflectance model and deduce the iteration formula in new frame.

Shape-from-shading Based on Hybrid Model
Classical shape-from-shading problem is based on the Lambertian model assumption.So many researchers limited the analytical process in this framework, even if for real images.But in the real world, intensity of each pixel in images can not be determined by the simple reflectance model.Even for synthetic images, calculation error in simulated process may lead to deviation.So a better way to improve the accuracy of shape recovery is designing a new method based on a more complex model instead of Lambertian.For this purpose, various reflectance components can be taken into account, for example: diffuse, specular, inter reflection and environment light.All these components are the same important in certain situation.In this study, only diffuse and specular reflectance components are considered.Ideal diffuse reflection surface can be represented by Lambertian reflectance model and the ideal specular surface can be represented by Phong reflectance model.The hybrid reflectance model is a linear combination of ideal diffuse reflection model and ideal specular reflection model.Then the mathematical form of the hybrid reflectance model is as follows: Assume that the vector of light source direction is ( , , )

S s s s
x y z .Camera direction vector is ( , , )

C c c c
x y z .The middle vertical line between light source and camera is defined as the vector ( , , ) s p s q s x i j y i j z R p q w i j i j i j p q s s s i j i j x y z k m p m q m x i j y i j z w i j p q m m m i j i j x y z is the exponential factor of specular reflection.
The reflectance equation can be rewritten as: Here j i I , is the intensity at a fixed point ) , ( j i in image.A linear approximation (Taylor series expansion up through the first order terms) of the function f about a given depth map Then for , the depth map at the n-th iteration can be solved directly: , can be deduced as following: p q p q pp qq d n s s s s f Z i j dZ p q p q p q p q i j s s s s k p q p q p q pp qq h h h h h h k p q p q p q p q p q p q h h h h h h

Reflectance probability
In this study, reflectance types can be determined by computing the probability of the surface point When we got the reflectance probability, in the equation ( 4) can be substituted by it.In experiments we normalized the reflectance probability by follow equation:

Recovery of Solder Joint's Surface
The PCB image consists of many surface mounting components, text labels and printed circuits, so the gray level in different region is complex and confused.Assume the image input has 256 gray levels, which is a set defined as below: , m n is the range of the pixel position.
, x y G is the gray level of current pixel.In virtue of the lightness varying between the component and the surface of the PCB, we can distinguish the solder joint, component body and else elements.The samples obtained by CCD are shown as Fig. 2. In the shooting and transaction process, images are contaminated by noise.These noises information will affect the final recovery.So, to overcome the problem a fitly filter must be used.Here, Gaussian Smoothing is selected.The Gaussian Smoothing is normally used to reduce noise in an image, somewhat like the mean filter.However, it often does a better job than the mean filter of preserving useful detail in the image.Fig. 3 lists the recovered surface shapes of solder joint base on the proposed method.
Figure 2. In the top row we show the images with excess solder; in the middle row we show the images with acceptable solder; in the bottom row we show the images with insufficient solder.Figure 3. Correspond to the Fig. 2, in the top row we show the recovered shape of the excess solder's surface; in the middle row we show the recovered shape of the acceptable solder's surface; in the bottom row we show the recovered shape of the insufficient solder's surface

Detection Method Based on Solder Joint's Surface Shape
In order to analyze the solder joint's surface morphology, we carried out the slice processing on the depth data.Studies(P.L. Worthington, E.R. Hancock, 1999) have shown that the inclination and elevation can be used to define the solder joint's surface shape at any point, in which, the inclination angle plays an extremely important role in determining the basic form of solder joint.Because the elevation only has less impact on the basic morphology, it can be said that the distribution of the inclination may reflect the major quality of solder joint's problems.It is the important theoretical basis of solder joints visual inspection.To this end, here we choose the slice direction as vertical, horizontal slice does not do analysis and processing.Slice position chooses the highest point of the recovered surface, because this can maintain the Morphological integrity greatly.The contour of the solder joint's surface shows as the black curve in Fig. 4. In order to better analyze the shape of the curve, curve fitting is carried out.Here, choose Gaussian function as the curve fitting objective functions; Fig. 4 is the result of the Gaussian function fitted.After obtained the vertical slice curve of the solder joint's surface, we calculate three important wetting angles closely related to the solder joint's quality.Here l θ is the transition angle between the pads to the solder; v θ is the transition angle of the solder; e θ is the transition angle between the solder to the pins of the Components.Specific geometric definition of the three transition angles are shown as Fig. 5.
For the three different types of solder joints: normal, insufficient and excess soldering conditions, the statistical results shown as Tab. 1.
We can see from Tab. 1, three transition angles calculated by this algorithm for the calculation can be used as a judgment of the solder volume, especially the transition angles e θ and v θ have a stable distribution, can be categorized by an experiential threshold.While the transition angle l θ is somewhat less and is not stable enough, so in the actual testing process e θ and v θ can be considered as an important criterion.
intensity diversity between the computed intensity value and the intensity of the input image.That is to say, known the surface normal computed intensity value under Lambertian model and specular model.According to this idea compute the reflectance probability of surface point ) , ( j i based on simulated annealing.The algorithm shows as Fig. 1.