Hardware-Software Complex for Laser Scanning in Color Sorting Machines

The paper concerns the problem of grain mixture analysis based on processing images synthesized during line-by-line scanning of each object in color sorting systems. The paper presents a hardware-software complex for sorting objects in the real time. The hardware part of the complex consists of two blocks: light source and device for reception and processing of images. The light source is a laser, passed through optical fiber and linearly expanded across the entire width of the photoseparator’s chute. Linear laser scan produces significant intensity of illumination. It is sufficient for working on the transmission of radiation through objects. The software part of the complex also consists of two blocks: the thresholding algorithm and an automated program for finding the optimal parameters of sorting on the basis of that algorithm. The algorithm calculates the number of connected defective pixels with arbitrary shape. Automated program works on the basis of pre-formed images of the objects of two classes: good and defective. As a result, the program displays in tabular form the most optimal sorting parameters. The program shows the dependence of the loss of good product from the missouts of the defective objects. The customer gets a clear choice of the most suitable sorting results. This complex was tested for sorting of unshelled rice seeds via transmission. It is shown that the complex allows to effectively detect hidden seed defects: red pigmentation, immaturity, fungal diseases, and others.


Introduction
The technologies of sorting elements of grain mixture (EGM) used in today's color sorting machines (Satake et al., 1998) are based on the following principle: constant material flow is sorted basing on the analysis of one, two or three spectra of reflected electromagnetic radiation.The objects are illuminated either by broadband light from a single source, or by several light sources of the respective spectral range -usually LED or other type lamps.To register the data, the CCD or CMOS image sensors can be used.They allow to quickly identify defective objects and immediately remove them from the material flow, usually by means of an air-valve system.
In such sorting systems, thresholding algorithms (see, for the example (Gonzalez and Woods, 2002;William, 2001;Sahoo et al., 1988;Lee et al., 1990;Cseke and Fasekas, 1990)) are usually sufficient.As an accept/reject criteria these algorithms often use such properties of the EGM as the reflected radiation intensity, the number of defective pixels, lateral dimensions, etc.Such systems, however, cannot be applied in any case.It is, therefore, necessary to develop new hardware systems and elaborate optimal sorting algorithm parameters.
As for the hardware, new approaches suggest using laser illumination systems.As lasers produce narrow beams of coherent light of a single wavelength, it is possible to analyse the objects not only by the amount of light scattered from the surface, but also by the transmitted light after short-term exposures, and by their luminescence spectra (Algazinov et al., 2013;Algazinov et al., 2014).
Using fibre optics to calculate various source laser light emission will further add to the benefits of laser systems, as it allows for flexible programme control over the laser illumination spectra.
Another way to improve the EGM sorting system is to automate the process of choosing the optimal sorting parameters.The choice is based on the analysis of a large number of object images, synthesised by the sorting equipment.
Thus, the aim of this paper is to describe a hardware-software complex for color sorting, basing on laser illumination, which can be used to solve the problems that the existing sorting equipment is not able to cope with.During the test the complex was applied to sort shelled rice grains.

Materials and Methods
The initial registration of data as well as the image processing and analysis of the EGM was performed using the constructed hardware-software complex with laser illumination system, integrated into an original color sorting machine F5.1 (Chuiko et al., 2009;Babishov et al., 2013).The flow chart of the complex is shown in Fig. 1.
Fig. 1 shows that the complex includes a video module (VM) connected to a personal computer (PC).The VM main element is a line-scan image sensor combined with a computing unit based on a dual-core processor.The image sensor used in the complex is a Hamamatsu CMOS linear image sensor of high sensitivity S11108 (Hamamatsu, 2013).Its photosensitive area consisting of 2048 pixels, each with a pixel size of 14 × 14 μm.When all the 2048 pixels are active, maximum line rate of the sensor is 4.67 kHz.
The output data is then digitalized by the analog-to-digital converter (ADC) and passes to a special signal processing processor -dual-core Analog Devices ADSP-BF561 Blackfin processor (AD, 2009).The integrated development environment Analog Devices VisualDSP++ 5.0 with Update 9.1 (AD, 2007) was chosen for software development of VM with BF561 processor.
The software for the dual-core processor functions in real time and includes the following elements: connection to the PC, linear image sensor control, linear sensor image analysis and processing, connection with the other modules of the sorting machine.
During the sorting process, the VM registers transmitted laser light line-by-line, converts received video data into digital format, processes the lines (frames) of the image using special algorithms and determines whether the objects are pure or should be removed from the material flow.
The connection interface between the PC and the VM is the industrial interface of RS485 standard.The communication protocol is Modbus RTU protocol, with the operator's computer as the master device and the video module as the slave device.
The described complex creates a two-dimensional image of each object of the material flow.Linear monochrome sensor is used to form the lines (frames) of every single object coming in.Processor BF561 accumulates in SDRAM memory all images.After accumulating enough images of various object classes, they are transferred to PC, in which images can be further processed by the software described below.
Illumination by a narrow laser light beam was provided using cylindrical lenses (Fig. 2).The lens LJ1695L2 and LK1982L2 (Thorlabs) was chosen for this purpose.The choice of these lenses was determined by photoseparator geometry and required parameters of the laser line.In this case, the formed laser line had length 400 mm and width 2 mm.Laser diodes ATC-S1-0-FS-665-5-F200 and АТС-С5000-200-AMF-808-5-F200 (ZAO Semiconductor Devices) were chosen as the illumination sources.The first one has wavelength 650 nm with optical power 1 Watt, and the second one is 808 nm with optical power 4 Watts.These wavelengths were the most suitable for our special task (shelled rice sorting).The radiation summation from different laser sources is one of the main advantages of the proposed lighting system.We use optical fiber from Thorlabs with diameter 200 μm and numerical aperture 0.22.The radiation from each laser source using focusing optics was introduced into common fiber (Shulgin et al., 2014).The benefits of such a system are obvious, as it provides flexible computer control over the laser illumination spectra.The system also allows to place lasers and the optical circuit in any place.
To apply laser light linear sweep effectively, the resulting intensity received by the linear sensor, was distributed evenly along the width of the chute with the help of special software.This allows to apply uniform sorting parameters for the whole width of the chute.Apart from the programmed distribution of the laser light sweep, it is also necessary to set correctly the background light to detect the objects edges effectively.
To analyse EGM and sort them into classes according to their biological features, various spectrum analysis methods were applied -reflectance spectroscopy, transmitted light spectroscopy, luminescence spectroscopy (Schmidt, 2007).Reflectance, luminescence and transmitted light spectrum, were registered using experimental equipment based upon the fiber optic spectrometer USB4000-VIS-NIR (Ocean Optics).The detailed description of the equipment is given in the following papers (Sarycheva et al., 2012;Algazinov et al., 2013;Algazinov et al., 2014).Spectral analysis for color sorting machine application forms two object classes: pure and defective.
Pure and defective objects recognition algorithm is an essential element of the VM software.Thresholding algorithms seem to meet all the requirements (Gonzalez and Woods, 2002;Sauvola and Pietikainen, 2000;Bichsel, 1998).
After an image frame is processed, a so called binary image is created, where pixels can be denoted only 0 or 1 depending on the pixel type -pure and defective.Further processing is performed upon the binary image.The next stage of the image frame processing within the sorting process includes finding defective areas (formed by the defective pixels) and determining their size.The material is thus sorted according to the size of the defective area on the object.
We have tested the most uniform variant of the sorting algorithm, when the defected area can be any shape.We can also call this algorithm the number of connected defective pixels.If the number of such pixels exceeds the set limit, then the object is rejected and removed from the flow.
The algorithm is thus based on two sorting criteria: the intensity threshold and the number of connected defective pixels.These criteria are then sent using Modbus protocol from the PC to the VM processor.
Before the sorting, it is, of course, necessary to adjust the sorting machine parameters to fit the objectives.In our work, we use automated search for each algorithm parameters based on the large number of object images of two classes: pure and defective.
Automated search for best sorting parameters presupposes determining best intensity threshold values and the number of defective pixels as a dependency of lost pure objects and accepted defective objects.This task is performed separately by the PC basing on the image analysis of a large number of objects of each class.We should also point out, that image frames of pure and defective objects are stored separately.

Results
Lets assume that there is a model sample of pure objects' images N and defective objects' images M. The search algorithm works as follows.First, the threshold T and the number of connected pixels C is set.All the images are then binarized according to the threshold T. With the given parameters {T,C} let the pure object images N 1 be recognized as defective, and the defective object images M 1 be classified as pure.Ratio Q=N 1 /N is the pure objects loss, and the ratio P=M 1 /M is the defective objects missout.Thus, Q and P are the functions of {T,C}, i.e.

Q=Q(T,C), P=P(T,C).
Apart from the object images the PC receives also the information about the background level of the captured objects.Let T bkgr be the background level of the sample objects.If the defective objects are brighter (lighter) than pure ones, the threshold T varies in the range [T bkgr , T max ]; if the defective objects are darker, [0, T bkgr ], where T max = 255 for 8-bit grayscale images.
For each T from the interval above, missouts Q(T,C), and losses P(T,C) are calculated as functions of the number of connected defective pixels C. For each object image for the given threshold T the maximum number of connected pixels is determined with the brightness interval (T,T max ] (for light defects) or [0,T) (for dark defects).The values are added to an array and sorted in descending order.Pure and defective objects are analysed separately.The defective pixels array for pure objects is then presented as {C 1 , C 2 , …, C N }, with C i > C i+1 .Similar array is formed for defective objects {C 1 , C 2 , …, C M }.Then for any set number of defective pixels C the number of object images from the initial sample that will be recognized as defective, can be easily determined.Indeed, let C lay within the interval C k > C > C k+1 , then with the set {T,C} k object images (either pure or defective) will be recognized as defective.Thus, there is no need to analyse the whole sample every time C changes.3(a).The higher the number of defective pixels C with the given threshold T the less losses Q and the more missouts P there will be.The losses and missouts' graphs will have an overlap region.If the graphs do not overlap, then, with the set threshold T the objects are completely parted in two classes.In this case, values {T,C} appear, where C is some number of defective pixels from the graphs' partitioning region, and the analysis is thus completed.Complete partitioning, however, is hardly ever possible with any threshold T, and losses and missouts' graphs always have an overlap region (Fig. 3(a)).
Lets examine, with the given threshold T, the overlap region of losses' curve Q and missouts' curve P. In the overlap region there is a parametric dependence P(Q), because for any number of defective pixels C form the overlap region P(C), Q(C) are determined.Subsequently applying all C values from the overlap region of the losses and missouts' curves (Fig. 3(a)), it is possible to form a dependency P(Q).Typical dependencies P(Q) for various thresholds T are shown in Figure 3(b).Function P(Q) will always monotonely decrease with growing Q, because with constant T, the higher are the losses Q of pure objects, the lower the missouts of defective objects

P.
This results in a set of dependencies P T (Q) for each threshold value T.
Each curve P T (Q) crosses the X-axis at some value Q=Q max (T) (Fig. 3(b)).Then the minimum value from the set Q max (T) is determined, defined as Q is the minimum level of pure product losses at which the number of defective objects' missouts P equals 0. The only way to reduce the number of losses to even lower values than min max Q is to introduce certain permissible level of defective objects' missouts P. To optimize the process, it is necessary for any given value of losses Q i < min max Q to find the minimal value of missouts P min (Q i ).
To determine the minimum missouts values P min (Q i ) we need to find the minimum value from functions P T (Q) with constant Q=Q i and all possible values of the threshold T. Thus, dependencies P min (Q i ), T opt (Q i ) are obtained, where T opt (Q i ) is the value of index T, i.e. the value of the sorting threshold, at which P T (Q i ) has minimal value In actual process, the range [0, min max Q ] is discretized with a step-size ΔQ.It is usually enough to select ΔQ=0.01, which fits the step-size of 1% loss.
From the loss function Q(T,C) (Fig. 3(a)) with T=T opt and a given value Q i the number of defective pixels C opt (Q i ) is found.Therefore, {T opt (Q i ),C opt (Q i )} are the threshold values and the number of connected defective pixels, which yield the losses Q i and missouts P min (Q i ).
For the sorting process it is vital to determine not the amount of defective objects missouts P min at the given value of losses Q i , but the quantity of dockage in the sorted grain mixture as compared to the initial grain mixture at the given value of losses Q i .Let us show, how this difference can be determined by the found parameters Assume that the initial EGM batch consists of N r defective and N a pure objects.Lets now introduce parameter γ=N r /N a .Then the quantity of dockage in the initial batch will be β=N r /(N r +N a )=γ/(1+γ).
After the sorting, with losses Q i and missouts P i , the sorted grain mixture will contain , the quantity of dockage will be As we can see, the less is m with constant [gamma], the less is the quantity of dockage [lambda] in the sorted material.To determine how much the quantity of dockage will reduce after the sorting, we need to know not only the value of m, but the value of [gamma] as well, i.e. the quantity of dockage at the initial stage.
As the quantity of dockage in the initial material is usually small, the value of [gamma] is not big.Then    .As losses P i and missouts Q i are usually under 0.5, we may assume that m≤1, an then mγ is also small, and . Thus, when the quantity of dockage in the raw material is small (≤10%) index shows, how does this quantity change after the sorting.
By the set of values Q i , P min (Q i ) we determine m(Q i )=P min /(1-Q i ) and get the following set of optimal sorting parameters: Q i , m(Q i ), T opt (Q i ), C opt (Q i ).These parameters are shown in a table with the step-size ΔQ i =0.01, starting with Q i =0.Each of the following set of parameters is put into the table provided that the growth of losses goes together with better sorting quality, i.e. if m(Q i ) reduces as compared to the previous value.From the table we can choose the best parameters as to the number of losses and the exit product purity depending on the certain objective.
Objects under examination: pure objects -shelled white-grain rice, provided by State Scientific Institution All-Russian Rice Research Institute, Krasnodar, Russia.Defective objects -grains with hidden structural defects: contaminated grains, unripe grains, cracked grains, chalky grains, and red rice grains.Pure objects sometimes look completely similar to the defective ones (Fig. 4) This can be explained by the fact, that, for example, a contaminated grain is damaged from the inside, and the defect does not show on the hull.The reflectance spectra analysis has demonstrated that it is very difficult to separate pure rice grains within the visible spectrum, even if more than one spectrum is analysed.
The transmittance spectra analyses yielded much better results.The transmitted light intensity for the defective grains in the visible spectrum is over 20% lower than for the pure ones.The best contrast ratio was registered, when the wavelength was 660 nm.
The spectral analysis data, therefore, demonstrates that shelled rice grain sorting can be successfully performed only by analysing the transmittance spectrum.The hardware-software complex described above allows to do this using laser light sources.As a result of the complex operation, several hundred images of pure and defective objects were obtained.The transmittance spectra analysis algorithm also allows to eliminate grains with very small defective areas (for example, slightly contaminated grains).
The results of the automated search for the best sorting parameters algorithm described above are shown in Table 1.The first column of Table 1 contains the percentage of pure grain loss compared to the initial state of the batch.
The second column shows dockage reduction index, i.e. dependence of the resulting and initial quantity of dockage.The third and the fourth columns contain the determined best sorting parameters of the threshold and the number of defective connected pixels.We should note, that when the transmittance method is used, defective objects will be darker than the pure ones.Hence, the pixels with brightness lower than the threshold values will be recognised as defective.
The Table demonstrates the flexibility of the automated search algorithm, as the consumer can choose any sorting parameters he finds appropriate.For example, if the consumer considers the 5% loss after the first sorting appropriate, then the expected percentage of the rejected defective objects is 96%, which demonstrates high efficiency of the sorting process.However, most of the lost objects can be regained during re-sorting of the rejected material.The final loss percentage will thus be minimum.

Discussion
Spectral analysis showed that unshelled rice seeds with hidden defects can not be sorting by reflecting method with high accuracy.The best result of separation can be achieved by registering the radiation passed through the seeds, since in this case there is maximum contrast between the good seeds and bad.So, the color sorting machines which used only reflecting method in visible and near infra-red spectral ranges are ineffective for solving such problem.
At the other point, some works (Grundas et al., 1999;Arkhipov, 2003;Hill et al., 1996) demonstrate that seeds with hidden defects can be recognition by using x-ray radiography.However, at present time these x-ray systems have some restrictions.First of all, such systems can not perform sorting in real-time scale.They can only reject all seeds in samples.Moreover, x-ray devices are complicated and respectively expensive.Also, the special care must be apply during working with x-ray devices.Our complex has no such disadvantages.Indeed, sorting in real-time scale gives us possibility do not reject all seeds, but reject only defective one.Therefore, we can significantly improve initial quality of seed batch.Also, laser sources do not require such special conditions for health safety as x-ray systems.
It should be noted that the using of laser sources for separation tasks is not new.The interesting schemes of laser sorting are proposed in some works (Ruymen et al., 2005;Calcoen et. al., 2010;Adams et. al., 2009).In these works, the laser beam is deployed to the entire width of the scanning area by the use of a special fast-rotating mirror.Reflected from the object radiation again falls on the same mirror, and then it is directed by different optical devices in the receiving detector.Obviously, high performance sorting requires that the laser beam ran across the scan area with a frequency of several thousand times per second, and respectively, high rotational speeds of the mirror is needed.In addition, the photodetectors are unlikely to be able to provide high resolution for sorting small objects (1-5 mm).
The above disadvantages are partially removed in the scheme described in the work (Dirix et al., 2010).In that scheme, the rotating mirror is only intended to sweep the laser beam and does not redirect the reflected light to the photodetectors.Photodetectors are located in a separate block.This fact allows to simplify the optical system of the device.In addition, CCD image sensors can be used instead of simple photodetectors.As a result, the resolution can significantly improve.However, this scheme also requires a high speed rotation of the mirror prism.
Most of the above mentioned solutions which used laser illumination can be implemented only in conveyor separators.At the other hand, there are many color sorting machines in the market, which have inclined chutes for product feeding and can not use such laser systems, because they are too bulky.Almost all world leaders (Buhler Sortex, Satake, ASM and other) have color sorting machines with inclined tray which use only lamps or LEDs as lighting.We propose hardware-software complex with fiber laser system, which can be organically fit in color sorting machine with inclined chutes.In our experiments, we use color sorting machine from Voronezhselmash (www.vselmash.ru).
The radiation summation from different laser sources in one optical fiber is one of the main advantages of the proposed lighting system, which allows to sort in a few wavelengths.The angle between input laser beam and fiber is an important factor of the proposed lighting system.In work (Shulgin et al., 2014), it is shown that this angle must be non-zero.In this case, there are only ring mods from optical fiber.At the same time, axial mods are not excited and, accordingly, speckle structure with low spatial frequencies is absent.This fact ensures the stability in time of the output light along the width of the scan line.
Work (Upadhyaya, 2007) describes the issue of the presence of red rice as a weed for white rice in North America.The same problem is present in the Krasnodarsky region of Russia (Vasilieva et al., 2013).At the other hand, in work (Kawube et al., 2005), chemical and manual sorting method for rice seeds is described in order to increase the yields, because there is a lot of seeds with diseases.All these facts show that our system may be perspective and demanded for the solution of such problems.
Thresholding algorithms are considered as one of the easiest and popular in real-time systems (see, for example, Palumbo et al., 1986;Sahoo et al., 1988;Lee et al., 1990;Glasbey, 1993;Trier et al., 1995;Sezgin et al., 2004).These works consider techniques with fixed threshold.At the same time, there are more complicated methods (Parker, 1991;Yang et al., 1994;Shen et al., 1997;Sauvola et al., 2000;Yang et al., 2000;Bradley et al., 2007), which is called adaptive thresholding.The adaptive thresholding technique is more effective in situation when the illumination varies spatially.Such situations are likely in color sorting machines.In our present work, we use fixed thresholding method, which shows good results.The adaptive thresholding techniques are not required in our work, because any spatial irregularity of illumination was eliminated by the software correction coefficients.These coefficients remain unchanged during sorting because illumination is stable in time.
There are much more effective algorithms for object recognition: neuro-network algorithms (Gonzalez and Woods, 2002;Principe et al., 2000;Osowski, 2000), support vector machine algorithms (Scholkopf, 1998;Burges, 1998).These methods works as algorithms with training.The complexity of these algorithms imposes certain conditions on their application in machine vision systems with real-time response.Indeed, these algorithms require appropriate hardware with much more high-performance computing resources.At the same time, our work represents special method for automated searching of the optimal parameters for certain thresholding algorithm.The flexibility of this method is that the user can choose a suitable quality of the final product as a function of losses of good objects.From a practical point of view, our approach can be considered as algorithm with training for two classes of objects (good and defective).

Conclusion
A new hardware-software complex with fibre optic laser illumination system was suggested.Its main feature is the laser light sweep with a stream of high density.Such system allows to sort objects according to their structural properties illuminated by transmitted light.
The software for the automated search for best sorting algorithm parameters was created.The said algorithm consists of two parameters: threshold and the number of connected defective pixels.The search results are presented in form of a specialised table, using which the consumer can consider various sorting options and choose the most suitable.
Despite of successful work, the proposed complex has some limitations.It is known that the intensity of the transmitted light considerably depends on the thickness of the object.Therefore, the complex will have some difficulty in sorting product, in which there is considerable variation in thickness of objects.Indeed, if accepted objects vary in thickness, so that the intensity of the transmitted through them radiation overlaps with the intensity of the defective object, there will be difficulties in recognition.We will receive a lot of defect missouts or large losses of good product.Successful sorting of the unshelled rice is due to the absence of the great variation in thickness.
Thus, the modernization of the complex is needed for taking into account the thickness of the objects.Modernization must be for the hardware part of the complex and the software one.Upgrading the hardware in the first place is to add another laser source.This is possible by entering into a single optical fiber laser sources from two different spectral ranges.Working with two sources will allow to exclude the influence of object's thickness.Also, two video-modules (with filters) are needed to introduce in the complex for receiving signals from both sources.
Software must also be modernized, because we will need to calculate the relationship of the radiation transmittances of the two spectral ranges for different types of objects (good and bad).In this regard, the sorting algorithm and automated program for finding of optimal sorting parameters will be upgraded.
These improvements are the subject of our further research.

Figure 1 .
Figure 1.Flow chart of the hardware-software complex

Figure 2 .
Figure 2. Optical lenses for laser light linear sweeping

Table 1 .
Best sorting parameters for shelled rice