Development of a Mechatronic System for Plant Recognition and Intra-row Weeding With Machine Vision

In this paper, an electro-mechanical integrated weed management system was designed, constructed, and tested. This laboratory-scale solution integrated elements of machine vision, controls, and mechanical actuation to selectively remove weeds from within a crop row. The device was validated in a controlled environment using corn crops. Various crop conditions were considered to ensure the robustness of the design. Though some design aspects should be reworked for improved results, the device can effectively be used to facilitate small-scale research for automated weeding strategies.

by reducing labor needs (Bonadies & Gadsden, 2019). These systems have been used for a variety of purposes including: soil sampling, irrigation management, precision spraying, mechanical weeding, and crop harvesting (Taylor, Charlton, & Yúnez-Naude, 2012;Billingsley, Visala, & Dunn, 2008;Vidoni, Bietresato, Gasparetto, & Mazzetto, 2015). While research into unmanned systems in agriculture has been conducted over the last three decades, commercial products have been slowly adopted by farmers (Bonadies et al., 2017). However, due to recent epidemiological crises around the global, the use of autonomous systems for agriculture is expected to grow rapidly in the next few years as labor shortages are expected.
The goal of this study was to design a small-scale apparatus for testing research of automated IWM, with a focus on intra-row weeding. A review of the current state of automated intra-row weeding technology revealed design challenges that can be broken into four distinct categories: plant detection, position tracking, weeding tool design, and experimental testing. Modern weeding methods use machine vision to translate color and position from image pixels into information for additional computation. This has been explored in a variety of ways. For example, the authors in (Åstrand & Baerveldt, 2002) designed a system to identify and classify sugar beets and weeds by identifying the color and shape features of both. Alternatively, Green and Owen in (Green & Owen, 2011) employed blob detection on cabbage rows. By finding areas where pixel properties were consistent this method effectively separated crops from the image background. After plant detection, the position of the plant relative to the weeding tool must be determined and tracked. Studies such as (Åstrand & Baerveldt, 2002) and (Bontsema, van Asselt, Lempens, & van Straten, 1998) have shown implementing encoders mounted on wheels to track tool position relative to the detected plan have been successful. From the review of current methods, five design considerations were outlined for the construction of the testing apparatus. The machine should be able to: 1) detect the location of an approaching crop; 2) monitor the location of a crop relative to the weeding tool; 3) control the weeding tool; 4) actuate the weeding tool; and, 5) move to simulate field operation. This paper is organized as follows. The design process and experimental setup is described in Section 2. The experiment setup and results are stated and discussed in Section 4. The paper is then concluded and future recommendations are provided.

Materials and Methods
In this section, environmental conditions, system design, and the experimental design for the small-scale apparatus for testing IWM are considered.

Environmental Conditions
This study considered corn, a common cash crop, as the plant of interest. As such, the environmental conditions surrounding corn growth and development governed the design process and testing. Design of the IWM apparatus was based on a mixture of realistic and ideal environmental factors, which was deemed acceptable as the intended use is for continuous research. Five environmental factors were considered: growth parameters, crop spacing, soil conditions, temperature conditions, and light intensity. Growth parameters refer to the rate of emergence for the crop. Corn seeds are normally planted at a depth of 2-4 cm below the soil surface and follow the early growth stages defined in Table 1. Weed removal is critical at the V3/V4 stage of development in order to prevent nutrient competition of the more developed root system, making this stage the focus of the experiment (Ransom & Endres, 2014 Crop spacing refers to the amount of intra-row crop spacing. The selected spacing for this experiment was 61 cm, which is larger than the average corn spacing of 18 cm. Soil conditions refer to the composition and density of the soil, which range widely across Ontario. This experiment employed PGX Pro-Mix for the soil medium because it had ideal nutrient retention capabilities. The soil was loosely packed into the soil bins as opposed to compacted like a real farm. Temperature conditions refer to the impact of seasonal changes on the growing temperature of the crops. The normal growing season for corn in Ontario is May to October, during which the average amount of daylight is 13-15 hours (Ransom & Endres, 2014). Artificial temperature conditions were selected as 16 hours of daylight with daytime temperature of 25 °C and nighttime temperature of 20 °C, which were more ideal than realistic. Light intensity refers to the brightness level of the lights in the testing facility.
Tests were conducted at the Crop Science Building of the University of Guelph. This facility was equipped with a mix of LED and incandescent light bulbs that created a well-illuminated environment. Again, this was deemed an acceptable deviation from realistic conditions in which light intensity would fluctuate severely.

System Design
The IWM prototype was intended to present an electro-mechanical solution using traditional tillage combined with mechatronic techniques. The mechanical systems were designed first, and centered around the design of the weeding tool and a mechanical structure for the device. The weeding tool required a means of actuation and a design for its end effector. Rotary actuation in similar style to (Green & Owen, 2011) was employed because it permitted a tool to directly connect to a motor output shaft. A NEMA 34 stepper motor and driver were selected for use with this method of actuation because of their low cost and ease of control. The authors in (Green & Owen, 2011) also inspired the use of a disc-shaped tool for the end effector. This disc has a cut out "safe region", a circular area of clearance around the crop stem that is included to compensate for potential error between the camera sensor and the actual location of a plant. In this study, the design was modified to include two serrated discs, one disc on either side of a crop, for better ground coverage. The tillage depth of the end effector was constrained by the corn seed planting depth of 2-4 cm. Based on this, a tillage depth of 2 cm or less was selected. This would allow the device to remove weeds with shallow root systems but not compromise the crops' root systems. The diameter of the blades was determined through trial and error, and was selected as 15 cm with a 7.5 cm safe region.
Traditional soil bin tests use mobile carriages. Thus, a rolling trolley design was implemented for the mechanical structure of the prototype. The design was physically constrained by the dimensions of the soil bin and the space constraints of the Crop Science facility at the University of Guelph. Vibration reduction and tool modulation were also considered in the design. Minimizing vibration was critical to ensure image quality of the machine vision sensor. To combat this, a four wheel, "straddling" trolley with rubber-coated wheels was employed. Tool modulation was considered to allow for an adjustable tool height in case of variations in soil level. Height adjustment was implemented using a pinned sleeve design, where the lowest pin places the tool at a maximum of 2 cm below the soil. The highest pin places the tool well above the expected soil height for transportation purposes. The mechatronic system, as shown in Figure 1, centered on the selection, programming, and integration of a camera system and a position tracking system. A CMUcam5 Pixy was selected as the machine vision sensor. The Pixy is programmable with XCTU and Pixymon firmware. Vision-based weeding can take two main approaches: crop and weed detection, distinctly identifying both plants, or pure crop detection, distinguishing only crop and not-crop areas. Pure crop detection was employed on the Pixy, programmed to recognize a corn crop based on its size and location in an image. The above ground stages of growth were programmed as "target descriptions", or jas.ccsenet.org Journal of Agricultural Science Vol. 13, No. 3;2021 the ideal value, of corn crop size. Thus, the size criteria determined if a plant detected in the image was of adequate size to be considered corn based on these values. If not of target size, a plant was regarded instead as a weed. The location of a plant in an image also factored into crop detection. Modern precision planting allows planters to have a rough idea on the location of the next expected crop, allowing an expected region of interest to be programmed into the software. This permits only plants in this region to be recognized as crops and all plants outside the zone to be considered as weeds. An example of this is displayed in Figure 2 which labels a Corn Detection Region.

Figure 2. Screen partitioning of corn detection region
Position tracking was also used to actuate the weeding tool. A rotary encoder was selected to track the progress of the trolley as it approached a crop, thereby providing position information. Mounted to the wheels of the trolley, the encoder measured wheel rotations in relation to crop detection, allowing the system to determine trolley distance from crops. This information was passed to a control algorithm which maneuvered the end effector from an unactuated to actuated position based on the Pixy image capture, as shown in Figure 3. Once a crop was recognized, the algorithm counted encoder ticks until it reached a set value representing full actuation of the end effector. As the trolley passed over the crop, it would return to its unactuated position. To improve end effector actuation, the motor resolution, pulses per step, and delay were changed as necessary. Motor resolution represents the number of steps the motors take to rotate 360 degrees, pulses per step represents the amount of encoder units between each step of the motors, and the delay represents the amount of encoder units to wait between plant detection and the blades reaching the plant. Control was implemented using two Arduino Uno microcontrollers, one for obtaining the image and encoder inputs and the other to send pulse width modulation signals to the motor drivers. Figure 4 represents the flow diagram of the implemented control algorithm.

Experimental Design
To verify the functionality of the IWM device, tests were conducted with soil bins that measured 1 m in width at the Crop Science facility. The goal of each test would be to determine if a crop was safe: did it survive the automated weeding process. Based on the results, aspects of the control algorithm would be changed until performance was satisfactory. No weeding was conducted in the experiments, so the effectiveness of the design on weed removal could not be measured. Testing occurred by rolling the IWM assembly over the soil bed and measuring the reaction to the stimuli. When the trolley was above a corn crop, the position of the blades with respect to the plant stem was measured as shown in Figure 5a. If the distance from the plant stem to the back tip of the blade crescent was within the concave portion of the blade (the safe region), the plant was marked as safe and given the binary value of "Yes". If the plant stem was outside this distance, it was marked as unsafe and given the binary value of "No". Based on a minimum of three trials of binary data, the motor resolution, pulses per step, and delay parameters of the control algorithm were changed to improve performance. Thus, the data collected are hit-miss values for plant survival and controller parameters for satisfactory trials. The IWM device was tested in four scenarios: Benchmark, Best Case, Spacing, and Bounds. The Benchmark test was used to ensure proper functionality and calibrate initial control parameters. Red cup lids, displayed in Figure  5b, were used in place of corn crops for this test because they had uniform shape and size, making them ideal for calibration purposes. The Best Case scenario represented an ideal crop harvesting situation, with corn crops at V3/V4 growth and intra-row spacing of 61 cm. The Spacing and Bounds cases each represented deviations from the Best Case. In the Spacing tests, intra-row spacing was varied to imitate cases where there is increased or decreased distance between crops. The increased spacing, deemed Spacing-Increased, represents when a crop does not sprout or dies, resulting in larger spaces than expected in crop rows. Alternatively, Spacing-Decreased scenarios imitate when crops are planted closer together to increase crop yield. Intra-row spacing for Spacing-Increased remained 61 cm but plants were removed from the row to simulate increased spacing. Spacing for Spacing-Decreased scenario was 30.5 cm. The Bounds test was used to determine the maximum and minimum plant size detectable by the machine vision algorithm. This is important for determining the impact of plant size on weeding actuation, as there are cases where harvested crops may be at varying stages of development. To conduct this test, plants at different growth stages were placed in the soil bed.

Results and Discussion
The results of each scenario are presented in Tables 3-6 and the final apparatus control parameters for each scenario are found in Table 7. There is no hit-miss data recorded for the Spacing-Increased scenario, to be explained shortly. As seen, the Best Case controller specifications required minimal modification from the Benchmark calibration. Across the five trials, all specimens survived the weeding maneuver, highlighting the apparatus can function in an "ideal" scenario. The results of the Spacing and Bounds scenarios indicate the device's potential robustness. No data was recorded for the Spacing-Increased scenario because the significant increase in plant distance resulted in the motors not activating between plants, which is expected. However, the Spacing-Decreased case emphasized the limitations of the actuators, shown by the decrease in success rate. In trials with "missed" plants, it was observed that the motors had not finished their actuation cycle from the previous plant prior to reaching the subsequent crop. As such, controller parameters need to be further adjusted to ensure the blades return to an unactuated position so as not to kill the next encountered crop. The Bounds scenario revealed that the minimum size plant recognizable by the machine vision algorithm is subjective. As the Pixy can detect small resolution objects, it is up to the user to determine the lowest number of pixels for which a crop can be recognized. Here that size was set to 7.5 cm tall after multiple Bounds trials. This scenario also revealed the maximum size plant recognizable to be 28 cm. Past this height, the machine vision algorithm correctly disregarded larger plants as viable crops because they failed to fit into the detection region, indicated by the misses on Plant 5 across all trials. The detection region can also be subjective because its design was based on corn growth stages, indicating that different upper bounds can be determined based on the desired acceptable growth stage. However, significantly increasing the crop detection region could undermine its purpose by introducing more crops into the field of vision or increasing processing time of the algorithm.    Results indicate the IWM apparatus functions sufficiently recognizing and not tilling crops. However, the testing conditions undeniably assisted in its effectiveness. Though testing conditions were intended to mimic reality, in practice they differed significantly regarding crop spacing, soil density, and light exposure. Crop spacing was much greater than average crop conditions at 61 cm row spacing as compared to the 18 cm average. This would significantly affect the image sensor's detection capabilities and the activation of the motors. The soil density was loose instead of compacted in order to allow the blades to move easier through the soil. Typical soil mediums are denser, which causes not only additional resistance to the blades but also soil to clump on tools, potentially damaging root systems of crops. Thus, the balance of soil resistance and root safety would need to be considered in blade design and actuator selection. Finally, the light exposure to the soil beds was constant in intensity and direction. Light was provided directly overhead test bins, eliminating the presence of shadows from testing. Varying light exposures would need to be considered to evaluate their impact on the image detection software, which would undoubtedly deteriorate in the presence of shadows.

Conclusions
A laboratory-scale, integrated weed management device was designed, built, and tested on intra-row weeding scenarios for a corn plant. Using an electro-mechanical approach, a control algorithm was designed and integrated with vision sensors to mechanically actuate stepper motors via encoder readings. Experiments verified the effectiveness of the prototype, which successfully identified crops and avoided uprooting them under a variety of conditions. Future work includes testing under increasingly rigorous scenarios that mimic real world harvesting and environmental conditions.