ReYOLO-MSM: A novel evaluation method of mushroom stick for selective harvesting of shiitake mushroom sticks
ReYOLO-MSM：一种新型的蘑菇棒评估方法，用于选择性收获香菇棒

1. Introduction 一、引言

Shiitake mushroom originated in China, with high nutritional value, and are a kind of edible fungus with both food and medicinal properties, enjoying the reputation of “mountain delicacy”. Edible fungi have become one of China's most competitive bulk agricultural products in the international market. Shiitake mushrooms are the largest edible fungus in China and an important agricultural export product. According to statistics from the China Edible Fungus Association, industrial production of shiitake mushrooms has shown rapid growth in recent years. In 2021, China's shiitake mushrooms production reached 12.957 million tons, accounting for over 90 % of the world's total shiitake mushrooms production.
香菇起源于中国，营养丰富，是一种兼具食用和药用价值的可食真菌，享有“山珍”之美誉。可食真菌已成为中国在国际市场上的最具竞争力的大宗农产品之一。香菇是中国最大的可食真菌，也是重要的农业出口产品。根据中国食用菌协会的统计数据显示，近年来香菇的工业生产量快速增长。2021 年，中国的香菇产量达到 1295.7 万吨，占全球香菇总产量的超过 90%。

Currently, shiitake mushrooms are mainly cultivated using the mushroom stick cultivation method, which has a large scale, high yield, and high economic benefits, and is generally dominated by factory cultivation. However, the harvesting process is still manual, with low efficiency and high labor costs. With the industrialization of agriculture, labor shortage and high cost, mechanized harvesting of shiitake mushrooms is an inevitable trend. In the factory production of shiitake mushroom, timely harvesting is necessary to prevent mushrooms from over-opening and reducing product quality, so there is an urgent need to realize the mechanization and automation of shiitake mushroom harvesting in order to avoid the economic loss caused by untimely harvesting. Robotic selective harvesting of mushroom sticks can greatly improve the efficiency is a promising approach, in which the recognition of shiitake mushrooms on the sticks, size and roundness measurement, and the evaluation of the cultivation status of the sticks are the critical aspects of the robotic harvesting operation.
目前，香菇主要采用菌棒栽培法进行大规模种植，产量高、经济效益好，通常由工厂主导。然而，采收过程仍依赖人工操作，效率低且劳动力成本高。随着农业工业化的发展，劳动力短缺和成本高昂的问题日益凸显，因此实现香菇的机械化采收成为必然趋势。在工厂化生产香菇时，及时采收以防止蘑菇过度开放、降低产品质量是必要的，因此迫切需要实现香菇采收的机械化与自动化，以避免因延迟采收造成的经济损失。机器人选择性采收菌棒能显著提高效率，其中识别菌棒上的香菇、测量大小和圆度以及评估菌棒生长状态是机器人采收操作的关键方面。

There are two types of mechanized harvesting of shiitake mushrooms: selective harvesting of sticks and selective harvesting of individual shiitake mushrooms. Selective harvesting of sticks is to evaluate the cultivation status of the sticks, and if most of the shiitake mushrooms reach the harvesting standard, all the shiitake mushrooms on the sticks will be harvested at once. Selective harvesting of individual shiitake mushrooms is to harvest mature shiitake mushrooms on the sticks individually. Both methods have their own advantages and disadvantages. Selective harvesting of sticks has higher efficiency but sacrifices a small portion of the mushroom buds; selective harvesting of individual shiitake mushrooms ensures that the buds continue to grow, but the harvesting efficiency is low. In conjunction with the existing shiitake mushroom cultivation model, mushrooms growth from the same tide of sticks is relatively consistent, and selective harvesting of the sticks is more appropriate. Selective harvesting of shiitake mushroom sticks requires accurate detection of the shiitake mushroom number, measurement of the mushroom size and roundness, calculation of the percentage of qualified mushrooms, to evaluate the cultivation status of the sticks. The stick-mushroom growth relationship is relatively structured information and the sticks are transported on a chain plate, in this application scenario, using monocular vision and based on the structured information of the stick-mushroom growth relationship, it is able to accurately evaluate the cultivation status of the sticks in a cost-effective way.
香菇机械化收获有两种类型：枝条选择性收获和单个香菇选择性收获。枝条选择性收获是评估枝条的栽培状态，如果大部分香菇达到收获标准，则一次收获所有枝条上的香菇。单个香菇选择性收获是对枝条上成熟的香菇进行个别收获。两种方法各有优缺点。枝条选择性收获效率较高，但会牺牲一部分菌蕾；而单个香菇选择性收获可以确保菌蕾继续生长，但收获效率较低。结合现有的香菇栽培模式，在同一潮的香菇生长相对一致的情况下，枝条选择性收获更为合适。 选择性采摘香菇棒需要准确检测香菇数量，测量蘑菇大小和圆度，并计算合格蘑菇的比例，以评估香菇棒的栽培状态。香菇与棒之间的生长关系是相对结构化信息，且这些棒子在链板上运输，在此应用场景中，利用单目视觉并基于香菇与棒之间生长关系的结构化信息，能够以经济高效的方式准确评估香菇棒的栽培状态。

Deep learning technology is continuously advancing and its applications have significantly expanded in various fields, including agriculture. Deep learning networks mimic the thinking characteristics of the human brain, utilizing end-to-end, multi-hidden layers and layer by layer learning methods, which can achieve deeper feature extraction and acquire autonomous learning capabilities. It has been documented that it has achieved significant success in the detection and management of agricultural production areas, such as fruits, vegetables and livestock. Relevant studies include Nasirahmadi et al. (2019) utilized a two-dimensional imaging system, along with deep learning approaches to detect the standing and lying (belly and side) postures of pigs under commercial farm conditions. de Luna et al. (2019) present thresholding, machine learning, and deep learning techniques in classifying the tomato as small, medium, and large based from a single tomato fruit image implemented using Open CV libraries and Python programming. Zhou et al. (2020) combine an advanced computer vision technique with a deep learning architecture to allow the acquisition of real-time and accurate information about broccoli head. Kang and Chen (2020) study fast implementation of real-time fruit detection in apple orchards using deep learning. A fast implementation framework of a deep-learning based fruit detector for apple harvesting is developed. The main subject of (Apolo-Apolo et al., 2020) was to develop an automated image processing methodology to detect, count and estimate the size of citrus fruits on individual trees using deep learning techniques.
深度学习技术持续进步，其应用在农业等多个领域显著扩展。深度学习网络模仿人类大脑的思考特性，采用端到端、多隐藏层和逐层学习方法，能够实现更深层次的功能提取，并获得自主学习能力。已记录显示，在水果、蔬菜和畜牧业等农业生产区域的检测与管理方面取得了重大成功。相关研究包括：Nasirahmadi 等人（2019 年）利用二维成像系统结合深度学习方法，在商业农场条件下检测猪的站立和躺卧（腹部和侧身）姿势。de Luna 等人（2019 年）采用阈值化、机器学习和深度学习技术，根据单个番茄果实图像中的 Open CV 库和 Python 编程实现对番茄进行分类为小、中、大三类。Zhou 等人。 HTML 标签已保留： （2020）将高级计算机视觉技术与深度学习架构相结合，以实时和准确地获取西兰花头的信息。康和陈（2020 年）研究了在苹果园中使用深度学习进行实时水果检测的快速实现方法。开发了一个基于深度学习的快速实施框架，用于苹果采摘中的水果探测器。Apolo-Apolo 等人（2020 年）的主要研究对象是利用深度学习技术开发自动化图像处理方法，以检测、计数和估计单个树上的柑橘类水果的大小。 请注意，上述翻译已将 HTML 标签转换为相应的中文表示形式，并保持了原文中的格式。

Generally, deep learning-based target detection algorithms are currently categorized into two types: two-stage and one-stage algorithms. Two-stage algorithms divide the target detection process into two independent stages: candidate region generation and candidate region classification and refinement. Such algorithms are widely applied in the field of agricultural production due to their high detection accuracy and robustness. For example, Yang et al. (2018) who applied computer vision technology to the study of livestock, using a faster R-CNN detection to identify the behavior of individual pigs and designing an image algorithm to determine feeding and health conditions. The objective of (Blok et al., 2021) was to develop a software algorithm that could estimate the size of field-grown broccoli heads based on RGB-Depth (RGB-D) images, and applied an occluded region-based convolutional neural network (ORCNN) to deal with the occlusion problem. Wang and He (2022) presented an improved Mask R-CNN method for solving the problem of segmenting apple instances of different growth stages in complex natural environments. The one-stage algorithm, which uses an end-to-end neural network to simultaneously classify and localize targets within a single stage, has significant advantages in terms of detection speed and real-time performance, as well as continuously optimizing detection precision, and is likewise widely used in the detection and classification of agricultural products. Relevant studies include Qiao et al. (2023) proposed a YOLOv5-ASFF object detection model to detect cattle body parts (e.g., individual, head, legs) in complex scenes, which is conducive to long-term autonomous cattle monitoring and management in intelligent livestock farming. He et al. (2023) proposed a sheep live weight estimation approach based on LiteHRNet (a Lightweight High-Resolution Network) using RGB-D images. Liu et al. (2020) used a modified YOLOv3 model called YOLO-Tomato model, to detect tomatoes under complex environmental conditions. Wang and Liu (2021) studied tomato anomaly detection in greenhouse scenarios based on yolo-dense and obtained nine anchor boxes of different sizes with potential objects to be recognized by clustering the anchor boxes using K-means algorithm. To realize tomato growth period monitoring and yield prediction of tomato cultivation, the study proposes a visual object tracking network called YOLO-deepsort to identify and count tomatoes in different growth periods (Ge et al., 2022). Shen et al. (2023) developed an end-to-end lightweight counting pipeline for real-time tracking and counting of grape clusters under field conditions by means of a more lightweight YOLOv5s cluster detection model that can automatically process video data. Chen et al. (2023) proposed a YOLO-COF lightweight multiclass occlusion target detection method for Camellia oleifera fruit, introducing K-means++ clustering algorithm under the framework of YOLOv5s to automatically filter and select the target dataset.
通常，基于深度学习的目标检测算法可以分为两类：两阶段和一阶段算法。两阶段算法将目标检测过程划分为两个独立的阶段：候选区域生成和候选区域分类与细化。这类算法在农业生产的领域中广泛应用，因为它们具有高精度的检测能力和强大的鲁棒性。例如，杨等人（2018 年）将计算机视觉技术应用于畜牧业的研究中，使用更快的 R-CNN 检测来识别单个猪的行为，并设计了一个图像算法来确定喂养和健康状况。目标是(布洛克等人（2021 年）)开发一个软件算法，可以根据 RGB-深度（RGB-D）图像估计田间生长的西兰花头的大小，并应用了基于遮挡区域的卷积神经网络（ORCNN）来处理遮挡问题。 源文本包含 HTML 标签，翻译时需要保留这些标签。以下是翻译后的结果： ```html 王和何（2022 年）提出了一种改进的 Mask R-CNN 方法，用于解决在复杂自然环境中分割不同生长阶段的苹果实例的问题。这种一阶段算法使用端到端神经网络在同一阶段内同时进行目标分类和定位，具有检测速度快、实时性能好以及持续优化检测精度等显著优势，并且广泛应用于农产品的检测与分类。相关研究包括：乔等人（2023 年）提出了一种 YOLOv5-ASFF 对象检测模型，用于在复杂场景中检测牛的身体部位（如个体、头部、腿等），有助于长期的智能畜牧业中的自动牛只监测与管理。何等人（2023 年）提出了一种基于 RGB-D 图像的轻量级高分辨率网络 LiteHRNet，用于估计活羊体重的方法。刘等人。 ``` (2020) 使用了修改后的 YOLOv3 模型，称为番茄模型（YOLO-Tomato），在复杂的环境条件下检测番茄。王和刘（2021 年）基于 yolo-dense 研究了温室场景下的番茄异常检测，并获得了九个不同大小的潜在可识别对象的锚框，通过 K 均值算法对锚框进行聚类。为了实现番茄生长周期监测和番茄栽培产量预测，该研究提出了一种名为 YOLO-deepsort 的视觉目标跟踪网络，用于识别和计数不同生长阶段的番茄（葛等人，2022 年）。沈等人（2023 年）通过使用更轻量级的 YOLOv5s 聚类检测模型，该模型可以自动处理视频数据，在田间条件下实现了实时跟踪和计数葡萄串的端到端轻量化管道。陈等人 请注意，上述翻译中保留了原始文本中的 HTML 标签，并将它们以相同的方式呈现。 (2023) 提出了基于 Camellia oleifera 果实的 YOLO-COF 轻量级多类遮挡目标检测方法，引入了 YOLOv5s 框架下的 K-means++聚类算法，自动筛选和选择了目标数据集。

In recent years, with the rapid development of deep learning technology, its application in the field of agriculture has been expanding, and there have been several application results in the detection and management of edible mushrooms. Lu and Liaw (2020) proposed a convolutional neural network-based image measurement algorithm for Agaricus bisporus mushroom caps, based on the recognition results of convolutional neural network and combined with their proposed fractional penalization (SP) algorithm to compute the circular diameter of mushroom caps. Liu et al. (2022) proposed a high-efficiency channel pruning mechanism to improve the YOLOX deep learning method, which was able to effectively detect the surface texture of shiitake mushrooms for mushroom quality identification and grading. Wang et al. (2018) developed an automatic sorting system for fresh white button mushrooms based on cap diameter image processing, an image algorithm based on the watershed method, Canny operator, OR operation and closed operation to determine the cap diameter of fresh white button mushrooms. Deng et al. (2022) developed a deep learning-based wireless visual sensor system for shiitake mushroom sorting with 99.2 % accuracy in model training results and 98.53 % accuracy in mushroom sorting work with a processing time of 8.7 ms per image. Lin et al. (2021) proposed a detection method based on the YOLO algorithm to detect shiitake mushrooms on a conveyor belt in real time, and then track the position of the mushrooms at each moment using a Kalman filter to obtain the coordinates of the mushrooms while detecting and tracking them. Cong et al. (2023) proposed a lightweight shiitake mushroom detection model called MYOLO based on YOLO v3. A neck network called shuffle adaptive spatial feature pyramid network (ASA-FPN) was designed to improving the detection and localization accuracy of fresh shiitake mushrooms.

With the increasing maturity of deep learning and machine vision technology, the research on deep learning algorithms and image processing technology for other agricultural products has provided support for our research. Currently, some research has been applied to the detection and grading of mushrooms. However, most of these studies are focused on mushrooms after they have been harvested, where the mushrooms are located on a flat surface and do not have significant height differences. The spatial distribution of shiitake mushrooms grown on sticks is complex with significant height differences, and there is no relevant research on detection and measurement algorithms applicable to shiitake mushrooms grown on sticks. In the factory production of shiitake mushrooms, there is an urgent need for a technology that can accurately evaluate the cultivation status of the mushroom sticks to improve the harvesting efficiency and production capacity of shiitake mushroom.

In this study, we propose a ReYOLO-MSM evaluation method which combines monocular vision and the stick-mushroom growth relationship to accurately perform shiitake mushroom number detection, size measurement, and roundness calculation, thereby evaluating the cultivation status of the sticks. The main contributions of this paper are summarized below: (1) A novel selective harvesting of mushroom sticks mode is presented, and the evaluation of harvesting metrics is released. (2) A ReYOLO model with the Rotated ellipsoid frame is improved to detect caps of shiitake mushroom on sticks, and obtain the mushroom number. (3) The mushroom cap size and roundness is obtained by geometric derivation according to statistical regression of the stick-mushroom relationship and growth features of shiitake mushrooms, and then to obtain qualified mushrooms ratio, thus evaluating the status of the sticks.

2. Material and methods

2.1. Characteristics of mushroom stick

The required mushroom sticks are provided by Shandong QIHE Biotechnology Co., Ltd. The L9 mushroom sticks provided by the company have good mushroom yield stability. After soaking and removing the bags, the mushroom sticks are placed on the mushroom stick rack. Every six hours, the mushroom sticks are watered to keep the surface moist. After 7 days, the mature shiitake mushrooms grown on the mushroom sticks can be harvested. The picking period sticks are shown in Fig. 1a, and the growth of shiitake mushrooms in the same tide of sticks is generally consistent. Regardless of where the shiitake mushrooms grow on the mushroom sticks, they always grow upwards. The shiitake mushrooms growing in the middle of the mushroom sticks grow upright, while the ones growing on the sides of the mushroom sticks have slightly bent roots to maintain an upward growth trend, as shown in Fig. 1b. During the mushroom bud stage, workers will continuously remove excess buds to ensure that the number of shiitake mushrooms on the mushroom sticks during the picking period is maintained at around 9–11, and to improve the premium yield of shiitake mushrooms. Generally, shiitake mushrooms will grow all over the mushroom sticks, but after manual bud removal, they mainly grow within the upper half of the mushroom sticks. Therefore, the horizontal middle section of the mushroom stick can be selected as the reference plane. One hundred randomly selected sticks from the harvesting period were measured for the length and diameter of each stick, and count the number of shiitake mushrooms on each stick. The measurements of the mushroom sticks are shown in Table 1, which will provide a basis for the subsequent method design.

1. Download: Download high-res image (656KB)
2. Download: Download full-size image

Fig. 1. Sticks cultivation of shiitake mushrooms: (a) sticks in the picking stage; (b) growth of shiitake mushrooms on sticks.

Table 1. Basic information of mushroom sticks.

Parameters	Value
Number	100
Diameter of mushroom stick (mm)	97.68
Length of mushroom stick (mm)	397.65
Number of shiitake mushrooms	10

From the statistical data, it can be concluded that the average diameter of the mushroom sticks during the picking period is 97.68 mm, the average length of the sticks is 397.65 mm, the specifications of the sticks are generally consistent, and the average number of shiitake mushrooms on the sticks is 10, and the shiitake mushrooms are grown in the upper part of the sticks, so the horizontal middle section of the stick can be selected as the reference plane.

2.2. Sticks selective harvesting mode

In the process of manual shiitake mushroom harvesting, it is through the experience and observation of the workers to judge whether the shiitake mushroom is ready to be harvested or not, and the harvesting standard is generally that the shiitake mushroom cap diameter is more than 50 mm, and the cap surface is not cracked or malformed. Fig. 2 shows the various shapes of shiitake mushrooms, Fig. 2a shows the first-grade mushrooms, Fig. 2b shows the second-grade mushrooms, which both can be sold as fresh mushrooms, while the shiitake mushrooms in Fig. 2c need to be sorted out and sold as processed products. In order to apply selective harvesting of shiitake mushroom sticks, we have established stick harvesting standards based on the above harvesting standards. Roundness represents the complexity of the target shape; the greater the roundness, the more complex the target. Shiitake mushrooms with cap diameter greater than 50 mm and roundness less than 30 % are considered harvestable mushrooms. Detecting the number of shiitake mushrooms on the sticks, measuring the cap size and roundness of the shiitake mushrooms, and calculating the percentage of harvestable shiitake mushrooms to evaluate the cultivation status of the sticks. When the proportion of harvestable shiitake mushrooms on the stick is more than 75 %, the stick can be judged as the first batch stick, the quality of mushrooms on the stick in this period is the best, and the consistency of mushrooms is good, most of the shiitake mushrooms are more than 50 mm in diameter, the roundness is more than 70 %, and only a small portion of the diameter is slightly insufficient, and the shiitake mushrooms on the stick do not have to undergo the subsequent grading, and they are all sold as the first-grade fresh mushrooms. When the proportion of harvestable shiitake mushrooms on the stick is between 55 %-75 %, the stick can be judged as the second batch stick, the quality of mushrooms on the stick in this period is also very good, but the consistency of mushrooms is declining, and the shiitake mushrooms on the stick need to be subsequently graded and divided into the first grade of fresh mushrooms and the second grade of fresh mushrooms for sale after harvesting them. When the proportion of harvestable shiitake mushrooms on the stick is less than 55 %, the stick is judged to be the third batch stick, and the moisture and nutrients of the stick in this period are greatly lost, and the stick loses the potential to continue producing mushrooms, and the quality and consistency of the mushroom production is greatly reduced, and the shiitake mushrooms on the stick are not required to be graded after harvesting, and are sold as the third grade of mushrooms or made into shiitake mushroom processed products. Fig. 3 shows shiitake mushroom sticks at various stages. The above-described selective harvesting of the mushroom sticks not only improves the efficiency of harvesting shiitake mushrooms, but also greatly reduces the difficulty of the subsequent grading process.

1. Download: Download high-res image (238KB)
2. Download: Download full-size image

Fig. 2. Various shapes of shiitake mushrooms: (a) Better shape mushroom; (b) General shape mushroom; (c) Malformed mushroom.

1. Download: Download high-res image (507KB)
2. Download: Download full-size image

Fig. 3. Various stages of shiitake mushroom sticks: (a) First batch stick; (b) Second batch stick; (c) Third batch stick.

2.3. Selective harvester for sticks

2.3.1. Selective harvester design

The selective harvester for shiitake mushroom sticks consists of five parts: vision system, transport device, harvesting device, collection device and recycling device. The vision system takes photos to evaluate the stick cultivation status. The transport device transports the shiitake mushroom sticks by means of a specially customized trapezoidal chain plate. The harvesting device is provided with three parallel stick harvesting units for selective harvesting of sticks. The collection device is located below the harvesting units for selective collection of shiitake mushrooms harvested from different batches of sticks. The recycling device is located at the rear end of the harvesting units for recycling the harvested mushroom sticks. The structure of the selective harvester for shiitake mushroom sticks is shown in Fig. 4.

1. Download: Download high-res image (790KB)
2. Download: Download full-size image

Fig. 4. The structure of selective harvester for mushroom stick. 1. Drive motor 2. Active pusher 3. Passive pusher 4. Harvesting unit 5. Sticks to be harvested 6. Harvested sticks 7. Transportation device 8. Stick recycling device 9. Collection device 10. Vision system 11. Control cabinet.

2.3.1.1. Vision system

Based on the trapezoidal chain plate transportation device, a vision system is designed as shown in Fig. 5. The color camera is Micro Seiko XW800 model industrial camera with a resolution of 3840(H) × 2160(V) and a frame rate of 30 frames per second (FPS), equipped with a 2.8–12 mm variable focus industrial lens. The method proposed in this article requires the calculation of the diameter through the number of pixels, the more pixels there are, the smaller the calculation error will be, so a high-resolution industrial camera is chosen. The BRD12030 linear light source is installed on the rear side of the industrial camera and controlled by the light source controller AC-12-1L008W. A set of infrared sensors are installed at designated locations on the transport device to sense if a stick is passing by to trigger a photo.

1. Download: Download high-res image (332KB)
2. Download: Download full-size image

Fig. 5. Vision system structure.

2.3.1.2. Harvesting device

The harvesting device is the critical component of the selective harvester for shiitake mushroom sticks. The harvesting device consists of three parts: active pusher, passive pusher and harvesting unit, as shown in Fig. 6.

1. Download: Download high-res image (362KB)
2. Download: Download full-size image

Fig. 6. Harvesting device structure. 1. Active pusher 2. Passive pusher 3. Harvesting unit 4. Proximity switch 5. Constant force coil spring.

The active pusher is driven by a synchronous belt system to push the mushroom stick along the harvesting direction. The contact head at the front of the active pusher is covered with a flexible material that increases friction, and the support structure of the active pusher is equipped with a constant force coil spring. A constant force coil spring is mounted to the support structure of the active pusher and the other side is attached to the support structure of the passive pusher to provide a clamping force for the stick and a force to pull back the passive pusher.

The passive pusher forms a straight line with the active pusher to assist in clamping the mushroom stick, and then continues to move in the harvesting direction through the harvesting unit. The passive contact head, mounted on the front of the passive pusher, is a self-resetting movable structure, which is self-resetting by means of four springs over four slidable columns. The front of the passive contact head is similarly affixed with a flexible material to increase friction, and a proximity switch is internally mounted to detect the clamping status.

The harvesting unit harvests the mushrooms from the sticks. The harvesting unit consists of a base, floating rods, raised supports, recessed supports and a flexible harvesting ring. Floating rods are mounted on the base, raised supports and recessed supports are mounted in a staggered arrangement on each of the floating rods, and the flexible picking ring is adhered to the raised supports and recessed supports to form a wave shape that floats to hold the sticks as they pass through.

2.3.1.3. Collection device

The collection device consists of two parts: a conveyor belt and sorting collection baskets. Three sorting collection baskets are placed on the conveyor belt for collecting mushrooms, which come from different batches of mushroom sticks. After evaluating the batches to which the mushroom sticks belong based on the ReYOLO-MSM method proposed in this paper, the corresponding collection basket are conveyed under the harvesting unit for collecting shiitake mushrooms from that batch, thus realizing selective collection of shiitake mushrooms from different batches of sticks.

2.3.1.4. Control section

The control section includes both the host computer and the peripheral device. The host computer is DELL-G3 3500, equipped with an Intel i7-10750H CPU@2.60 GHz processor, 32 GB of RAM, and an NVIDIA GeForce GTX1650Ti graphics card. The 64-bit Windows 10 operating system is installed, and the software development environment is Pytorch 1.8 + CUDA11.1. The peripheral device uses a XINJE XD5E-30 T-E PLC controller as the control core, and connects the PLC to the laptop through the serial communication port.

2.3.2. Working process

The working process of the selective harvester for shiitake mushroom sticks is divided into 4 stages as follows:

• (1)

  Stick transportation process: The sticks are placed in groups of three uninterruptedly onto a trapezoidal chain plate, and the transport device transports the sticks in the harvesting direction to the position of the harvesting unit. The transport process is shown in Fig. 7a.

  

  1. Download: Download high-res image (2MB)
  2. Download: Download full-size image

  Fig. 7. Working process.

• (2)

  Stick evaluation process: When there is a mushroom stick on the chain plate and it reaches the position for taking photos, the infrared sensor is triggered, which in turn triggers the camera to take photos and obtain images of the mushroom stick. Then according to the ReYOLO-MSM evaluation method, the cultivation status of the mushroom stick is evaluated, the batch to which the stick belongs is determined, the information is sent to the upper computer, and the relevant data is saved to the terminal, and then the transport device continues to run to the harvesting device. As shown in Fig. 7a.

• (3)

  Mushroom harvesting process: After the stick reaches the harvesting position, the active pusher moves forward to push the stick to press on the passive pusher (Fig. 7b), and the proximity switch of the passive contact head triggers to sense the stick clamping status. Subsequently, the clamped stick is pushed through the harvesting unit, and the mushrooms are separated from the stick under the pressure of the harvesting ring (Fig. 7c), and fall into the collection basket below. In addition, different batches of shiitake mushrooms fall into corresponding collection baskets with the cooperation of the collection device.

• (4)

  Stick recovery process: After the mushroom sticks are harvested, the active pusher and passive pusher continue to clamp the sticks and move them until the sticks reach the recycling device, the clamping state is released, and the recycling device begins to transport the harvested sticks away. As shown in Fig. 7d.

2.4. Sticks and mushrooms dataset

Image acquisition was performed by placing the picking stage mushroom sticks on a homemade simple data acquisition platform. The acquisition device is a Micro Seiko XW800 model monocular color industrial camera (device parameters: resolution 3840 x 2160 pixels, frame rate 30 FPS). The acquisition equipment is installed directly above the mushroom stick, 50 cm from the horizontal center section of the stick, and the installation position is shown in Fig. 8. Images of sticks under different light intensities were collected at 9:00 ∼ 11:00 and 15:00 ∼ 17:00 every day. A total of 880 images of the mushroom sticks were captured in the mushroom room environment, and Fig. 9 shows the captured images of the mushroom sticks. The LabelImg annotation software is used to manually annotate the sticks and shiitake mushrooms in the 880 mushroom stick images, which are produced as a TXT format dataset, and the dataset was divided into a training set and a validation set to the ratio of 9:1.

1. Download: Download high-res image (70KB)
2. Download: Download full-size image

Fig. 8. Image acquisition device.

1. Download: Download high-res image (312KB)
2. Download: Download full-size image

Fig. 9. Images of the collected mushroom sticks. (a) Images in the morning, (b) Images in the afternoon.

2.5. ReYOLO-MSM evaluation method

This paper proposes a ReYOLO-MSM mushroom stick evaluation method based on monocular vision and stick-mushroom growth relationship. ReYOLO refers to the improved rotating elliptic target detection algorithm based on YOLOv5; MSM refers to monocular vision and the stick-mushroom growth relationship. The ReYOLO-MSM evaluation method is divided into two critical stages: (1) fast and accurate recognition of stick and shiitake mushrooms based on the improved ReYOLO rotating elliptic algorithm; (2) measurement of shiitake mushroom size and roundness based on the stick-mushroom growth relationship. In addition, the percentage of qualified shiitake mushrooms needs to be calculated so as to evaluate stick cultivation status. Fig. 10 shows the workflow of the ReYOLO-MSM evaluation method.

1. Download: Download high-res image (689KB)
2. Download: Download full-size image

Fig. 10. ReYOLO-MSM evaluation method workflow.

2.5.1. Improved ReYOLO rotational elliptic model

YOLOv5 consists of 4 main parts: Input, Backbone, Neck and Head. The input module incorporates Mosaic data augmentation, adaptive anchor frame calculation, and adaptive image scaling for dataset enrichment and improved model robustness. The backbone network comprises the Focus module for down-sampled feature maps, Convolutional Layer for feature extraction, CSP Connection Layer for splitting and fusing features, and SPP Layer for multi-scale feature capture. The neck network employs FPN and PAN structures for semantic information fusion and localization features conveyance. The FPN+FAN structure enhances multi-scale target detection, improving overall model performance. The head network generates object detection results by combining multi-scale feature maps with anchor frames followed by NMS and bounding box regression. The structure of YOLOv5s is shown in Fig. 11.

1. Download: Download high-res image (415KB)
2. Download: Download full-size image

Fig. 11. YOLOv5s model structure.

Although YOLOv5 is widely used in various target detection tasks, there are still some problems in applying it to the detection of shiitake mushrooms cultivated on sticks: the diameter and roundness of shiitake mushrooms are not measured accurately enough due to the random distribution, differences in height, varying sizes and shapes, and cap tilting of shiitake mushroom caps on sticks. Shiitake mushrooms are basically elliptical in shape, based on the shape of Shiitake mushrooms, the YOLOv5s network model is selected and improved to ReYOLO on the basis of this model so that it is suitable for rotating elliptical box target detection of shiitake mushrooms. In this study, the ReYOLO model is improved from the following five aspects:

a. Input improvement

The more precise the labeling method is, the less redundant information is provided to the network for training, and the better it is for constraining the training direction of the network and reducing the convergence time of the network. The original YOLOv5s network uses a horizontal rectangular box to label the location and category of the target. The parameters of the horizontal rectangular box include the horizontal and vertical coordinates of the center point of the detection box, the height, and the width of four parameters, which are usually expressed as (x, y, w, h). The rotational detection method uses a rotating rectangular box with an angle, and a new direction angle parameter θ is added compared to the general horizontal rectangular box, using a five-parameter representation (x, y, w, h, θ). The rotating rectangular box adopts the method of defining the long side, where (x, y) are the coordinates of the center point, the long side is defined as w, the short side is defined as h, and the angle of rotation θ is defined as the angle between the long side h and the x-axis, as shown in Fig. 12. The label file format of the rotating box is [class, x, y, w, h, θ]. Improvements to the labeling method and labeling parameters of the input, as well as turning off the random affine transform data enhancement method to prevent causing a change in the target's angle, were subsequently fed into the network for training.

1. Download: Download high-res image (90KB)
2. Download: Download full-size image

Fig. 12. Long side definition method.

b. RepVGG structure

To enhance the feature extraction capability of ReYOLO backbone, this study adopts the RepVGG structure, comprising two main components: RepVGG Block and RepVGG Network. The RepVGG Block is the basic building block of the network, which contains a series of convolutional layers (Conv), batch normalization layer (BN), and ReLU activation function. The RepVGG Network consists of multiple RepVGG Blocks, which are stacked to form a deep network. The RepVGG architecture unifies the convolutional and fully connected layers into a convolutional operation, which reduces the number of parameters and computation of the network and improves the efficiency of the model training and inference through the use of learnable convolutional kernel parameters. In this study, parallel 1x1 convolutional branches and identical mapping branches are added to each original 3 × 3 convolutional layer in the backbone network to form a RepVGG Block, and multiple RepVGG Blocks are stacked to form a RepVGG network. The improved backbone network makes two convolutional layers participate in feature extraction simultaneously, fuses the feature information extracted by different convolutional modules, and improves the information extraction ability of the overall structure while reducing the number of model parameters. The improved backbone network is shown in Fig. 13.

1. Download: Download high-res image (153KB)
2. Download: Download full-size image

Fig. 13. The improved backbone structure.

c. Coordinate attention mechanism

Although the rotating rectangular box labeling method can label the target more accurately, it also adds the additional parameter of rotation angle, and the computation of the model will be increased, which will also have an impact on the overall detection accuracy of this model. Therefore, this study introduces the coordinated attention mechanism (CA) into the Neck part of the ReYOLO network, and the structure is shown in Fig. 14. In this study, adding the coordinated attention mechanism after the CSP2_1 module promotes stronger interconnections between feature maps, enabling the network to selectively emphasize relevant spatial and channel information, which helps to improve the network's discriminative ability and increase the sensitivity to key features, thus realizing more accurate localization and identification of target objects. At the same time, after processing by the attention mechanism module, the original feature-related part of the region is given a higher weight, while the feature-irrelevant background region is given a lower weight, which enables the network to focus on the processing of the feature-related part of the region and better capture the complex relationship between the object and its surroundings, improves the network's understanding of the context around the target, and mitigates the influence of irrelevant information, such as background, on the detection results, thus improving the detection accuracy.

1. Download: Download high-res image (112KB)
2. Download: Download full-size image

Fig. 14. The improved Neck structure.

d. Loss function improvement

The original YOLOv5 uses a horizontal rectangular detection box with a loss function that includes only regression loss, confidence loss, and classification loss. In order to realize rotating target detection, an angle loss function that handles angle information needs to be added. When rotating targets are detected, there is a problem of discontinuity in the boundaries of the detection box, which generates loss values at the boundaries and affects the detection effect. In this study, the circular smoothing label (CSL) is introduced into the ReYOLO network to transform the angle regression problem into a classification problem. CSL (Yang et al., 2022) can be used to smooth the tilt angles of the prediction box and the target box by setting the window function to avoid the effect of the surge of boundary loss due to the periodicity of the angle, and the CSL expression is as follows:(1)$\mathit{CSL}(x)=\left\{\begin{array}{c}g(x),\begin{array}{cc}& \theta -r<\text{x}<\theta +\text{r}\end{array}\\ 0,\begin{array}{ccc}& & \mathit{otherwise}\end{array}\end{array}\right.$where: g(x) is the window function, r is the radius of g(x) (pixels), and θ is the tilt angle of the detection frame (°). The window function in CSL satisfies the four attributes of periodicity, monotonicity, symmetry, and maximum value of 1. Its four attributes are utilized to solve the problem of loss due to the periodicity of the angle, and since the angle regression task has been transformed into a classification task, the angle information can be processed by CSL, and then the loss can be calculated by using the binary cross entropy loss function (BCEWithLogits) with the following equation:(2)$L_{\theta }=\frac{1}{N}\sum _{n-1}^N{l}_n$In the Eq.,(3)$l_n=-{\omega }_n\left[y_n·l_n(\sigma (x_n))+(1-y_n)\right]$and(4)$\sigma (x)=\frac{1}{1+\exp (-x)}$Where $L_{\theta }$ is the angle loss function, N is the batch size for model training, $l_n$ is the loss value corresponding to the nth sample, $x_n$ is the output of the nth sample processed by the sigmod activation function, $y_n$ is the category corresponding to the nth sample, and ${\omega }_n$ is the hyperparameter.

YOLOv5 classification loss and confidence loss both use the binary cross entropy loss function (BCEWithLogits), where the classification loss only calculates the loss of the positive samples, while the confidence loss calculates the loss of all the samples, and these two types of loss calculations perform well. The equations for the classification loss function and the confidence loss function are as follows:(5)$L_{\mathit{cla}}=-\sum _i^N\sum _j^C{I}_i^{\mathit{obj}}\left[O_{\mathit{ij}}\ln \left({\widehat{C}}_{\mathit{ij}}\right)+(1-O_{\mathit{ij}})\ln \left(1-{\widehat{C}}_{\mathit{ij}}\right)\right]$(6)$L_{\mathit{conf}}=-\sum _i^N\left[O_i\ln \left({\widehat{C}}_i\right)+(1-O_i)\ln \left(1-{\widehat{C}}_i\right)\right]$

The regression loss function used in YOLOv5 is the CIoU loss, which integrates the centroid distance, aspect ratio, and overlap area between the target and prediction boxes. It introduces an aspect ratio adjustment term to penalize the difference in aspect ratio between the prediction box and the target box, and a diagonal distance term to consider the spatial relationship between the target box and the prediction box more comprehensively. The $L_{\mathit{CIoU}}$ loss function is formulated as follows:(7)$\mathit{CIoU}=IoU-\left(\frac{{\rho }^2(b,b^{\mathit{gt}})}{c^2}+\alpha \upsilon \right)$(8)$\upsilon =\frac{4}{{\pi }^2}{\left(\arctan \frac{w^{\mathit{gt}}}{h^{\mathit{gt}}}-\arctan \frac{w}{h}\right)}^2$(9)$\alpha =\frac{\upsilon }{\left(1-IoU\right)+\upsilon }$(10)$L_{\mathit{CIoU}}=1-CIoU$

Overall, the improved loss function in this paper consists of the classification loss function, the confidence loss function, the regression loss function, and θ angle loss function, which is calculated as follows:(11)$\mathit{Loss}=L_{\mathit{cla}}+L_{\mathit{conf}}+L_{\mathit{CIoU}}+L_{\theta }$

e. Rotating ellipse box fitting

After completing the previous four steps to improve the strategy, it has been achieved to use the rotated rectangular box for target detection, and the information obtained includes the center point coordinates of the rotated rectangular box, the lengths of the long and short edges, the angle of the rotated rectangular box, the classification, and the confidence degree. Considering that shiitake mushrooms are usually elliptical in shape, the improved rotational ellipse detection algorithm is more appropriate. Based on the center point coordinates, long side, short side, rotation angle obtained above, the elliptical box can be fitted, the center point coordinates are the elliptical box center point coordinates, the long side and the short side are the elliptical box long and short axes, respectively, and the rotation angle is also the elliptical box rotation angle. The bounding box drawing function in the post-processing part of the model is improved to accurately depict the shape characteristics of the target by superimposing the fitted rotating ellipse box onto the original image. The length of the long axis obtained at this point is the maximum diameter of the cap of the shiitake mushroom, and in addition, the roundness of the mushroom can be easily calculated based on the elliptical contour.

2.5.2. ReYOLO-MSM evaluation method

The second stage of the ReYOLO-MSM evaluation method requires summarizing and regressing the stick-mushroom growth relationship, based on this relationship three critical parameters are obtained: shiitake mushroom growth height, shiitake mushroom root height, and relationship between distance and pixel ratio. In addition, mushroom cap diameter is measured and mushroom cap roundness is calculated based on these three parameters, and then the percentage of qualified shiitake mushrooms is calculated, thereby evaluating the mushroom stick cultivation status.

2.5.2.1. Stick-Mushroom growth relationship

By observing a large number of mushroom sticks, we found that the following two growth relationships exist between the stick-mushroom: (1) In general, the larger the shiitake mushroom cap, the higher the growth height of the shiitake mushroom; (2) The shiitake mushrooms grow on the upper half-circular arc of the stick and will maintain a vertical upward trend of growth, and the closer the growth point is to the vertical center section of the stick, the higher the root height of the shiitake mushrooms, and the farther the growth point is from the vertical center section of the stick, the lower the root height of the shiitake mushrooms. By summarizing and regressing the above growth relationships, the following critical parameters are obtained.

(a) Shiitake mushroom growth height regression

The ReYOLO-MSM mushroom stick evaluation method designed in this paper, combines monocular vision and the stick-mushroom growth relationship, necessitating the utilization of the growth height of shiitake mushrooms. However, the growth height of shiitake mushrooms on the mushroom stick is difficult to directly obtain through processing images taken by a monocular camera in a top-down perspective. Adding the side camera will not only increase the algorithm difficulty, but also increase the cost, and according to the actual application scenario that the mushroom sticks are transported on the chain plate, it is impossible to add a side camera. Therefore, it is considered to design a method to estimate the growth height of shiitake mushroom by processing the overhead image of the mushroom stick. In general, the larger the shiitake mushroom cap, the higher the growth height of the shiitake mushroom. There is a certain growth relationship between the two, which is summarized and regressed, and can be used to estimate the growth height of shiitake mushroom based on this growth relationship by substituting the shiitake mushroom cap size initially obtained by processing the top view of the mushroom stick. In order to investigate the correlation between shiitake mushroom caps size and their growth height, we selected the mushroom sticks during the picking period in the mushroom room No. 16 of Shandong Province QIHE Biotechnology Co. Ltd. and collected 100 shiitake mushrooms with cap diameters within the range of 30–60 mm, and measured their cap diameters and the corresponding growth heights, respectively, as shown in Fig. 15, and recorded the data in Table 2.

1. Download: Download high-res image (77KB)
2. Download: Download full-size image

Fig. 15. Shiitake mushroom parameters.

Table 2. Basic information of measured mushrooms.

No. of mushroom	1	2	3	4	5	6	7	8	9	10
Cap diameter (mm)	43.8	46.2	47.5	52.3	52	48.9	53.1	52.1	51.8	49.7
Growth height (mm)	42.7	45.3	46.4	55.1	49.7	48.9	50.6	46.9	52.5	49.1
No. of mushroom	11	12	13	14	15	16	17	18	19	20
Cap diameter (mm)	50.1	42.5	35.9	46.2	42.2	41.2	44.1	39.3	55.1	53.9
Growth height (mm)	51.7	44.9	42.1	44.8	43.9	39.3	44.7	39.5	54.6	54.7
… …
No. of mushroom	81	82	83	84	85	86	87	88	89	90
Cap diameter (mm)	43.6	43.8	40.8	52.9	47.6	50.3	47.4	32.7	34.3	32.7
Growth height (mm)	46.2	40.3	40.4	52.1	40	50.7	45.4	39.4	35.6	35.2
No. of mushroom	91	92	93	94	95	96	97	98	99	100
Cap diameter (mm)	33.1	51	31.5	60.2	51.2	51.8	59.6	51.2	46.8	47.3
Growth height (mm)	35.3	46.2	36.8	51.4	50.4	52.5	54.7	54.7	47.5	43.6

We imported measured data into Origin 2021, creating a scatter plot with mushroom cap diameter (X) and growth height (Y). Using “Mask Points on Active Plot” command to mask some data points with large deviations to minimize deviations in subsequent curve fitting. Since the scatter plot showed a linear trend, we selected “Fit Linear” from the “Fitting” menu, and the fitted line is shown in Fig. 16. The results displayed a fitted line equation: y = 0.72265*x + 12.56044, with the slope of 0.72265 and intercept of 12.56044. The correlation coefficient squared ($R^2$=0.96445) indicated a strong fit.

1. Download: Download high-res image (138KB)
2. Download: Download full-size image

Fig. 16. Linear fitting.

(b) Shiitake mushroom root height regression

The root height of the shiitake mushroom is defined as the distance between the shiitake mushroom root on the stick and the horizontal middle section of the stick, as shown in Fig. 17. The root height of shiitake mushrooms is also a critical parameter in the ReYOLO-MSM evaluation method. However, the root height of shiitake mushrooms on the mushroom stick is also difficult to directly obtain through processing images taken by a monocular camera in a top-down perspective. Adding the side camera will not only increase the algorithm difficulty, but also increase the cost, and according to the actual application scenario that the mushroom sticks are transported on the chain plate, it is impossible to add a side camera. Therefore, it is considered to design a method to estimate the root height of shiitake mushrooms by processing the overhead image of the mushroom stick. We observed a large number of mushroom sticks and found that shiitake mushrooms grow on the upper half-circular arc of the stick and will maintain a vertical upward trend of growth, and the closer the growth point is to the vertical center section of the stick, the higher the root height of the shiitake mushrooms, and the farther the growth point is from the vertical center section of the stick, the lower the root height of the shiitake mushrooms. There exists a certain growth relationship between these two, which is summarized and regressed, the mushroom root height can be estimated based on this growth relationship by substituting the distance from the mushroom cap center point to the vertical middle cross-section of the stick obtained by processing the overhead image of the mushroom stick. In order to regress the relationship between the distance from the center point of shiitake mushroom cap to the vertical middle section of the mushroom stick and the root height of the shiitake mushroom, we measured 100 shiitake mushrooms with cap diameters within the range of 30–60 mm in the 16th mushroom room of Shandong Province QIHE Bio-technology Co. Ltd. and the distance from the center point of the shiitake mushroom cap to the vertical middle section of the mushroom stick and the corresponding root height were measured respectively as shown in Fig. 17, and the data were recorded in Table 3.

1. Download: Download high-res image (396KB)
2. Download: Download full-size image

Fig. 17. Shiitake mushroom root height.

Table 3. Basic information of measured mushrooms.

No. of mushroom	1	2	3	4	5	6	7	8	9	10
Distance (mm)	52	55	55	36	29	49	28	52	48	46
Root height (mm)	24	26	27	37	42	15	35	26	24	28
No. of mushroom	11	12	13	14	15	16	17	18	19	20
Distance (mm)	59	61	23	70	24	67	56	74	52	70
Root height (mm)	12	15	38	0	39	0	28	0	23	0
… …
No. of mushroom	81	82	83	84	85	86	87	88	89	90
Distance (mm)	23	35	60	53	16	64	72	32	47	20
Root height (mm)	37	25	14	13	40	12	0	35	21	42
No. of mushroom	91	92	93	94	95	96	97	98	99	100
Distance (mm)	42	55	20	47	53	40	46	42	40	19
Root height (mm)	23	11	37	20	10	25	19	25	26	42

We imported measured data into Origin 2021, creating a scatter plot with the distance from the shiitake mushroom cap center to the vertical middle section of the stick (X) and mushroom root height (Y). Using “Mask Points on Active Plot” command to mask some data points with large deviations to minimize deviations in subsequent curve fitting. Since the scatter plot indicated an inverse relationship, we selected “Fit Polynomial” from the “Fitting” menu, and the fitted curve is shown in Fig. 18. Results displayed a fitted curve equation: y = -0.24769*x-0.00544*x² + 56.83587, with primary term slope of −0.24769, quadratic term slope of −0.00544, and intercept of 47.26755. The square of the correlation coefficient ($R^2$=0.9396) affirmed a strong fit.

1. Download: Download high-res image (134KB)
2. Download: Download full-size image

Fig. 18. Polynomial fitting.

(c) Regression of the relationship between distance and pixel ratio.

The method proposed in this paper takes the horizontal middle cross-section of the mushroom stick as the reference plane, the growth of shiitake mushrooms on the stick varies, and the height of their caps from the reference plane varies, i.e., the distance between the lens and the cap varies, so it is necessary to know the relationship between the distance and the pixel ratio in order to use it to calculate the pixel ratios corresponding to different distances, so as to measure the diameter of the shiitake mushroom caps and calculate the roundness of shiitake mushroom caps. This paper designs a way to establish the relationship between distance and pixel ratio. A 60 mm diameter circle is drawn on a plane as a marker, with a 60 mm diameter circular sign positioned to the right, elevated 70 mm above the plane. A camera captures images from a height of 50 cm directly above. The circular sign's height increases incrementally by 5 mm, up to 160 mm, with an image taken at each height. Image processing of these pictures yields the relationship between distance and pixel ratio, detailed in Fig. 19.

1. Download: Download high-res image (89KB)
2. Download: Download full-size image

Fig. 19. Processed images.

The pixel value of the right circular sign from the photographed images are converted to diameters by image processing, and the pixel ratio at different heights can be obtained by dividing this diameter value by the left marker's diameter, and the pixel ratios at different heights of the circular signs are tabulated in Table 4. We imported collected data into Origin 2021, creating a scatter plot with circular sign height (X) and pixel ratio (Y). Using “Mask Points on Active Plot” command to mask some data points with large deviations to minimize deviations in subsequent curve fitting. Since the scatter plot showed a linear trend, we selected “Fit Linear” under the “Fitting” menu, and the fitted line is shown in Fig. 20. The results displayed a fitted line equation: y = 0.00333*x + 0.87233, with the slope of 0.00333 and intercept of 0.87233. The correlation coefficient squared ($R^2$=0.99015) indicated a strong fit.

Table 4. Pixel ratios for different heights.

Height (mm)	Original diameter (mm)	Processed image diameter (mm)	Pixel ratio
70	60	67	1.1167
75	60	67.9	1.1317
80	60	68.7	1.145
85	60	69.6	1.16
90	60	70.4	1.1733
95	60	71	1.1833
100	60	72	1.2
105	60	73.1	1.2183
110	60	74	1.2333
115	60	74.8	1.2467
120	60	75.7	1.2617
125	60	76.7	1.2783
130	60	77.6	1.2933
135	60	78.8	1.3133
140	60	80	1.3333
145	60	81.2	1.3533
150	60	82.9	1.3817
155	60	83.9	1.3983
160	60	85.6	1.4267

1. Download: Download high-res image (94KB)
2. Download: Download full-size image

Fig. 20. Linear fitting.

2.5.2.2. Mushroom cap diameter measurement

In the previous section, the stick-mushroom growth relationship has been summarized and regressed, and this section will be based on the relationship to measure the shiitake cap diameter. After the process of stick and shiitake mushroom cap recognition by the improved ReYOLO rotating ellipse algorithm in Section 2.5, we can obtain the mushroom stick center coordinates, as well as the center coordinates and the long axis length of each shiitake mushroom on the stick. However, at this time we only get the pixel center coordinates, pixel length, and the growth of shiitake mushrooms on the stick varies, and the shiitake mushroom caps are at different height from the set reference plane, so it is not possible to uniformly convert the pixel length to the real shiitake mushroom cap size. Based on the regressed stick-mushroom growth relationship, the mushroom growth height and mushroom root height can be calculated, thereby deriving the depth of each mushroom to the reference plane, and finally the actual size of the mushroom cap can be converted by the pixel ratio corresponding to each mushroom.

The schematic diagram of the stick-mushroom growth relationship is shown in Fig. 21. The frontal schematic of the stick-mushroom growth relationship is shown in Fig. 21a, with the horizontal middle cross-section of the stick as the reference plane, the point p represents the optical center of the camera, f is the focal length, H is the distance between the point p and the reference plane, and S is the distance between the point p and the shiitake mushroom cap. Since the diameter values obtained after the recognition of stick and shiitake mushroom cap are in pixel coordinates, it is necessary to convert the pixel coordinates to the actual coordinates. The specific method is as follows: place a disc with a diameter of 50 mm on the set reference plane as a reference object, and then detect the pixel coordinate difference between the reference object diameters several times and take the average value, and utilize the Eq. (12) to calculate the ratio between the pixel coordinates and the actual coordinates:(12)$k=\frac{L_0}{D}$Where k is the number of pixels per unit length on the reference plane, $L_0$ represents the pixel width of the disc diameter, and D represents the actual width of the disc diameter. By dividing the obtained diameter value in pixel coordinates by k, the initial diameter value of the shiitake mushroom cap can be obtained. As can be seen from Fig. 21a, the shiitake mushrooms are all growing above the reference plane, and thus the initial shiitake mushroom cap diameter obtained base on the reference plane is equivalent to being enlarged. Therefore, it is necessary to further convert the actual diameter value of the shiitake mushroom by combining the stick-mushroom growth relationship.

1. Download: Download high-res image (153KB)
2. Download: Download full-size image

Fig. 21. Schematic diagram of the stick − mushroom growth relationship.

In Fig. 21b, a schematic diagram of the side view of the stick-mushroom growth relationship is shown, H is the distance between point p and the reference plane, S is the distance between point p and the shiitake mushroom cap, L is the distance from the shiitake mushroom cap center point to the vertical middle cross-section of the stick, $h_1$ is the shiitake mushroom growth height, $h_2$ is the shiitake mushroom root height, and R is the radius of the stick, which has a basically the same specification. Through the regression and fitting of the stick-mushroom growth relationship in Section 2.5.1, the relationship equation between the initial diameter of the mushroom cap and the mushroom growth height (y = 0.72265*x + 12.56044) and the relationship equation between the distance from the mushroom cap center point to the vertical middle cross section of the stick and the mushroom root height (y = -0.24769*x-0.00544*x² + 56.83587) were obtained, respectively. Where the distance from the shiitake mushroom cap center point to the vertical middle cross-section of the stick is calculated by the absolute value of the difference between the coordinate value $y_m$ of the center point of each shiitake mushroom cap and the coordinate value $y_s$ of the stick center point.

The value of k under the height of H (on the reference plane) has already been calculated by Eq. (12), and then we can preliminarily estimate the shiitake mushroom cap diameter value, but this estimated diameter value is enlarged, and in order to accurately obtain the shiitake mushroom cap diameter value, it is necessary to obtain the pixel ratio of the distance between the cap center and the reference plane, and then the exact shiitake mushroom cap diameter can be estimated by dividing the preliminary estimate of shiitake mushroom cap diameter by the obtained pixel ratio. In Section 2.5.2.1, we have obtained the relationship equation between distance and pixel ratio, and the corresponding pixel ratio can be calculated by bringing the distance between the shiitake mushroom cap center point and the reference plane into the relationship equation. As can be seen in Fig. 21b, the distance between the shiitake mushroom cap center point and the reference plane is then equal to the sum of the shiitake mushroom growth height and the shiitake mushroom root height.

The above process can be derived the accurate shiitake mushroom cap diameter by using the following equation:(13)$D_0=\frac{D_p}{k}$(14)$h_1=k_1*D_0+b_1$(15)$L=\left|y_m-y_s\right|$(16)$h_2=k_2*L+b_2$(17)$h_s=h_1+h_2$(18)$k_s=k_3*h_s+b_3$(19)$D_m=\frac{D_0}{k_s}$Where $D_p$ represents the shiitake mushroom cap diameter value in pixel coordinates, $D_0$ is the preliminarily estimated shiitake mushroom cap diameter value, k is the number of pixels per unit length on the reference plane, $h_1$ is the shiitake mushroom growth height, $h_2$ is the shiitake mushroom root height, L is the distance from the shiitake mushroom cap center point to the vertical middle section of the mushroom stick, $y_m$ is the y-value of the shiitake mushroom cap center point, $y_s$ is the y-value of the center point of the fungus stick, $h_s$ is the distance between the shiitake mushroom cap center point and the reference plane, $k_s$ is the pixel ratio corresponding to the distance between the shiitake mushroom cap center point and the reference plane, and $D_m$ is the exact shiitake mushroom cap diameter.

Fig. 22 shows the measurement results. The green point shown in the figure is the center point of the mushroom stick, the white coordinate at the center of each mushroom prediction frame indicates the positional coordinate of the mushroom cap center point under the pixel coordinate system, and the blue coordinate indicates the measured size of the shiitake mushroom cap.

1. Download: Download high-res image (784KB)
2. Download: Download full-size image

Fig. 22. Shiitake mushroom cap size measurement results.

2.5.2.3. Mushroom cap roundness calculation

Normally mushroom caps are close to circular. For a round target, the larger the roundness Q value is, the more complex the target shape is. Therefore, roundness can be used as a measure of the complexity of the target shape. In this study, roundness was used to measure the shape characteristics of shiitake mushrooms. Roundness of not more than 30 % is qualified mushrooms. According to the formula of circle circumference and circle area, the formula of roundness Q is as follows:(20)$Q=(\frac{4\pi A}{L^2}-1)/100\%$Where, A is the area of the mushroom area; L is the perimeter of the mushroom area.

From the formula of roundness, it can be seen that in order to find the roundness Q, the area and perimeter of the studied shiitake mushroom must be obtained first. We used a rotating ellipsoid box detection algorithm that can easily derive these two parameters. The area A can be obtained by calculating the total number of pixels of the shiitake mushroom in the shiitake image; the perimeter L is the length of the contour line of the shiitake mushroom boundary, because there are movements on the contour line in the vertical and horizontal directions as well as in the diagonal direction, and if the pixel values on the contour line are simply calculated cumulatively, the lengths in the vertical and horizontal directions will be exaggerated, and for this reason, the pixels in the 2 directions are categorized for the calculation. The formula for calculating the perimeter length is as follows:(21)$L=n_e+\sqrt{2}·n_o$Where, $n_e$ is the number of pixels in the horizontal and vertical directions on the contour line; $n_o$ is the number of pixels in the diagonal direction on the contour line.

Based on the measured diameter of shiitake mushrooms in section 2.5.2.2 and the calculated roundness of shiitake mushrooms in section 2.5.2.3, one can ascertain the eligibility of shiitake mushrooms attached to the stick. This enables the determination of the percentage of shiitake mushrooms meeting the specified criteria, facilitating a selective harvesting for shiitake mushroom sticks.

2.6. Performance evaluation

2.6.1. ReYOLO model evaluation

The model was trained and validated on the NVIDIA GeForce GTX1650Ti graphics card (32 GB memory), Intel i7-10750H CPU@2.60 GHz processor with Pytorch 1.8 and CUDA11.1. The model was trained over 300 epochs with a learning rate of 0.001 and a minimum batch size of 16. To evaluate the performance of the mushroom stick and shiitake mushroom cap identification model, precision (P), recall (R), mean average precision (mAP), and single-frame image detection time (t) are used as evaluation metrics. The relevant evaluation metrics are as follows:

(1) Precision. Precision represents the proportion of true positive samples in the predicted positive samples within the recognized image.(22)$P=\frac{\mathit{TP}}{\mathit{TP}+FP}\times 100\%$Where TP represents the number of true positive samples predicted by the model (originally positive samples predicted as positive samples), and FP represents the number of false positive samples predicted by the model (originally negative samples predicted as positive samples).

(2) Recall rate. The recall rate refers to the proportion of correctly predicted positive samples to the total positive samples.(23)$R=\frac{\mathit{TP}}{\mathit{TP}+FN}\times 100\%$Where FN refers to the number of positive samples that were predicted as negative samples by the model (originally positive samples, but predicted as negative samples).

(3) mAP. Taking Recall and Precision as the x-axis and y-axis, respectively, the P-R curve is obtained, with the area underneath being the AP value. The mAP value is the average value of AP, which is an important parameter showing the performance of the constructed neural network model, and the higher the value is, the better the detection effect of the algorithm.$\mathit{AP}={\int }_0^{1}P\left(r\right)dr$(24)$\mathit{mAP}=\frac{1}{n}\sum _{i=1}^n\mathit{AP}(i)$

2.6.2. Cap measurement precision experiment

To evaluate the precision of the ReYOLO-MSM method proposed in this paper to measure the cap diameter of shiitake mushrooms, an experimental platform was set up in the No. 17 mushroom cultivation shed of Shandong QIHE Biological Technology Co., Ltd. in Zibo City, Shandong Province. The experimental environment is shown in Fig. 23, and the image acquisition is done using the XW800 industrial camera from Micro Seiko.

1. Download: Download high-res image (727KB)
2. Download: Download full-size image

Fig. 23. Experimental platform.

As shown in Fig. 23, the mushroom sticks are placed on the transportation device, the camera is located directly above and in the middle of the sticks, and the camera lens is fixed at a distance of 53 cm from the horizontal middle section of the sticks (the set reference plane). The 20 images of mushroom sticks are captured and the maximum diameter of each shiitake mushroom on the stick is measured using a vernier caliper and recorded one by one into the captured images for subsequent one by one comparison, and the manually measured value of mushroom cap diameter is taken as the actual value $D_A$ , as shown in Fig. 24a. Then, the shiitake mushroom cap diameter values are measured using the ReYOLO-MSM evaluation method proposed in this paper, and the blue diameter value is marked at the center of each shiitake mushroom cap in the picture, denoted as the measured value $D_M$, as shown in Fig. 24b. The absolute difference between $D_M$ and $D_A$ is divided by $D_A$ to calculate the error, which serves as an indicator to evaluate the measurement precision of this method, as shown in Eq. (25). The experimental results are recorded as Table 6 in Section 3.2.(25)$E=\frac{\left|D_M-D_A\right|}{D_A}\times 100\%$

1. Download: Download high-res image (617KB)
2. Download: Download full-size image

Fig. 24. Mushroom cap diameter: (a) Manually measured cap diameter; (b) Evaluation method measured cap diameter.

Table 5. Training results for the mushrooms in test set.

Models	Precision (%)	Recall (%)	mAP (%)	Average Time(ms)
ReYOLO-MSM	99.3	98.5	98.9	7.1
YOLOv5s	99.3	98.2	98.4	7.5
YOLOv5m	98.9	98.3	98.6	8.1
YOLOv5l	99.1	98.5	98.8	8.5
YOLOv5x	99.3	98.6	98.8	8.9

Table 6. Experimental precision experimental results.

No.1	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	57.8	59.6	48.4	49	54.5	52.4	50.9
$D_M$(mm)	59.74	61.64	48.17	49.87	54.11	53.13	51.26
Error value (mm)	1.94	2.04	−0.23	0.87	−0.39	0.73	0.36
Error (%)	3.36	3.42	0.48	1.78	0.72	1.39	0.71
No.2	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	50.8	52.9	28.8	48.4	52.5	36.8
$D_M$(mm)	52.98	53.8	28.43	50.9	53.52	38.53
Error value (mm)	2.18	0.9	−0.37	2.5	1.02	1.73
Error (%)	3.63	1.70	1.28	5.17	1.94	4.70
No.3	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	51.6	40.7	47.9	52.9	52	28	24.6	50.9	34.6
$D_M$(mm)	51.5	43.0	49.72	53.45	51.55	27.43	26.1	52.3	39.15
Error value (mm)	−0.1	2.3	1.82	0.55	−0.45	−0.57	1.5	1.4	4.55
Error (%)	0.19	5.65	3.80	1.03	0.87	2.04	6.10	2.75	13.15
.	.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.	.
No.19	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	34.5	54	29.2	32.3	55.3	46.2	58.9	50.5	41.5	47.1
$D_M$(mm)	37.27	56.67	30.67	35.78	54.88	47.56	57.56	49.18	43.35	49.88
Error value (mm)	2.77	2.67	1.47	3.48	−0.42	1.36	−1.34	−1.32	1.85	2.78
Error (%)	8.03	4.94	5.03	10.77	0.76	2.94	2.28	2.61	4.46	5.90
No.20	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	38.2	59.5	50.8	49.3	50.7	39.9	54.1	50.2	43.2	51.9
$D_M$(mm)	39.54	58.36	50.84	48.29	51.57	39.68	55.29	50.01	41.86	52.65
Error value (mm)	1.34	−1.14	0.04	−1.01	0.87	−0.22	1.19	−0.19	−1.34	0.75
Error (%)	3.51	1.92	0.08	2.05	1.72	0.55	2.20	0.38	3.10	1.45
Average error: 2.18 %

2.6.3. Sticks evaluation experiment

In order to evaluate the performance of the ReYOLO-MSM evaluation method proposed in this paper for selective harvesting evaluation of mushroom sticks, a comparative experiment between the manual stick evaluation and the stick evaluation by the ReYOLO-MSM method was set up to verify the accuracy of the evaluation results of the ReYOLO-MSM method for mushroom sticks.

Randomly selecting 50 mushroom sticks at different cultivation stages, they were sequentially placed on the transportation device and evaluated in two ways. Firstly, manual procedures were employed to count the number of shiitake mushrooms, measure the size of shiitake mushrooms, and calculate the percentage of harvestable shiitake mushrooms, so as to evaluate the cultivation status of each stick. Subsequently, the proposed ReYOLO-MSM method was employed to detect the number of shiitake mushrooms, measure shiitake mushroom size and calculate mushroom roundness, calculate the percentage of harvestable shiitake mushrooms, so as to evaluate the cultivation status of each stick. The results of the comparative experiments between the two methods are recorded as Table 7 in Section 3.3. By contrasting the evaluation results of manual evaluation with those of the proposed ReYOLO-MSM method evaluation, the performance of the ReYOLO-MSM method was verified. Fig. 25 shows the mushroom sticks in different cultivation status.

Table 7. Results of the sticks evaluation experiment.

Empty Cell	ReYOLO-MSM evaluation				Manual evaluation
Empty Cell	No. of mushrooms	No. of harvestable mushrooms	Percentage	Stick status	No. of mushrooms	No. of harvestable mushrooms	Percentage	Stick status
S1	10	8	80 %	first batch	10	8	80 %	first batch
S2	11	10	90.91 %	first batch	11	10	90.91 %	first batch
S3	10	7	70 %	second batch	10	7	70 %	second batch
S4	8	6	75 %	first batch	8	7	87.5 %	first batch
… …
S47	10	7	70 %	second batch	10	7	70 %	second batch
S48	4	1	25 %	third batch	4	1	25 %	third batch
S49	9	7	77.78 %	first batch	9	8	88.89 %	first batch
S50	9	7	77.78 %	first batch	9	7	77.78 %	first batch

1. Download: Download high-res image (527KB)
2. Download: Download full-size image

Fig. 25. Sticks at different cultivation status: (a) First batch stick; (b) Second batch stick; (c) Third batch stick.

3. Results and discussion

3.1. Model performance

The improved model was trained using the shiitake mushroom training set and the model performance was evaluated by inputting the validation set. The performance of the model during training is shown in Fig. 26. Fig. 26a shows the loss function curve for training, the smaller the value the more accurate the target detection is, the minimum value is 0.00964 and the detection performance is good. Fig. 26b shows the P (precision) curve with a maximum precision of 99.3 %. Fig. 26c shows the R (recall) curve with a maximum recall of 98.5 %. Fig. 26d shows the average precision curve when the IOU threshold is set to 0.5. Fig. 26e shows the P-R curve generated during training with a threshold of 0.5. The P-R curve reflects the relationship between precision and recall, which is an important evaluation tool in the process of model training. Generally, recall is set as the horizontal coordinate and precision as the vertical coordinate. The area enclosed under the P-R curve is the AP, and the average of the AP of all the categories is mAP. The larger the value, the better the model recognition effect, and the mAP of 98.9 % indicates a good performance of the trained model.

1. Download: Download high-res image (227KB)
2. Download: Download full-size image

Fig. 26. Model training results. (a) Loss function curve; (b) P (precision) curve; (c) R (recall) curve; (d) mAP@0.5 curve; (e) P-R curve.

To further validate the detection performance of the ReYOLO-MSM model for mushroom sticks and shiitake mushroom caps, we compared the ReYOLO-MSM model with the original YOLOv5s, YOLOv5l, YOLOv5m, and YOLOv5x on 100 images from the validation set. The target distribution of the validation set is actually 100 sticks and 983 shiitake mushrooms, and the validation set images are fed into the above model separately. Table 5 shows the comparison results of each evaluation metric for the different recognition models mentioned above. At a confidence level of 0.5, the mAP values for ReYOLO-MSM and YOLOv5 (s, l, m, and x) are 98.9 %, 98.4 %, 98.6 %, 98.8 %, and 98.8 %, respectively, which indicates the good performance of all the models. In addition, the average time required to detect an image is 7.1 ms, 7.5 ms, 8.1 ms, 8.5 ms and 8.9 ms, respectively. It can be seen that the ReYOLO-MSM model has the fastest detection speed and maintains a high detection performance at the same time.

3.2. Measurement precision experimental results

The experimental results based on the proposed ReYOLO-MSM method of shiitake mushroom cap diameter measurement are shown in Table 6. Therein is a detailed record of the number of each stick, the number of the mushroom on each stick, and the actual diameter value $D_A$, measured diameter value $D_M$, relative error value and error corresponding to each mushroom. Absolute error values and errors for all tested shiitake mushroom cap diameters are counted in Fig. 27. The absolute error values and errors of the shiitake mushroom cap diameters at different locations on the mushroom sticks are counted in Fig. 28, with the vertical middle section of the sticks as the demarcation (at point 0), and the distance between the shiitake mushroom cap center point and the vertical middle section of the sticks as the X-axis, with growth on the left side being negative and growth on the right side being positive.

1. Download: Download high-res image (230KB)
2. Download: Download full-size image

Fig. 27. Shiitake mushroom cap diameter error statistic.

1. Download: Download high-res image (137KB)
2. Download: Download full-size image

Fig. 28. Error statistic of shiitake mushroom cap diameter at different locations.

As can be seen from Table 6 and Fig. 27, the maximum error of the shiitake mushroom cap diameter value is 13.15 % (orange), and its actual diameter value is 34.6 mm, the measured diameter value is 39.15 mm, the size of the error value is 4.55 mm, which is not a large error value. The minimum error of the shiitake mushroom cap diameter value is 0.08 % (blue), its actual diameter value is 50.8 mm and the measured diameter value is 50.84 mm, the size of the error value is 0.04 mm. The average error of all shiitake mushroom samples was 2.18 %, and the largest error value was only 4.55 mm, which is not a large error value for mushroom diameter measurement.

As can be seen from Fig. 28, the different growth positions of the shiitake mushrooms on the stick have an effect on the measurement error of shiitake mushroom cap diameter, the closer the shiitake mushrooms grow to the center of the stick, the smaller the measurement error of the shiitake mushroom cap diameter, and the further the shiitake mushrooms grow away from the center of the stick, the larger the measurement error of the shiitake mushroom cap diameter. This is due to the fact that the method proposed in this paper measures the shiitake mushroom cap diameter based on the height between the mushroom cap and the reference plane. This height is the sum of the shiitake mushroom growth height and the shiitake mushroom root height. For shiitake mushrooms growing in the middle of the stick, its root height is the radius of the stick, and the closer to the middle of the stick, the more accurate is the shiitake mushroom root height, thereby minimizing the impact of root height errors on diameter measurement accuracy. Consequently, the closer the shiitake mushroom is to the middle of the stick, the more accurate the measurement of its cap diameter will be.

The above experimental data indicates that the method (ReYOLO-MSM) proposed in this paper, which combines monocular vision and the stick-mushroom growth relationship, has good accuracy and can be further applied to the industrialized production of shiitake mushrooms. In addition, the possible reasons for the large errors in the estimation of individual shiitake mushroom diameters still present in the test results are: these shiitake mushrooms may grow abnormally on the stick due to insufficient de-budding, mushroom house environment or diseases, etc., which are not quite consistent with the mushroom stick-shiitake mushroom growth relationship, and thus the algorithm based on this growth relationship will have a large error in the estimation.

3.4. Comparison of monocular and binocular camera

This paper compares the effectiveness of monocular and binocular camera for measuring the mushroom cap diameter. The experimental results of the shiitake mushroom cap diameter measurement combining D435i depth information are shown in Table 8, where the actual value $D_A$, the measured value $D_M$, and the errors for all tested shiitake mushroom caps are recorded. Absolute error values and errors for all tested shiitake mushroom cap diameters are counted in Fig. 29.

Table 8. Experimental results of the shiitake mushroom cap diameter measurement combining depth information.

No.1	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	51.15	51.16	55.9	41.24	52.13	39.52
$D_M$(mm)	55.37	51.63	57.98	44.67	55.66	41.51
Error value (mm)	4.22	0.47	2.08	3.43	3.53	1.99
Error (%)	8.25	0.92	3.72	8.32	6.77	5.04
No.2	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	22.13	41.37	41.06	45.84	47.93	24.26	50.16	55.3	45.2
$D_M$(mm)	25.37	40.3	44.52	49.02	51.37	25.93	51.64	53.88	47.56
Error value (mm)	3.24	−1.07	3.46	3.18	3.44	1.67	1.48	−1.42	2.36
Error (%)	14.64	2.59	8.43	6.94	7.18	6.88	2.95	2.57	5.22
No.3	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	49.97	41	36.53	54.38	44.81	50.16	38.82
$D_M$(mm)	54.84	42.27	38.34	61.62	46.3	52.81	42.55
Error value (mm)	4.87	1.27	1.81	7.24	1.49	2.65	3.73
Error (%)	9.75	3.10	4.95	13.31	3.33	5.28	9.61
.	.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.	.
No.19	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	42.34	35.81	23.53	34.74	42.04	50.14	52.15	49.16
$D_M$(mm)	44.79	37.93	26.68	37.55	43.48	50.26	55.47	51.53
Error value (mm)	2.45	2.12	3.15	2.81	1.44	0.12	3.32	2.37
Error (%)	5.79	5.92	13.39	8.09	3.43	0.24	6.37	4.82
No.20	M1	M2	M3	M4	M5	M6	M7	M8	M9	M10
$D_A$(mm)	51.24	49.89	51.07	29.54	39.9	48.36	50.86	47.3
$D_M$(mm)	54.48	54.67	52.23	30.47	43.35	51.08	51.64	48.33
Error value (mm)	3.24	4.78	1.16	0.93	3.45	2.72	0.78	1.03
Error (%)	6.32	9.58	2.27	3.15	8.65	5.62	1.53	2.18
Average error: 5.24 %

1. Download: Download high-res image (300KB)
2. Download: Download full-size image

Fig. 29. Shiitake mushroom cap diameter error statistic.

As can be seen from Table 8 and Fig. 29, in the results of applying the binocular camera to measure the mushroom cap diameter, the maximum error of the shiitake mushroom cap diameter value is 14.64 % (orange), its actual diameter value is 23.03 mm, and the measured diameter value is 26.98 mm, and the size of error value is 3.95 mm. The minimum error of the shiitake mushroom cap diameter value is 0.24 % (blue), its actual diameter value is 50.14 mm, and the measured diameter value is 50.26 mm, and the size of error value is 0.12 mm. The average error of all shiitake mushroom samples is 6.24 %, with the largest error value of 7.24 mm (red). The comparison shows that the monocular camera has 3.06 percentage points less error than the binocular camera in measuring the cap diameter, and the maximum error value is 2.69 mm smaller. The above data show that the ReYOLO-MSM evaluation method combining monocular vision and the stick-mushroom growth relationship proposed in this paper has higher measurement precision than the application of a binocular camera. The proposed evaluation method only requires the use of a monocular industrial camera, which is less costly compared to the Intel Realsense D435i binocular depth camera.

For subsequent comparison and analysis of the factors affecting the measurement precision of the two methods, monocular and binocular camera measurements of the depth of the shiitake mushrooms were carried out, and the depth measured is the distance between the mushroom cap center point and the reference plane, and the process is shown in Fig. 30.

1. Download: Download high-res image (768KB)
2. Download: Download full-size image

Fig. 30. Depth measurement with two cameras.

As shown in Fig. 30, Fig. 20 images of mushroom sticks are captured using the monocular industrial camera and the D435i binocular camera, respectively, and the distance $h_s$ between the center point of each shiitake cap and the reference plane is measured one by one as the actual measurement distance $S_a$. The monocular industrial camera measures the distance $h_s$ based on the stick-shiitake mushroom growth relationship, which yields the shiitake mushroom growth height $h_1$ and the shiitake mushroom root height $h_2$, respectively, and thus sums up to the distance $h_s$, which is measured in this way is denoted as $S_m$. The distance $h_s$ measured by the D435i binocular camera is derived by subtracting the camera-measured distance S from the shiitake mushroom cap from the known height H. The distance measured in this manner is denoted as $S_d$. The distance measurement error of the monocular industrial camera and the distance measurement error of the D435i binocular camera are calculated by Eqs. (26), (27), respectively, and statistically presented in Fig. 29.(26)$e_m=S_m-S_a$(27)$e_d=S_d-S_a$Where $e_m$ is the distance measurement error of the monocular industrial camera, $S_m$ is the distance measured by the monocular industrial camera, $e_d$ is the distance measurement error of the D435i binocular camera, $S_d$ is the distance measured by the D435i binocular camera, and $S_a$ is the actual measured distance.

The errors in measuring the shiitake mushroom depth by the two cameras have been obtained above, and the reasons for the low precision of applying the binocular camera to measure the mushroom cap diameter will be discussed below from two aspects (resolution size and mushroom depth error). On the one hand, the reason is as follows. The camera field of view was determined by the size of the actual object to be detected, and was determined to be 550 mm based on the average length of the mushroom stick of 397.65 mm derived in section 2.1, where only the lateral direction is discussed. When the camera field of view is determined, the height of the camera is determined based on the camera field of view. In the case of a camera field of view of 550 mm, the height of the monocular industrial camera is 530 mm, and the height of the D435i binocular camera is 380 mm. Pixel accuracy represents the distance a pixel represents in the real world, i.e., the actual physical size of a pixel, and is determined by the size of the camera field of view and the camera resolution, which can be derived from Eq. (28):(28)$\mathit{PA}=\frac{\mathit{CV}}{\mathit{CR}}$Where, PA is the pixel accuracy (mm / pix), CV is the camera field of view (mm), and CR is the camera resolution (pix). The monocular industrial camera has a horizontal resolution of 3840 pix and the D435i binocular camera has a horizontal resolution of 1280 pix. Calculated from Eq. (10), the pixel accuracy of the monocular industrial camera is 0.143 mm / pix, and the pixel accuracy of the D435i binocular camera is 0.430 mm / pix. Set the detection target diameter of 50 mm, its pixel width in the monocular industrial camera imaging is 349.65 pix, in the D435i binocular camera imaging pixel width of 116.28 pix. By reviewing the D435i binocular camera parameters and manually testing its accuracy, in the closer distance, the D435i binocular camera's distance measurement error is generally in the range of 0.02 m-0.03 m. In the calculation, the height error is set to $\pm 30$ mm, and the pixel width of the detection target diameter corresponding to the upper and lower errors of the two cameras can be calculated according to Eq. (29), which is shown below:(29)$\frac{\mathit{OPW}}{\mathit{OD}}=\frac{\mathit{NPW}}{\mathit{ND}}$Where, OPW is the original pixel width of the detection target diameter (349.65 pix / 116.28 pix), OD is the original distance (530 mm / 380 mm), NPW is the new pixel width of the detection target diameter and ND is the new distance. With a height error of $\pm 30$ mm, the new distances corresponding to the monocular industrial camera are 560 mm and 500 mm, respectively, and the corresponding new pixel widths of the detected target diameters are 369.442 pix and 329.858 pix, respectively; the new distances corresponding to the D435i binocular camera are 410 mm and 350 mm, respectively, and the corresponding new pixel widths of the detected target diameters are 125.46 pix and 107.1 pix. The actual width of the detected target diameter corresponding to the upper and lower height errors of the two cameras can be calculated according to Eq. (30), which is shown below:(30)$\mathit{AW}=PA\times NPW$Where, AW is the actual width of the detection target diameter, PA is the pixel accuracy (mm / pix), and NPW is the new pixel width of the detection target diameter. The actual width of the detection target diameter corresponding to the up and down error of the monocular industrial camera is 50 ± 2.83 mm, and the actual width of the detection target diameter corresponding to the up and down error of the D435i binocular camera is 50 ± 3.95 mm. From the above calculations, it can be seen that the measurement accuracy of the high-resolution monocular industrial camera is better than that of the D435i binocular camera when the shiitake mushroom height error is the same.

On the other hand, the reason is as follows. Fig. 31 shows the mushroom depth error of the monocular industrial camera and the D435i binocular camera. From Fig. 31a, it can be seen that the mushroom depth error using the monocular industrial camera in combination with the stick-mushroom growth relationship ranged from 5-20 mm, with an average error value of 12.62 mm. Fig. 31b shows that the mushroom depth error using the D435i binocular camera ranged from 15-30 mm, with an average error value of 23.18 mm. Setting the detection target diameter of 50 mm, the actual width of the detection target diameter corresponding to the ± 12.62 mm height error of the monocular industrial camera is 50 ± 1.19 mm, and the actual width of the detection target diameter corresponding to the ± 23.18 mm height error of the D435i binocular camera is 50 ± 3.05 mm, according to Eqs. (28), 29, and 30 can be calculated, respectively. From the above results, it can be seen that the mushroom depth error using the monocular camera and combining the stick-mushroom growth relationship is less than that using the D435i binocular camera, and similarly, the cap diameter measurement precision combining monocular vision and the stick-mushroom growth relationship is superior to that using the D435i binocular camera.

1. Download: Download high-res image (516KB)
2. Download: Download full-size image

Fig. 31. Distance measurement error. (a) Distance measurement error of monocular industrial camera; (b) Distance measurement error of the D435i binocular camera.

In addition, the application scenario of shiitake mushroom cap diameter measurement is small, and the shiitake mushroom cap is also small, the D435i binocular camera is closer to the target, and affected by its own accuracy, the depth information it collects is inaccurate, which affects the accuracy of measuring the shiitake mushroom cap diameter. In contrast, the method proposed in this paper is based on the stick-mushroom growth relationship to estimate the distance between the shiitake mushroom cap and the reference plane, and this growth relationship is obtained by counting and summarizing a large number of stick-mushroom, and most of the shiitake mushrooms are in line with the stick-mushroom growth relationship, so that the method proposed in this paper estimates the distance between the shiitake mushroom cap and the reference plane more accurately, which has high accuracy for the measurement of the shiitake mushroom cap diameter. In the application scenario of detecting mushroom sticks on the conveyor line where information is relatively structured, the high-definition monocular vision measurement is more accurate and cost-effective.

4. Conclusion and future work

This paper proposes a ReYOLO-MSM evaluation method for the mushroom stick selective harvesting evaluation by combining monocular vision and the stick-mushroom growth relationship. A novel selective harvesting of mushroom sticks mode is presented, and the evaluation of harvesting metrics is released. A ReYOLO model with the Rotated ellipsoid frame is improved to detect caps of shiitake mushroom on sticks, and obtain the mushroom number. The mushroom cap size and roundness is obtained by geometric derivation according to statistical regression of the stick-mushroom relationship and growth features of shiitake mushrooms, and then to obtain qualified mushrooms ratio, thus evaluating the status of the sticks. Finally, the proposed method was experimented on the designed mushroom sticks selective harvester for detection performance, measurement precision and selective harvesting evaluation. The main conclusions are as follows:

• (1)

  The improved ReYOLO model achieved rotating elliptical box target detection with the detection accuracy of 99.3 %, the mAP reached 98.9 %, and the average detection speed of a single frame was 7.1 ms, which outperformed the original YOLOv5 model.

• (2)

  The proposed ReYOLO-MSM method showed that the maximum error value was only 4.55 mm for mushroom cap size measurement, with maximum error 13.15 % and average error 2.18 %.

• (3)

  The evaluation results of the ReYOLO-MSM method for selective harvesting of mushroom sticks were generally consistent with the manual evaluation, which proofed the feasibility of proposed method to be applied to mushroom stick selective harvester in factory production of shiitake mushrooms.

For future work, we will further explore this research, store the obtained information on the mushroom stick cultivation status, and the number and size of shiitake mushrooms, and correlate it with the environmental parameters in the mushroom house at that time, which can be used to explore the impact of environmental parameters on mushroom growth, and help producers to better manage mushroom houses to increase production and improve profitability.

CRediT authorship contribution statement

Kai Tao: Writing – review & editing, Writing – original draft, Validation, Methodology, Investigation, Data curation. Jian Liu: Validation, Methodology, Investigation, Data curation. Zinuo Wang: Validation, Methodology, Investigation, Data curation. Jin Yuan: Writing – review & editing, Supervision, Funding acquisition, Conceptualization. Lin Liu: Writing – review & editing, Investigation, Conceptualization. Xuemei Liu: Writing – review & editing, Resources, Project administration, Funding acquisition.