Elsevier

Computers and Electronics in Agriculture
计算机与电子在农业中

Volume 227, Part 2, December 2024, 109554
2024 年 12 月,第 227 卷,第 2 部分,109554
Computers and Electronics in Agriculture

Efficient three-dimensional reconstruction and skeleton extraction for intelligent pruning of fruit trees
高效的三维重建和果树骨架提取,以实现智能修剪

https://doi.org/10.1016/j.compag.2024.109554Get rights and content 获取版权和内容

Highlights 亮点

  • Propose a fast 3D reconstruction method for fruit trees.
    提出一种快速的水果树三维重建方法。
  • Investigate a rapid trunk segmentation and skeleton extraction method.
    研究一种快速树干分割和骨架提取方法。
  • Support for the development of user-oriented equipment for fruit tree pruning.
    支持用户导向的果树修剪设备开发。

Abstract 摘要

The three-dimensional reconstruction of fruit trees plays a crucial role in assessing their growth status, analyzing agronomic traits, and categorizing their organs. This is vital for implementing intelligent orchard management. This study aims to develop a cost-effective and efficient method for the three-dimensional reconstruction and skeleton extraction of fruit trees. The proposed method leverages the 3D geometric structure captured by Time-of-Flight (TOF) sensors and addresses common issues such as occlusion and perspective ambiguity. Firstly, the TOF sensor and its supporting components are used to build an acquisition platform to collect the full range point cloud of fruit trees in the key growth period. The noise information is filtered through the point cloud preprocessing operation to obtain the complete target point cloud and extract its structural invariant features. The IWOA-RANSAC-NDT algorithm is introduced for 3D model registration. Secondly, the Delaunay triangulation algorithm and Dijkstra shortest path algorithm are used to calculate the Minimum Spanning Tree. Branch segmentation is expedited using the Kd-tree data structure. The Levenberg Marquardt algorithm and the cylindrical fitting method are used to obtain the full fruit tree skeleton model. Finally, taking walnut tree as the experimental object, a high-precision fruit tree point cloud model is constructed, and the actual verification is carried out based on the measured data. Findings indicate that the proposed methodology can accurately construct both 3D point cloud and skeleton models of fruit trees with accuracy deviations from the measured data remaining within 7 %. The proposed method offers valuable data and technical support for the future development of highly autonomous, practical, and user-oriented fruit tree pruning systems.
果树的三维重建在评估其生长状况、分析农艺特性和对器官进行分类中起着至关重要的作用。这对于实施智能果园管理至关重要。本研究旨在开发一种经济高效的方法,用于果树的三维重建和骨架提取。提出的方法利用了飞行时间(TOF)传感器捕获的 3D 几何结构,并解决了遮挡和透视模糊等常见问题。首先,使用 TOF 传感器及其支持组件构建一个采集平台,以收集关键生长期果树的全范围点云。通过点云预处理操作过滤噪声信息,以获得完整的目标点云并提取其结构不变特征。引入 IWOA-RANSAC-NDT 算法进行 3D 模型配准。其次,使用 Delaunay 三角剖分算法和 Dijkstra 最短路径算法计算最小生成树。利用 Kd 树数据结构加速分支分割。 Levenberg-Marquardt 算法和圆柱拟合方法用于获取完整果树骨骼模型。最后,以核桃树为实验对象,构建了高精度果树点云模型,并基于测量数据进行实际验证。研究发现,所提出的方法可以准确构建果树的 3D 点云和骨骼模型,与测量数据的精度偏差保持在 7%以内。所提出的方法为未来高度自主、实用和用户导向的果树修剪系统的发展提供了宝贵的数据和技术支持。

Keywords 关键词

Fruit trees
Computer vision
3D reconstruction
Skeleton extraction

果树 计算机视觉 三维重建 骨架提取

1. Introduction 1. 引言

Xinjiang walnut is a famous fruit tree in the world, with high economic, nutritional and medicinal value (Nguyen and Vu, 2023). Pruning, an essential operation in walnut production and management, is vital for constructing high-yield tree type and producing high-quality fruit. However, traditional pruning methods are labor-intensive, costly, and predominantly reliant on human expertise entails high labor intensity, and incurs significant pruning costs. Furthermore, the selection of pruning methods predominantly relies on human experience, lacking a comprehensive intelligent solution (Zahid et al., 2022). To reduce labor costs and enhance pruning efficiency, researchers worldwide have investigated intelligent and labor-saving pruning related technologies (Giang and Ryoo, 2023, Zeng et al., 2022). Pioneering studies have focused on using advanced technology to model and measure a single fruit tree's three-dimensional structure in real-time, estimate its three-dimensional spatial structure information, and accurately identify the distribution of primary branches is a crucial preliminary stage in the intelligent pruning process (Zahid et al., 2021).
新疆核桃是世界著名的果树,具有很高的经济、营养和药用价值(Nguyen 和 Vu,2023)。修剪是核桃生产和管理的必要操作,对于构建高产树型和生产高质量果实至关重要。然而,传统的修剪方法劳动强度大、成本高,主要依赖人工经验,劳动强度大,修剪成本高。此外,修剪方法的选择主要依赖人工经验,缺乏全面智能的解决方案(Zahid 等人,2022)。为了降低劳动成本和提高修剪效率,全球研究人员已研究了智能和省力修剪相关技术(Giang 和 Ryoo,2023,Zeng 等人,2022)。开创性研究集中在利用先进技术实时建模和测量单棵果树的立体结构,估算其立体空间结构信息,并在智能修剪过程中准确识别主枝分布,这是智能修剪过程中的关键初步阶段(Zahid 等人,2021)。
Numerous studies have been conducted on the three-dimensional reconstruction of fruit trees based on high-precision sensors, Lindenmayer systems and multi-view images (Itakura et al., 2019, Okura, 2022). At present, common high-precision sensors are used to directly obtain 3D point cloud models of fruit trees, such as laser scanners (Sun et al., 2021), depth sensors (Fu et al., 2020, Yang et al., 2019), and structured light (Li et al., 2020). However, numerous factors in the agricultural application environment, such as large amount of point cloud data, diverse types of reconstruction targets, complex working environments, and sensor interference, restrict the practical application of related research.
众多研究已针对基于高精度传感器、林德迈耶系统和多视角图像的果树三维重建进行了探讨(Itakura 等,2019,Okura,2022)。目前,常用的高精度传感器可直接获取果树的三维点云模型,如激光扫描仪(Sun 等,2021)、深度传感器(Fu 等,2020,Yang 等,2019)和结构光(Li 等,2020)。然而,农业应用环境中存在大量因素,如大量点云数据、多样化的重建目标、复杂的工作环境和传感器干扰,这些都限制了相关研究的实际应用。
With the advancement of computer vision and artificial intelligence technologies, many issues encountered in the reconstruction process, such as low registration efficiency of point cloud data, poor quality of 3D reconstruction models, and high hardware configuration requirements, have been significantly improved. Achieving high-precision three-dimensional reconstruction through obtaining multi-angle point cloud data and matching point cloud families based on feature points offers advantages of low cost, high efficiency, and high precision. Key feature point extraction algorithms, including SIFT (Lowe, 2004), SURF (Viola and Jones), FAST (Maier et al.), and ORB (Rosten and Drummond) can be used to find and match the object's key feature points, map the spatial point cloud position, and estimate the camera position. Adjacent point cloud registration-related algorithms, such as RANSAC (Li et al., 2017), bundle adjustment (Triggs et al.) and ICP (Besl and McKay, 1992), can be used for adjacent point cloud registration. In the actual three-dimensional reconstruction of plants, the use of TOF sensors enables the acquisition of high-density and high-precision three-dimensional point cloud data for crops such as soybeans, tomatoes, and corn. This facilitates a more detailed and comprehensive analysis of crop structures and growth dynamics (Wang et al., 2022, Zhang et al., 2022, Zhu et al., 2023). For trees, equipment such as laser radar (LiDAR) and high-precision laser scanners remain predominant, producing highly accurate three-dimensional models (Lu et al., 2020, Wu et al., 2019). Additionally, studies involving RGB-D sensors for tree reconstruction and skeleton extraction have demonstrated the method's balance between efficiency and cost, though improvements in the reconstruction process, algorithm development, and real-time performance are still necessary (Kok et al., 2023, Ma et al., 2021).
随着计算机视觉和人工智能技术的进步,重建过程中遇到的问题,如点云数据注册效率低、3D 重建模型质量差、硬件配置要求高等问题,已得到显著改善。通过获取多角度点云数据并基于特征点匹配点云族,实现高精度三维重建,具有成本低、效率高、精度高的优势。包括 SIFT(Lowe,2004)、SURF(Viola 和 Jones)、FAST(Maier 等人)和 ORB(Rosten 和 Drummond)在内的关键特征点提取算法,可用于寻找和匹配物体的关键特征点,映射空间点云位置,并估计相机位置。相邻点云注册相关算法,如 RANSAC(Li 等人,2017)、捆绑调整(Triggs 等人)和 ICP(Besl 和 McKay,1992),可用于相邻点云注册。 在实际植物的三维重建中,使用 TOF 传感器能够获取大豆、番茄和玉米等作物的密集和高精度三维点云数据。这有助于更详细和全面地分析作物结构和生长动态(王等,2022,张等,2022,朱等,2023)。对于树木,激光雷达(LiDAR)和高精度激光扫描仪等设备仍然占主导地位,能够产生高度精确的三维模型(卢等,2020,吴等,2019)。此外,涉及 RGB-D 传感器进行树木重建和骨骼提取的研究表明,该方法在效率和成本之间取得了平衡,尽管在重建过程、算法开发和实时性能方面仍需改进(科克等,2023,马等,2021)。
In summary, significant research has been conducted on the three-dimensional reconstruction and skeleton extraction of crops, with notable advancements. However, developing an autonomous pruning system suitable for complex orchard environments remains a substantial engineering challenge. Current methods include three-dimensional scanning, which offers rapid data acquisition and high fidelity but requires controlled environmental conditions, involves time-consuming data processing, and incurs high costs. Laser radar techniques provide comprehensive coverage but are cumbersome, computationally intensive, and expensive. Methods based on RGB-D sensors are cost-effective, easy to operate, and perform well in real-time but suffer from issues related to lighting and distance, resulting in lower accuracy. These foundational algorithms require specific enhancements to meet the demands of fruit tree self-pruning tasks.
To address these challenges, targeted improvements and optimizations are essential. Developing a method that is low-cost, efficient, and sufficiently accurate is crucial. Our TOF sensor-based 3D reconstruction method for fruit trees strikes a balance between efficiency and accuracy in the automatic extraction of tree trunks. This achievement provides theoretical support and practical references for the future development of practical, commercial, and user-friendly fruit tree pruning robots.

2. Three-dimensional reconstruction and skeleton extraction principle

2.1. Multi-view data acquisition based on kinect sensor

Fruit trees possess characteristics such as large size, complex structure, and high reconstruction difficulty. This paper proposes a method for collecting initial point cloud data of fruit trees from six perspectives using a single sensor. Fig. 1 illustrates the acquisition setup. The initial point cloud data is extracted by rotating the camera counterclockwise around the tree. The angle between each pair of adjacent cameras is 60 degrees to satisfy the overlap requirements during the point cloud registration process. The process of multi-angle point cloud capture is depicted in Fig. 1(a). The camera's shooting distance from the fruit tree and the camera's height from the ground are measured using the LDM-40 laser rangefinder. Considering measurement errors, each measurement point is measured five times, and the mean value is taken as the actual value, as shown in Fig. 1(b).
  1. Download: Download high-res image (171KB)
  2. Download: Download full-size image

Fig. 1. Multi-view data acquisition schematic diagram.

2.2. Rough registration method based on IWOA-RANSAC algorithm

The extraction of feature points for point cloud registration from the unstructured environment of an orchard is complicated by the presence of substantial noise in the data, posing significant challenges. The Random Sample Consensus (RANSAC) algorithm is adept at robustly estimating model parameters and operates swiftly, facilitating the precise estimation and fitting of initial point cloud data from fruit trees. However, this algorithm is notably sensitive to the choice of the initial point. Consequently, this study employs the RANSAC algorithm for initial registration and integrates an enhanced Whale Optimization Algorithm (WOA) to expediently determine the optimal solution for the RANSAC algorithm. This combination aims to improve the accuracy and efficiency of point cloud feature extraction.

2.2.1. Random Sample Consensus (RANSAC) algorithm

The RANSAC algorithm (Derpanis, 2010) operates by randomly selecting six matching points using a normalization method, identifying the optimal transformation matrix to eliminate error matching, and solving the required unknowns. The expression T of the optimal transformation matrix is as follows:(1)xyz1=R11R12R13TxR21R22R23TyR310R320R330Tz1xyz1Where (x,y,z) and (x,y,z) are the Cartesian coordinates of two point clouds to be registered, respectively.
Due to the large volume of point cloud data, obtaining the optimal solution within the specified number of iterations often presents a challenge, and the registration accuracy may also be impacted.

2.2.2. Nonlinear constraint-based whale algorithm

The WOA (Mirjalili and Lewis, 2016) boasts a robust global optimization capability, a straightforward structure, and minimal parameter settings, enhancing the robustness and efficiency of the process. Nonetheless, the global search ability of the traditional WOA is contingent upon its convergence factor, which decreases with an increasing number of iterations. This reduction in the convergence factor lead. To address these limitations, Li et al. incorporated the concept of inertia weight from the Particle Swarm Optimization (PSO) algorithm, introducing adaptive parameters as inertia weight factors. This adaptation, referred to as the IWOA (Inertia Weighted Whale Optimization Algorithm), harnesses the optimal solution more effectively and enhances the algorithm's optimization accuracy (Li et al., 2022a). This innovative approach aims to improve the overall performance of the WOA in complex optimization scenarios.
The flowchart of the algorithm is as shown in Fig. 2:
  1. Download: Download high-res image (248KB)
  2. Download: Download full-size image

Fig. 2. Flow chart of preliminary registration.

Among them, t represents the current time of the iteration, Tmax represents the maximum iteration time, A is the optimization parameter, f(t) represents the fitting function, which is defined as follows:(2)ft=argmini=1nHl=0.5l2,l<ml0.5ml2l-ml,l>mlWhere Hl denotes the Huber loss, ml denotes the preset threshold, and l is the distance difference after the transformation of the corresponding points in the ith group.

2.3. Accurate registration method based on 3D-NDT algorithm

While rough registration can correct the initial attitude of the point cloud model, accurate registration of the point cloud is still needed to eliminate positional deviation. The 3D-NDT algorithm, derived from the non-destructive testing algorithm, performs well in point cloud fine registration (Yu et al., 2019). It is a registration algorithm that combines standard optimization and normal detection. The flowchart of the algorithm is as shown in Fig. 3.
  1. Download: Download high-res image (322KB)
  2. Download: Download full-size image

Fig. 3. Flow chart of accurate registration. Pi is any point of the fruit tree point cloud. Tβ,pi is a spatial transformation function, which represents the coordinate transformation between two clouds.

2.4. Skeleton extraction method based on Delaunay triangulation and Dijkstra algorithm

Point cloud Delaunay triangulation (Alexa, 2020) involves the triangulation of a group of scattered points in three-dimensional space based on Delaunay rules to generate an optimized spatial triangulation network, reflecting the topological structure relationship between data points and their adjacent points. In this experiment, this method is used to restore the shape of trees from discrete point clouds.
Initially, the mesh initialization of the fruit tree point cloud is conducted, involving the tetrahedron division of the point cloud. Given a known point cloud set, any triangular mesh is established and edge exchange operations are performed until all triangles satisfy the Delaunay condition, as depicted in Fig. 4.
  1. Download: Download high-res image (142KB)
  2. Download: Download full-size image

Fig. 4. Edge exchange in K2.

Subsequently, triangular network optimization. Select a point to be processed, use the point location search algorithm to find the tetrahedron containing the point in the tetrahedral data structure, connect the point with the four vertices of the tetrahedron, divide the tetrahedron into four new tetrahedrons, and control the quality of the generated new tetrahedron to ensure that it meets the requirements of Delaunay triangulation to improve the quality and stability of the mesh. Repeat the steps until the point set P is exhausted.
Finally, local optimization of the generated triangle network is performed, and the sub-triangle network is merged. In the three-dimensional point cloud space, because the Lawson local edge exchange method is not applicable, this paper optimizes each vertex or edge locally based on the tetrahedral optimization algorithm, so that the entire mesh has Delaunay properties. Some smaller triangles are merged into a larger triangle to reduce the complexity and computational complexity of the entire triangular mesh. Dijkstra's algorithm facilitates the determination of the shortest path to realize the extraction of the minimum spanning tree model.

3. Materials and methods

3.1. Walnut tree 3D reconstruction platform

The constructed 3D data acquisition platform is depicted in Fig. 5. Component ① is an application-level Kinect v2 camera, which is mainly used for the collection of basic point cloud data. Component ② is a custom-made Kinect v2 camera support device with an adjustable height ranging from approximately 50–250 cm. Component ③ is a graphics workstation equipped with an Intel i5-13490F processor operating at 2.50 GHz, 32 GB RAM memory, a 500 GB Kingston solid-state hard drive, and an NVIDIA GeForce RTX 3060 graphics card, Windows 10 operating system and UBUNTU20.04 operating system. Component ④ is an LDM-40 laser distance meter. Component ⑤ for the measurement tools, including tape and vernier caliper. Component ⑥ is a Xinjiang walnut tree with a height of 2.7 m and a crown diameter of 2.57 m, located in the walnut orchard in Kashgar, Xinjiang.
  1. Download: Download high-res image (151KB)
  2. Download: Download full-size image

Fig. 5. Three-dimensional reconstruction platform of fruit tree.

The workflow primarily includes the mapping of RGB and depth information, the control of camera shooting distance and error, the customized point cloud alignment process, the construction of the 3D model of fruit trees.
(1) Image Mapping: Initially, the intrinsic and extrinsic parameters of the Kinect camera are calibrated. Subsequently, the extracted depth image is mapped and calibrated with the RGB image (F et al., 2022) using equations (3)-(5). Finally, point cloud preprocessing is conducted (Li et al., 2022b).(3)ZrgbPrgb=KrgbRir2rgbKir-1ZirPir+KrgbTir2rgb(4)Rir2rgb=RrgbRir-1(5)Tir2rgb=Trgb-Rir2rgbTirWhere ZrgbPrgb represents the mapping of RGB camera data, ZirPir represents the mapping of near-infrared camera data, and Prgb=urgbvrgb1T and Pir=uirvir1T are the pixel coordinates of the RGB and infrared images, respectively. Krgb=fx_rgb0cx_rgb0fy_rgbcy_rgb001 and Kir=fx_ir0cx_ir0fy_ircy_ir001 are the intrinsic and extrinsic parameter matrices of the RGB and near-infrared cameras. fx_rgb and fy_rgb represent the focal lengths of the RGB camera. cx_rgb and cy_rgb are the principal point coordinates. Additionally, fx_irfy_ircx_rgbcy_rgbRrgb and Trgb are the external parameters of the RGB camera, while Rir and Tir are the external parameters of the depth camera.
(2) Camera shooting distance and error control: To ensure the accuracy of the collected data, the experiments position the Kinect v2 camera at a specific height above the ground (H) and a certain distance from the collection object tree (D). H and D are determined by combining the Kinect v2 camera's own parameters and the height of the collection object after numerous experiments. To minimize the measurement error, the LDM-40 laser rangefinder is used to calculate the average value of five measurements as the actual value. The camera height H is set at one-third of the height of the tree from the ground for optimal data acquisition.
(3) Self-Alignment of Walnut Tree Point Cloud: Initially, the Kinect v2 camera is positioned around the peach tree according to the rule shown in Fig. 1, and the point cloud data of the walnut tree are sequentially extracted from six angles at 60-degree intervals to obtain the initial walnut tree point cloud data. Based on the region of interest, these point cloud data undergo noise reduction, filtering, and extraction of the region of interest. Subsequently, the key points of adjacent point cloud blocks are extracted and their Fast Point Feature Histograms (FPFHs) are calculated. Using the optimized RANSAC and IWOA algorithms, corresponding key points are identified, automatic registration is achieved, and the transformation matrix is calculated. Experimentally, the Pseudo-Huber loss function(Wang et al.) is selected as the basis for judging the performance of coarse registration, and the optimal adjacent point cloud block transformation matrix is obtained. Based on the 3D-NDT algorithm, accurate registration of the point cloud is achieved. Finally, a complete point cloud model of fruit trees is obtained, as shown in Fig. 6.
  1. Download: Download high-res image (328KB)
  2. Download: Download full-size image

Fig. 6. Walnut Tree Point Cloud Alignment Process.

3.2. Walnut tree skeleton construction system

Skeleton extraction is a technique that creates a representation of a plant's branch and trunk structure through a skeleton or skeleton graph. In the context of walnut tree pruning, skeleton extraction holds significant value. Firstly, it aids in branch identification. Skeleton extraction allows for the clear identification of the main trunk and major branches of walnut trees, assisting in determining which parts require pruning. Secondly, it facilitates structural assessment. Skeleton extraction provides information on the overall structure of a walnut tree, including the distribution, angle, and length of branches. This data can be analyzed to assess whether the tree's morphological structure is balanced and stable. If branches are too dense or overgrown, targeted pruning can be conducted to promote light and air circulation, reduce weight burden, and prevent tree collapse in wind and rain. Lastly, it assists in pest and disease management. Skeleton extraction can help detect possible pests and diseases on walnut trees, as certain pests and diseases may lead to abnormal changes in branch morphology. By examining the skeleton map, affected sections can be detected early and pruned to curb the spread of pests and diseases. In summary, skeleton extraction offers both visual and quantitative data pertinent to walnut tree pruning, guiding decisions on where to prune to foster robust tree development and ensure structural integrity.
  • (1)
    Skeleton Self-Extraction Process and Method: To generate trunks and branches with high-precision geometric and topological structures, this study is based on the coarse-refinement alignment algorithm for fruit tree 3D model construction, and the fruit tree skeleton is extracted based on the Alternating Decision Tree (AdTree) method (Du et al., 2019). Initially, the initial fruit tree 3D model is obtained by filtering and coarse-refinement alignment based on the initial point cloud data, followed by Delaunay triangulation(Su et al., 2022) and Minimum Spanning Tree (MST) extraction, an idea originating from tree nutrient transport paths in ecology, as shown in Fig. 7. The Delaunay triangular sectioning lays the foundation for MST computation, which helps complete missing regions or incomplete branches, and ensures the robustness of input point clouds with poor data quality. After obtaining the triangulated graph, all the edges are weighted with the lengths of the edges defined in the Euclidean space, and the Dijkstra shortest path algorithm(Straub et al., 2022) is used to calculate the MST from the triangulation. The quality of the skeleton can be improved by pre-determining and centralizing the major branching points, assigning weight values to the vertices and edges, and removing small noise components based on this value. Finally, the proximity between neighboring vertices is checked by iterative proximity between vertices and merging neighboring vertices to reconstruct the lightweight fruit tree skeleton.
    1. Download: Download high-res image (394KB)
    2. Download: Download full-size image

    Fig. 7. Schematic diagram of tree water transport path.

  • (2)
    Skeleton Geometry Cylindrical Fitting Method: Based on the final fruit tree skeleton, a cylindrical fitting method is used to obtain a geometric model of the fruit tree. Using the optimization method, the exact branch geometry is obtained, the K-dimensional tree (Kd-tree) data structure is used for spatial nearest neighbor search, the branches are quickly segmented, and the Levenberg Marquardt algorithm(Hu et al., 2020) is used to solve the nonlinear least-squares problem to obtain the radii of the subsequent branches. Based on the corresponding pole points, a cylinder is fitted to approximate the branch geometry, and each cylinder that makes up the trunk or all branches can be considered as a closed convex packet polyhedron fit with convex polygons on the top and bottom surfaces of the polyhedron.

3.3. Data acquisition and experimental design

The experimental data were collected from a walnut orchard in Kashgar, Xinjiang, situated in a warm temperate continental arid climate zone. The orchard featured open-center shaped walnut trees, each approximately 2.7 m in height with a crown diameter of around 2.57 m and a trunk diameter up to 0.13 m. The trees were spaced 5 m apart in rows, with a 4-meter spacing between individual plants. Data collection commenced in July 2022 and concluded in December 2022.
During the data collection process, measurements were conducted at noon under sunny conditions to minimize sunlight interference with the sensor's point cloud collection. Additionally, low wind speeds were maintained to prevent disturbances to the trees and leaves. The necessary RGB and depth data of the walnut tree for this research were gathered following the methodology presented in Fig. 1, using the devices ①, ②, ③, ④ and ⑤ shown in Fig. 5. Simultaneously, the height of the walnut tree and the diameter of the five points shown in Fig. 8. were collected as measured data. To minimize the measurement error, the data for each actual measurement point were measured seven times and obtained by the 3σ method for statistics.
  1. Download: Download high-res image (347KB)
  2. Download: Download full-size image

Fig. 8. Measured data collection points.

The experiment involves extracting point cloud data of a walnut tree through sensors and computers and constructing a three-dimensional model and skeleton model of the fruit tree. The aim is to provide technical support for intelligent pruning by pruning robots. Due to the process of collecting point cloud data, alignment errors and cumulative errors can occur, considering the need for multiple collections and alignments of point cloud data. Therefore, this study adopts the method of multiple reconstructions to reduce the error. The accuracy of the fruit tree reconstruction process can be assessed by comparing the reconstructed 3D model branch data with the separately measured walnut tree branch data. The experimental setup guarantees precise alignment of the fruit trees' multi-angle point clouds and validates the precision of the 3D point cloud models.

4. Results

4.1. Establishment of walnut tree point cloud model and accuracy analysis

4.1.1. Point cloud modeling of walnut trees

This study realized a 3D reconstruction model and phenotypic data measurement of a walnut tree with a height of 2.70 m and a crown diameter of 2.57 m, as depicted in Fig. 9. The multi-angle point cloud was acquired by a Kinect sensor, positioned around the walnut tree at 60-degree intervals for one week. The time interval for extracting this complete point cloud data was measured to be between 8 and 10 min, which is also the time required to acquire the 3D image of the walnut tree. The time interval for the denoising and alignment process of the point cloud slices was between 3 and 5 min.
  1. Download: Download high-res image (851KB)
  2. Download: Download full-size image

Fig. 9. Walnut tree skeleton extraction results.

Fig. 9(a)① displays the obtained 3D reconstruction model of the walnut tree. For the point cloud slices captured from multiple angles, the data quality varies due to differences in the acquisition viewpoints, which is partly the source of the error of the reconstructed 3D point cloud model. During the alignment process, considering that the cumulative error can significantly affect the accuracy of the walnut tree model, this study adopts the alignment of two neighboring point cloud slices to reduce the cumulative error.
Fig. 9(a) ② shows the preliminary classification results of the upper and lower parts of the walnut tree model from the Scalar fields (SF) data acquired by the sensor. As shown in the fig., the SF value of around 0 is the threshold for the upper and lower parts, which can provide support for the future extraction of the main trunk portion of the walnut tree.
Fig. 9(b) displays the effect of Delaunay triangulation of the walnut tree. Where ① is the aligned 3D model of the walnut tree, consisting of a total of 164,520 data points. ② is the surface normal of all the above points calculated, totaling 164520. ③ is the Delaunay triangulation operation for all points in the 3D model, resulting in a total of 2,192,821 surfaces. ④ is a schematic diagram that preserves the key triangular mesh, with a total of 29,886 surfaces.
Fig. 9(c) shows the minimum mesh model based on Delaunay triangulation and the complete point cloud model after filling the missing point cloud. Among them, ① is the minimum mesh model based on Delaunay triangulation, and ② and ③ are the local zoom effects of the mesh model, demonstrating that the method is effective in filling the missing point cloud and the obtained mesh model can meet the requirements of extracting the complete point cloud model. ④ is the complete point cloud model extracted from the mesh model, which has the characteristics of smaller data volume and complete model information compared to the initial point cloud model Fig. 10(a)①, which is conducive to the subsequent extraction of the point cloud skeleton. ⑤⑥ are localized images of model ④.
  1. Download: Download high-res image (659KB)
  2. Download: Download full-size image

Fig. 10. Walnut tree skeleton extraction results.

The time interval used for the Delaunay triangulation calculation of the above walnut tree model, the construction of the grid model Fig. 9(c)①, and the extraction process of the final point cloud model Fig. 9(c)④ was between 1 and 2 min.

4.1.2. Modeling of the walnut tree skeleton

Based on the extracted point cloud model, the walnut tree skeleton is constructed, as depicted in Fig. 10. Fig. 10①-④ demonstrate the process of determining the optimal skeleton by employing 1000 randomly chosen skeleton points.. Based on the concept of tree water transport, such as Fig. 7, combined with the shortest path algorithms, the initial distribution of skeleton points is obtained (Fig. 10⑤). Subsequently, the point cloud skeleton is expanded and optimized based on the final point cloud model in Fig. 9(c)④. Further iterations are performed to examine the proximity between adjacent vertices, and neighboring vertices are merged to reconstruct a lightweight tree skeleton. The derived point cloud skeleton is depicted in Fig. 10⑥ and Fig. 10⑦.
Fig. 10⑧ represents the final fitted cylindrical skeleton model of the walnut tree. Based on the simplified skeleton, a cylindrical fitting method is employed to obtain the geometric model of the tree. Each cylindrical component representing a trunk or branch can be considered as a closed convex hull polyhedron. The top and bottom faces of the polyhedron are convex polygons. The reconstructed model is shown in Fig. 10⑧, while Fig. 10⑨ and ⑩ provide local detail enlarged graphs of the model.

4.2. Experimental result analysis

4.2.1. Point cloud registration accuracy analysis

The impact of the IWOA algorithm on the performance of the RANSAC-NDT algorithm was investigated, with a focus on recording its execution time. These experiments utilized data from fruit trees and were conducted on the graphics workstation depicted in Fig. 5. The implementation was carried out in C++ language. Quantitative analysis involved calculating the average Euclidean distance between matching points of two point cloud groups after precise registration. Ideally, the Euclidean distance should be zero when the point clouds are perfectly matched. This study compares the mean registration error and processing time between the RANSAC-NDT algorithm and the IWOA-enhanced RANSAC-NDT algorithm. As shown in Table 1, the addition of the IWOA algorithm did not significantly alter the accuracy of point cloud registration, maintaining it at 0.0152 m, but it notably reduced the registration time from 25.2 s to 18.1 s, demonstrating the efficiency of the IWOA algorithm.

Table 1. Comparison of RANSAC-NDT with IWOA-RANSAC-NDT.

Registration MethodAmount of point cloudRegistration time/sRegistration error/m
IWOA-RANSAC-NDT69,56318.10.0152
RANSAC-NDT69,56325.20.0153
The proposed IWOA-RANSAC-NDT algorithm is compared with the classical ICP algorithm, RANSAC algorithm, and Depth Filtering-ICP algorithm in terms of average registration distance error and registration processing time. The comparative results are presented in Table 2. The findings indicate that the IWOA-RANSAC-NDT algorithm achieves the highest registration accuracy with an average registration distance error of 0.0152 m. Furthermore, the registration time is significantly reduced compared to the other methods while maintaining accuracy.

Table 2. Comparison of different point cloud registration algorithms.

Registration MethodAmount of point cloudRegistration time/sRegistration error/m
ICP69,56314.70.0825
Depth Filtering-ICP69,56325.80.0313
IWOA-RANSAC-NDT69,56318.10.0152

4.2.2. Experimental analysis of skeleton extraction

The skeleton of fruit trees is extracted based on the constructed three-dimensional model, utilizing core algorithms such as 3D-DT (3D Delaunay Triangulation), DSP (Dijkstra's Shortest Path) analysis, and LM-CF (Levenberg-Marquardt Cylindrical Fitting). This experiment was conducted on the graphics workstation depicted in Fig. 5, with the code implemented in C++ language. Table 3 documents the time required for each key algorithm involved in skeleton extraction. The results indicate that the 3D-DT operation takes approximately 3.35 s to generate the triangular mesh of fruit trees, the DSP operation requires about 5.58 s to compute the minimum spanning tree, and the LM-CF operation concludes in about 2.23 s. These findings demonstrate that the skeleton extraction algorithm effectively balances computational efficiency and algorithmic complexity.

Table 3. Skeleton extraction algorithm.

MethodAmount of point cloudAmount of facesRegistration time/s
3D-DT164,5202,192,8213.35
DSP164,520298,8605.58
LM-CF164,520298,8602.23

4.3. Precision analysis of fruit tree branch reconstruction

To further enhance the usability of our research model and validate its accuracy, a method for calculating the diameter of cylindrical branches and tree height was designed. In the point cloud model, coordinate transformations were employed to obtain tree height data and diameter data for key branches.
To mitigate the impact of data extraction, point cloud registration, and modeling processes on accuracy, this study reconstructed the fruit tree point cloud model ten times. The diameters of five key branches and tree height were statistically analyzed in comparison with the values from the reconstructed models to assess their accuracy, as shown in Table 4.

Table 4. Statistical Tables for Tree Height and Diameter Data.

ParameterDiameter of branch ①/mmDiameter of branch ②/mmDiameter of branch ③/mmDiameter of branch ④/mmDiameter of branch ⑤/mmtree height/m
The real value121.6993.3548.4945.2346.822693.00
Reconstructed model measurements1126.8596.8147.1247.3649.122764.78
Reconstructed model measurements2116.0689.5446.9442.8243.072583.12
Reconstructed model measurements3124.6296.4352.1948.9545.642741.09
Reconstructed model measurements4118.7888.6746.5243.0945.362609.25
Reconstructed model measurements5119.3791.1847.7243.6144.712630.66
Reconstructed model measurements6125.5495.5848.3343.2950.092793.87
Reconstructed model measurements7118.2990.7651.6441.8542.782755.03
Reconstructed model measurements8126.1197.8146.6347.3147.922765.41
Reconstructed model measurements9127.2596.1252.9849.4549.832813.34
Reconstructed model measurements10116.3289.2446.8141.0343.172594.91
Average reconstructed value122.1592.6848.8045.3447.542702.43
Average deviation of reconstruction4.163.342.072.782.5982.94
The average error of reconstruction3.42 %3.58 %4.27 %6.15 %5.54 %3.08 %
According to the table above, after conducting ten reconstructions and calculating the mean values, the average deviations of the diameter and height of the five key branches were 4.16 mm, 3.34 mm, 2.07 mm, 2.78 mm, 2.59 mm, and 82.94 mm, respectively. The average errors were 3.42 %, 3.58 %, 4.27 %, 6.15 %, 5.54 %, and 3.08 %, respectively. This demonstrates that the quality of diameter and tree height parameters extracted based on the Kinect v2 sensor has good accuracy. It indicates that the diameter and tree height parameters based on the sensor reconstruction model are within the error tolerance range, which can meet the requirements of orchard measurement and pruning.

5. Discussion

5.1. Three-dimensional reconstruction system

Three-dimensional reconstruction is a crucial aspect of intelligent pruning of fruit trees. This study is based on the extraction and registration of multi-angle point cloud data. The registration process, which involves coarse registration and fine registration based on feature points, takes about 3 min and does not meet the needs of real-time three-dimensional reconstruction. To address this issue, the real-time 3D dense reconstruction algorithm could be considered to replace the current point cloud registration method, thereby achieving real-time 3D reconstruction. To address issues related to the quality of crown reconstruction, some existing studies have proposed new research ideas to tackle these challenges (Liang et al., 2022, Yao et al., 2021).

5.2. Design and analysis of experiments

This experiment is conducted based on collected multi-angle point cloud data of fruit trees. The data undergoes pre-processing and alignment to enable 3D reconstruction and skeleton extraction, thereby constructing a 3D reconstruction and skeleton extraction system for fruit trees. Considering the unstructured environment of the orchard, multiple acquisitions are employed for initial data acquisition to mitigate the accumulation of initial errors. The established initial point cloud data processing, 3D reconstruction, and skeleton extraction system serve as a reference for the development of vision models for smart orchards and pruning robots.
However, this experiment still has certain limitations. The process of initial target point cloud extraction in unstructured environments using sensors still requires further exploration. The methods employed in the data acquisition, 3D model reconstruction, and skeleton extraction processes still consume a significant amount of time, severely restricting the operational efficiency of the system and the real-time reconstruction requirements for fruit trees. This remains the primary issue to be addressed in future research.

6. Conclusion

The three-dimensional model of fruit trees is of great significance for the study of intelligent pruning of fruit trees. Our work demonstrates that it is possible to reconstruct the three-dimensional model of fruit trees with low cost and high efficiency based on a depth camera. In this paper, we designed the extraction process of RGB data and depth data of fruit trees and the self-registration and skeleton self-extraction system of point cloud data of fruit trees. The system can obtain a three-dimensional model and skeleton model of fruit trees and perform phenotypic structure measurement of fruit trees, providing support for intelligent pruning of fruit trees. Finally, a field experiment was carried out in a walnut industrial park. The experimental results show that the proposed three-dimensional reconstruction and skeleton extraction methods are reliable, meet the reconstruction error requirements of fruit tree intelligent pruning, and the reconstruction model can provide a reference for digital orchards.
Future work will continue to study the method of real-time 3D reconstruction, with a focus on improving the efficiency of 3D reconstruction and skeleton extraction. The goal is to provide technical support for robotic pruning systems and sophisticated orchard management. This could potentially involve exploring new algorithms, utilizing more advanced sensors, or optimizing the current processes. Additionally, further research could also look into improving the accuracy and robustness of the system in different environmental conditions.

CRediT authorship contribution statement

Xiaojuan Li: Funding acquisition, Investigation, Project administration, Supervision, Writing – review & editing, Conceptualization. Bo Liu: Conceptualization, Investigation, Software, Visualization, Writing – original draft, Data curation. Yinggang Shi: Data curation, Supervision. Mingming Xiong: Data curation, Supervision. Dongyu Ren: Conceptualization, Investigation, Software. Letian Wu: Data curation, Investigation. Xiangjun Zou: Project administration, Writing – review & editing, Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This research was funded by the Xinjiang Uygur Autonomous Region project ' Research on visual perception and adaptive control system of humanoid picking robot ' (20227SYCCX0061), the National Natural Science Foundation of China project ' Research on bionic vision and adaptive grasping method of Xinjiang long-staple cotton picking robot ' (52265003) and the Xinjiang Uygur Autonomous Region major project ' Research on automatic and intelligent mechanical equipment for vegetables around Tarim Basin ' (2022A02005-5).

Data availability

Data will be made available on request.

References

Cited by (0)

View Abstract