Efficient three-dimensional reconstruction and skeleton extraction for intelligent pruning of fruit trees

doi:10.1016/j.compag.2024.109554

Computers and Electronics in Agriculture
计算机与电子在农业中

Volume 227, Part 2, December 2024, 109554
2024 年 12 月，第 227 卷，第 2 部分，109554

https://doi.org/10.1016/j.compag.2024.109554 Get rights and content 获取版权和内容

Highlights 亮点

•
Propose a fast 3D reconstruction method for fruit trees.
提出一种快速的水果树三维重建方法。
•
Investigate a rapid trunk segmentation and skeleton extraction method.
研究一种快速树干分割和骨架提取方法。
•
Support for the development of user-oriented equipment for fruit tree pruning.
支持用户导向的果树修剪设备开发。

Abstract 摘要

The three-dimensional reconstruction of fruit trees plays a crucial role in assessing their growth status, analyzing agronomic traits, and categorizing their organs. This is vital for implementing intelligent orchard management. This study aims to develop a cost-effective and efficient method for the three-dimensional reconstruction and skeleton extraction of fruit trees. The proposed method leverages the 3D geometric structure captured by Time-of-Flight (TOF) sensors and addresses common issues such as occlusion and perspective ambiguity. Firstly, the TOF sensor and its supporting components are used to build an acquisition platform to collect the full range point cloud of fruit trees in the key growth period. The noise information is filtered through the point cloud preprocessing operation to obtain the complete target point cloud and extract its structural invariant features. The IWOA-RANSAC-NDT algorithm is introduced for 3D model registration. Secondly, the Delaunay triangulation algorithm and Dijkstra shortest path algorithm are used to calculate the Minimum Spanning Tree. Branch segmentation is expedited using the Kd-tree data structure. The Levenberg Marquardt algorithm and the cylindrical fitting method are used to obtain the full fruit tree skeleton model. Finally, taking walnut tree as the experimental object, a high-precision fruit tree point cloud model is constructed, and the actual verification is carried out based on the measured data. Findings indicate that the proposed methodology can accurately construct both 3D point cloud and skeleton models of fruit trees with accuracy deviations from the measured data remaining within 7 %. The proposed method offers valuable data and technical support for the future development of highly autonomous, practical, and user-oriented fruit tree pruning systems.
果树的三维重建在评估其生长状况、分析农艺特性和对器官进行分类中起着至关重要的作用。这对于实施智能果园管理至关重要。本研究旨在开发一种经济高效的方法，用于果树的三维重建和骨架提取。提出的方法利用了飞行时间（TOF）传感器捕获的 3D 几何结构，并解决了遮挡和透视模糊等常见问题。首先，使用 TOF 传感器及其支持组件构建一个采集平台，以收集关键生长期果树的全范围点云。通过点云预处理操作过滤噪声信息，以获得完整的目标点云并提取其结构不变特征。引入 IWOA-RANSAC-NDT 算法进行 3D 模型配准。其次，使用 Delaunay 三角剖分算法和 Dijkstra 最短路径算法计算最小生成树。利用 Kd 树数据结构加速分支分割。 Levenberg-Marquardt 算法和圆柱拟合方法用于获取完整果树骨骼模型。最后，以核桃树为实验对象，构建了高精度果树点云模型，并基于测量数据进行实际验证。研究发现，所提出的方法可以准确构建果树的 3D 点云和骨骼模型，与测量数据的精度偏差保持在 7%以内。所提出的方法为未来高度自主、实用和用户导向的果树修剪系统的发展提供了宝贵的数据和技术支持。

Keywords 关键词

Fruit trees

Computer vision

3D reconstruction

Skeleton extraction

果树计算机视觉三维重建骨架提取

1. Introduction 1. 引言

Xinjiang walnut is a famous fruit tree in the world, with high economic, nutritional and medicinal value (Nguyen and Vu, 2023). Pruning, an essential operation in walnut production and management, is vital for constructing high-yield tree type and producing high-quality fruit. However, traditional pruning methods are labor-intensive, costly, and predominantly reliant on human expertise entails high labor intensity, and incurs significant pruning costs. Furthermore, the selection of pruning methods predominantly relies on human experience, lacking a comprehensive intelligent solution (Zahid et al., 2022). To reduce labor costs and enhance pruning efficiency, researchers worldwide have investigated intelligent and labor-saving pruning related technologies (Giang and Ryoo, 2023, Zeng et al., 2022). Pioneering studies have focused on using advanced technology to model and measure a single fruit tree's three-dimensional structure in real-time, estimate its three-dimensional spatial structure information, and accurately identify the distribution of primary branches is a crucial preliminary stage in the intelligent pruning process (Zahid et al., 2021).
新疆核桃是世界著名的果树，具有很高的经济、营养和药用价值（Nguyen 和 Vu，2023）。修剪是核桃生产和管理的必要操作，对于构建高产树型和生产高质量果实至关重要。然而，传统的修剪方法劳动强度大、成本高，主要依赖人工经验，劳动强度大，修剪成本高。此外，修剪方法的选择主要依赖人工经验，缺乏全面智能的解决方案（Zahid 等人，2022）。为了降低劳动成本和提高修剪效率，全球研究人员已研究了智能和省力修剪相关技术（Giang 和 Ryoo，2023，Zeng 等人，2022）。开创性研究集中在利用先进技术实时建模和测量单棵果树的立体结构，估算其立体空间结构信息，并在智能修剪过程中准确识别主枝分布，这是智能修剪过程中的关键初步阶段（Zahid 等人，2021）。

Numerous studies have been conducted on the three-dimensional reconstruction of fruit trees based on high-precision sensors, Lindenmayer systems and multi-view images (Itakura et al., 2019, Okura, 2022). At present, common high-precision sensors are used to directly obtain 3D point cloud models of fruit trees, such as laser scanners (Sun et al., 2021), depth sensors (Fu et al., 2020, Yang et al., 2019), and structured light (Li et al., 2020). However, numerous factors in the agricultural application environment, such as large amount of point cloud data, diverse types of reconstruction targets, complex working environments, and sensor interference, restrict the practical application of related research.
众多研究已针对基于高精度传感器、林德迈耶系统和多视角图像的果树三维重建进行了探讨（Itakura 等，2019，Okura，2022）。目前，常用的高精度传感器可直接获取果树的三维点云模型，如激光扫描仪（Sun 等，2021）、深度传感器（Fu 等，2020，Yang 等，2019）和结构光（Li 等，2020）。然而，农业应用环境中存在大量因素，如大量点云数据、多样化的重建目标、复杂的工作环境和传感器干扰，这些都限制了相关研究的实际应用。

With the advancement of computer vision and artificial intelligence technologies, many issues encountered in the reconstruction process, such as low registration efficiency of point cloud data, poor quality of 3D reconstruction models, and high hardware configuration requirements, have been significantly improved. Achieving high-precision three-dimensional reconstruction through obtaining multi-angle point cloud data and matching point cloud families based on feature points offers advantages of low cost, high efficiency, and high precision. Key feature point extraction algorithms, including SIFT (Lowe, 2004), SURF (Viola and Jones), FAST (Maier et al.), and ORB (Rosten and Drummond) can be used to find and match the object's key feature points, map the spatial point cloud position, and estimate the camera position. Adjacent point cloud registration-related algorithms, such as RANSAC (Li et al., 2017), bundle adjustment (Triggs et al.) and ICP (Besl and McKay, 1992), can be used for adjacent point cloud registration. In the actual three-dimensional reconstruction of plants, the use of TOF sensors enables the acquisition of high-density and high-precision three-dimensional point cloud data for crops such as soybeans, tomatoes, and corn. This facilitates a more detailed and comprehensive analysis of crop structures and growth dynamics (Wang et al., 2022, Zhang et al., 2022, Zhu et al., 2023). For trees, equipment such as laser radar (LiDAR) and high-precision laser scanners remain predominant, producing highly accurate three-dimensional models (Lu et al., 2020, Wu et al., 2019). Additionally, studies involving RGB-D sensors for tree reconstruction and skeleton extraction have demonstrated the method's balance between efficiency and cost, though improvements in the reconstruction process, algorithm development, and real-time performance are still necessary (Kok et al., 2023, Ma et al., 2021).
随着计算机视觉和人工智能技术的进步，重建过程中遇到的问题，如点云数据注册效率低、3D 重建模型质量差、硬件配置要求高等问题，已得到显著改善。通过获取多角度点云数据并基于特征点匹配点云族，实现高精度三维重建，具有成本低、效率高、精度高的优势。包括 SIFT（Lowe，2004）、SURF（Viola 和 Jones）、FAST（Maier 等人）和 ORB（Rosten 和 Drummond）在内的关键特征点提取算法，可用于寻找和匹配物体的关键特征点，映射空间点云位置，并估计相机位置。相邻点云注册相关算法，如 RANSAC（Li 等人，2017）、捆绑调整（Triggs 等人）和 ICP（Besl 和 McKay，1992），可用于相邻点云注册。在实际植物的三维重建中，使用 TOF 传感器能够获取大豆、番茄和玉米等作物的密集和高精度三维点云数据。这有助于更详细和全面地分析作物结构和生长动态（王等，2022，张等，2022，朱等，2023）。对于树木，激光雷达（LiDAR）和高精度激光扫描仪等设备仍然占主导地位，能够产生高度精确的三维模型（卢等，2020，吴等，2019）。此外，涉及 RGB-D 传感器进行树木重建和骨骼提取的研究表明，该方法在效率和成本之间取得了平衡，尽管在重建过程、算法开发和实时性能方面仍需改进（科克等，2023，马等，2021）。

In summary, significant research has been conducted on the three-dimensional reconstruction and skeleton extraction of crops, with notable advancements. However, developing an autonomous pruning system suitable for complex orchard environments remains a substantial engineering challenge. Current methods include three-dimensional scanning, which offers rapid data acquisition and high fidelity but requires controlled environmental conditions, involves time-consuming data processing, and incurs high costs. Laser radar techniques provide comprehensive coverage but are cumbersome, computationally intensive, and expensive. Methods based on RGB-D sensors are cost-effective, easy to operate, and perform well in real-time but suffer from issues related to lighting and distance, resulting in lower accuracy. These foundational algorithms require specific enhancements to meet the demands of fruit tree self-pruning tasks.

To address these challenges, targeted improvements and optimizations are essential. Developing a method that is low-cost, efficient, and sufficiently accurate is crucial. Our TOF sensor-based 3D reconstruction method for fruit trees strikes a balance between efficiency and accuracy in the automatic extraction of tree trunks. This achievement provides theoretical support and practical references for the future development of practical, commercial, and user-friendly fruit tree pruning robots.

2. Three-dimensional reconstruction and skeleton extraction principle

2.1. Multi-view data acquisition based on kinect sensor

Fruit trees possess characteristics such as large size, complex structure, and high reconstruction difficulty. This paper proposes a method for collecting initial point cloud data of fruit trees from six perspectives using a single sensor. Fig. 1 illustrates the acquisition setup. The initial point cloud data is extracted by rotating the camera counterclockwise around the tree. The angle between each pair of adjacent cameras is 60 degrees to satisfy the overlap requirements during the point cloud registration process. The process of multi-angle point cloud capture is depicted in Fig. 1(a). The camera's shooting distance from the fruit tree and the camera's height from the ground are measured using the LDM-40 laser rangefinder. Considering measurement errors, each measurement point is measured five times, and the mean value is taken as the actual value, as shown in Fig. 1(b).

2.2. Rough registration method based on IWOA-RANSAC algorithm

The extraction of feature points for point cloud registration from the unstructured environment of an orchard is complicated by the presence of substantial noise in the data, posing significant challenges. The Random Sample Consensus (RANSAC) algorithm is adept at robustly estimating model parameters and operates swiftly, facilitating the precise estimation and fitting of initial point cloud data from fruit trees. However, this algorithm is notably sensitive to the choice of the initial point. Consequently, this study employs the RANSAC algorithm for initial registration and integrates an enhanced Whale Optimization Algorithm (WOA) to expediently determine the optimal solution for the RANSAC algorithm. This combination aims to improve the accuracy and efficiency of point cloud feature extraction.

2.2.1. Random Sample Consensus (RANSAC) algorithm

The RANSAC algorithm (Derpanis, 2010) operates by randomly selecting six matching points using a normalization method, identifying the optimal transformation matrix to eliminate error matching, and solving the required unknowns. The expression T of the optimal transformation matrix is as follows:(1)

[\begin{matrix} \begin{matrix} x \\ y \end{matrix} \\ \begin{matrix} z \\ 1 \end{matrix} \end{matrix}] = [\begin{matrix} R_{11} & R_{12} & \begin{matrix} R_{13} & T_{x} \end{matrix} \\ R_{21} & R_{22} & \begin{matrix} R_{23} & T_{y} \end{matrix} \\ \begin{matrix} R_{31} \\ 0 \end{matrix} & \begin{matrix} R_{32} \\ 0 \end{matrix} & \begin{matrix} \begin{matrix} R_{33} \\ 0 \end{matrix} & \begin{matrix} T_{z} \\ 1 \end{matrix} \end{matrix} \end{matrix}] [\begin{matrix} \begin{matrix} x^{'} \\ y^{'} \end{matrix} \\ \begin{matrix} z^{'} \\ 1 \end{matrix} \end{matrix}]

Where (

x^{'}, y^{'}, z^{'}

) and (

x, y, z

) are the Cartesian coordinates of two point clouds to be registered, respectively.

Due to the large volume of point cloud data, obtaining the optimal solution within the specified number of iterations often presents a challenge, and the registration accuracy may also be impacted.

2.2.2. Nonlinear constraint-based whale algorithm

The WOA (Mirjalili and Lewis, 2016) boasts a robust global optimization capability, a straightforward structure, and minimal parameter settings, enhancing the robustness and efficiency of the process. Nonetheless, the global search ability of the traditional WOA is contingent upon its convergence factor, which decreases with an increasing number of iterations. This reduction in the convergence factor lead. To address these limitations, Li et al. incorporated the concept of inertia weight from the Particle Swarm Optimization (PSO) algorithm, introducing adaptive parameters as inertia weight factors. This adaptation, referred to as the IWOA (Inertia Weighted Whale Optimization Algorithm), harnesses the optimal solution more effectively and enhances the algorithm's optimization accuracy (Li et al., 2022a). This innovative approach aims to improve the overall performance of the WOA in complex optimization scenarios.

The flowchart of the algorithm is as shown in Fig. 2:

Among them, t represents the current time of the iteration,

T_{m a x}

represents the maximum iteration time, A is the optimization parameter, f(t) represents the fitting function, which is defined as follows:(2)

f (t) = a r g m i n \sum_{i = 1}^{n} H (l) = \{\begin{matrix} 0.5 l^{2}, |l| < m_{l} \\ 0.5 m_{l} (2 |l| - m_{l}), |l| > m_{l} \end{matrix}

Where

H (l)

denotes the Huber loss,

m_{l}

denotes the preset threshold, and

l

is the distance difference after the transformation of the corresponding points in the

i

^th group.

2.3. Accurate registration method based on 3D-NDT algorithm

While rough registration can correct the initial attitude of the point cloud model, accurate registration of the point cloud is still needed to eliminate positional deviation. The 3D-NDT algorithm, derived from the non-destructive testing algorithm, performs well in point cloud fine registration (Yu et al., 2019). It is a registration algorithm that combines standard optimization and normal detection. The flowchart of the algorithm is as shown in Fig. 3.

2.4. Skeleton extraction method based on Delaunay triangulation and Dijkstra algorithm

Point cloud Delaunay triangulation (Alexa, 2020) involves the triangulation of a group of scattered points in three-dimensional space based on Delaunay rules to generate an optimized spatial triangulation network, reflecting the topological structure relationship between data points and their adjacent points. In this experiment, this method is used to restore the shape of trees from discrete point clouds.

Initially, the mesh initialization of the fruit tree point cloud is conducted, involving the tetrahedron division of the point cloud. Given a known point cloud set, any triangular mesh is established and edge exchange operations are performed until all triangles satisfy the Delaunay condition, as depicted in Fig. 4.

Subsequently, triangular network optimization. Select a point to be processed, use the point location search algorithm to find the tetrahedron containing the point in the tetrahedral data structure, connect the point with the four vertices of the tetrahedron, divide the tetrahedron into four new tetrahedrons, and control the quality of the generated new tetrahedron to ensure that it meets the requirements of Delaunay triangulation to improve the quality and stability of the mesh. Repeat the steps until the point set P is exhausted.

Finally, local optimization of the generated triangle network is performed, and the sub-triangle network is merged. In the three-dimensional point cloud space, because the Lawson local edge exchange method is not applicable, this paper optimizes each vertex or edge locally based on the tetrahedral optimization algorithm, so that the entire mesh has Delaunay properties. Some smaller triangles are merged into a larger triangle to reduce the complexity and computational complexity of the entire triangular mesh. Dijkstra's algorithm facilitates the determination of the shortest path to realize the extraction of the minimum spanning tree model.

3. Materials and methods

3.1. Walnut tree 3D reconstruction platform

The constructed 3D data acquisition platform is depicted in Fig. 5. Component ① is an application-level Kinect v2 camera, which is mainly used for the collection of basic point cloud data. Component ② is a custom-made Kinect v2 camera support device with an adjustable height ranging from approximately 50–250 cm. Component ③ is a graphics workstation equipped with an Intel i5-13490F processor operating at 2.50 GHz, 32 GB RAM memory, a 500 GB Kingston solid-state hard drive, and an NVIDIA GeForce RTX 3060 graphics card, Windows 10 operating system and UBUNTU20.04 operating system. Component ④ is an LDM-40 laser distance meter. Component ⑤ for the measurement tools, including tape and vernier caliper. Component ⑥ is a Xinjiang walnut tree with a height of 2.7 m and a crown diameter of 2.57 m, located in the walnut orchard in Kashgar, Xinjiang.

The workflow primarily includes the mapping of RGB and depth information, the control of camera shooting distance and error, the customized point cloud alignment process, the construction of the 3D model of fruit trees.

(1) Image Mapping: Initially, the intrinsic and extrinsic parameters of the Kinect camera are calibrated. Subsequently, the extracted depth image is mapped and calibrated with the RGB image (F et al., 2022) using equations (3)-(5). Finally, point cloud preprocessing is conducted (Li et al., 2022b).(3)

Z_{rgb} * P_{rgb} = K_{rgb} * R_{i r 2 r g b} * K_{ir}^{- 1} * Z_{ir} * P_{ir} + K_{rgb} * T_{i r 2 r g b}

(4)

R_{i r 2 r g b} = R_{rgb} * R_{ir}^{- 1}

(5)

T_{i r 2 r g b} = T_{rgb} - R_{i r 2 r g b} * T_{ir}

Where

Z_{r g b} * P_{r g b}

represents the mapping of RGB camera data,

Z_{i r} * P_{i r}

represents the mapping of near-infrared camera data, and

P_{r g b} = {[\begin{matrix} u_{r g b} \\ v_{r g b} \\ 1 \end{matrix}]}^{T}

and

P_{i r} = {[\begin{matrix} u_{i r} \\ v_{i r} \\ 1 \end{matrix}]}^{T}

are the pixel coordinates of the RGB and infrared images, respectively.

K_{r g b} = [\begin{matrix} f_{x_r g b} & 0 & c_{x_r g b} \\ 0 & f_{y_r g b} & c_{y_r g b} \\ 0 & 0 & 1 \end{matrix}]

and

K_{i r} = [\begin{matrix} f_{x_i r} & 0 & c_{x_i r} \\ 0 & f_{y_i r} & c_{y_i r} \\ 0 & 0 & 1 \end{matrix}]

are the intrinsic and extrinsic parameter matrices of the RGB and near-infrared cameras.

f_{x_r g b}

and

f_{y_r g b}

represent the focal lengths of the RGB camera.

c_{x_r g b}

and

c_{y_r g b}

are the principal point coordinates. Additionally,

f_{x_i r}

、

f_{y_i r}

、

c_{x_r g b}

、

c_{y_r g b}

、

R_{r g b}

and

T_{r g b}

are the external parameters of the RGB camera, while

R_{i r}

and

T_{i r}

are the external parameters of the depth camera.

(2) Camera shooting distance and error control: To ensure the accuracy of the collected data, the experiments position the Kinect v2 camera at a specific height above the ground (H) and a certain distance from the collection object tree (D). H and D are determined by combining the Kinect v2 camera's own parameters and the height of the collection object after numerous experiments. To minimize the measurement error, the LDM-40 laser rangefinder is used to calculate the average value of five measurements as the actual value. The camera height H is set at one-third of the height of the tree from the ground for optimal data acquisition.

(3) Self-Alignment of Walnut Tree Point Cloud: Initially, the Kinect v2 camera is positioned around the peach tree according to the rule shown in Fig. 1, and the point cloud data of the walnut tree are sequentially extracted from six angles at 60-degree intervals to obtain the initial walnut tree point cloud data. Based on the region of interest, these point cloud data undergo noise reduction, filtering, and extraction of the region of interest. Subsequently, the key points of adjacent point cloud blocks are extracted and their Fast Point Feature Histograms (FPFHs) are calculated. Using the optimized RANSAC and IWOA algorithms, corresponding key points are identified, automatic registration is achieved, and the transformation matrix is calculated. Experimentally, the Pseudo-Huber loss function(Wang et al.) is selected as the basis for judging the performance of coarse registration, and the optimal adjacent point cloud block transformation matrix is obtained. Based on the 3D-NDT algorithm, accurate registration of the point cloud is achieved. Finally, a complete point cloud model of fruit trees is obtained, as shown in Fig. 6.

3.2. Walnut tree skeleton construction system

Skeleton extraction is a technique that creates a representation of a plant's branch and trunk structure through a skeleton or skeleton graph. In the context of walnut tree pruning, skeleton extraction holds significant value. Firstly, it aids in branch identification. Skeleton extraction allows for the clear identification of the main trunk and major branches of walnut trees, assisting in determining which parts require pruning. Secondly, it facilitates structural assessment. Skeleton extraction provides information on the overall structure of a walnut tree, including the distribution, angle, and length of branches. This data can be analyzed to assess whether the tree's morphological structure is balanced and stable. If branches are too dense or overgrown, targeted pruning can be conducted to promote light and air circulation, reduce weight burden, and prevent tree collapse in wind and rain. Lastly, it assists in pest and disease management. Skeleton extraction can help detect possible pests and diseases on walnut trees, as certain pests and diseases may lead to abnormal changes in branch morphology. By examining the skeleton map, affected sections can be detected early and pruned to curb the spread of pests and diseases. In summary, skeleton extraction offers both visual and quantitative data pertinent to walnut tree pruning, guiding decisions on where to prune to foster robust tree development and ensure structural integrity.

(1)
Skeleton Self-Extraction Process and Method: To generate trunks and branches with high-precision geometric and topological structures, this study is based on the coarse-refinement alignment algorithm for fruit tree 3D model construction, and the fruit tree skeleton is extracted based on the Alternating Decision Tree (AdTree) method (Du et al., 2019). Initially, the initial fruit tree 3D model is obtained by filtering and coarse-refinement alignment based on the initial point cloud data, followed by Delaunay triangulation(Su et al., 2022) and Minimum Spanning Tree (MST) extraction, an idea originating from tree nutrient transport paths in ecology, as shown in Fig. 7. The Delaunay triangular sectioning lays the foundation for MST computation, which helps complete missing regions or incomplete branches, and ensures the robustness of input point clouds with poor data quality. After obtaining the triangulated graph, all the edges are weighted with the lengths of the edges defined in the Euclidean space, and the Dijkstra shortest path algorithm(Straub et al., 2022) is used to calculate the MST from the triangulation. The quality of the skeleton can be improved by pre-determining and centralizing the major branching points, assigning weight values to the vertices and edges, and removing small noise components based on this value. Finally, the proximity between neighboring vertices is checked by iterative proximity between vertices and merging neighboring vertices to reconstruct the lightweight fruit tree skeleton.
Download: Download high-res image (394KB)
Download: Download full-size image
Fig. 7. Schematic diagram of tree water transport path.
(2)
Skeleton Geometry Cylindrical Fitting Method: Based on the final fruit tree skeleton, a cylindrical fitting method is used to obtain a geometric model of the fruit tree. Using the optimization method, the exact branch geometry is obtained, the K-dimensional tree (Kd-tree) data structure is used for spatial nearest neighbor search, the branches are quickly segmented, and the Levenberg Marquardt algorithm(Hu et al., 2020) is used to solve the nonlinear least-squares problem to obtain the radii of the subsequent branches. Based on the corresponding pole points, a cylinder is fitted to approximate the branch geometry, and each cylinder that makes up the trunk or all branches can be considered as a closed convex packet polyhedron fit with convex polygons on the top and bottom surfaces of the polyhedron.

3.3. Data acquisition and experimental design

The experimental data were collected from a walnut orchard in Kashgar, Xinjiang, situated in a warm temperate continental arid climate zone. The orchard featured open-center shaped walnut trees, each approximately 2.7 m in height with a crown diameter of around 2.57 m and a trunk diameter up to 0.13 m. The trees were spaced 5 m apart in rows, with a 4-meter spacing between individual plants. Data collection commenced in July 2022 and concluded in December 2022.

During the data collection process, measurements were conducted at noon under sunny conditions to minimize sunlight interference with the sensor's point cloud collection. Additionally, low wind speeds were maintained to prevent disturbances to the trees and leaves. The necessary RGB and depth data of the walnut tree for this research were gathered following the methodology presented in Fig. 1, using the devices ①, ②, ③, ④ and ⑤ shown in Fig. 5. Simultaneously, the height of the walnut tree and the diameter of the five points shown in Fig. 8. were collected as measured data. To minimize the measurement error, the data for each actual measurement point were measured seven times and obtained by the 3σ method for statistics.

The experiment involves extracting point cloud data of a walnut tree through sensors and computers and constructing a three-dimensional model and skeleton model of the fruit tree. The aim is to provide technical support for intelligent pruning by pruning robots. Due to the process of collecting point cloud data, alignment errors and cumulative errors can occur, considering the need for multiple collections and alignments of point cloud data. Therefore, this study adopts the method of multiple reconstructions to reduce the error. The accuracy of the fruit tree reconstruction process can be assessed by comparing the reconstructed 3D model branch data with the separately measured walnut tree branch data. The experimental setup guarantees precise alignment of the fruit trees' multi-angle point clouds and validates the precision of the 3D point cloud models.

4. Results

4.1. Establishment of walnut tree point cloud model and accuracy analysis

4.1.1. Point cloud modeling of walnut trees

This study realized a 3D reconstruction model and phenotypic data measurement of a walnut tree with a height of 2.70 m and a crown diameter of 2.57 m, as depicted in Fig. 9. The multi-angle point cloud was acquired by a Kinect sensor, positioned around the walnut tree at 60-degree intervals for one week. The time interval for extracting this complete point cloud data was measured to be between 8 and 10 min, which is also the time required to acquire the 3D image of the walnut tree. The time interval for the denoising and alignment process of the point cloud slices was between 3 and 5 min.

Fig. 9(a)① displays the obtained 3D reconstruction model of the walnut tree. For the point cloud slices captured from multiple angles, the data quality varies due to differences in the acquisition viewpoints, which is partly the source of the error of the reconstructed 3D point cloud model. During the alignment process, considering that the cumulative error can significantly affect the accuracy of the walnut tree model, this study adopts the alignment of two neighboring point cloud slices to reduce the cumulative error.

Fig. 9(a) ② shows the preliminary classification results of the upper and lower parts of the walnut tree model from the Scalar fields (SF) data acquired by the sensor. As shown in the fig., the SF value of around 0 is the threshold for the upper and lower parts, which can provide support for the future extraction of the main trunk portion of the walnut tree.

Fig. 9(b) displays the effect of Delaunay triangulation of the walnut tree. Where ① is the aligned 3D model of the walnut tree, consisting of a total of 164,520 data points. ② is the surface normal of all the above points calculated, totaling 164520. ③ is the Delaunay triangulation operation for all points in the 3D model, resulting in a total of 2,192,821 surfaces. ④ is a schematic diagram that preserves the key triangular mesh, with a total of 29,886 surfaces.

Fig. 9(c) shows the minimum mesh model based on Delaunay triangulation and the complete point cloud model after filling the missing point cloud. Among them, ① is the minimum mesh model based on Delaunay triangulation, and ② and ③ are the local zoom effects of the mesh model, demonstrating that the method is effective in filling the missing point cloud and the obtained mesh model can meet the requirements of extracting the complete point cloud model. ④ is the complete point cloud model extracted from the mesh model, which has the characteristics of smaller data volume and complete model information compared to the initial point cloud model Fig. 10(a)①, which is conducive to the subsequent extraction of the point cloud skeleton. ⑤⑥ are localized images of model ④.

The time interval used for the Delaunay triangulation calculation of the above walnut tree model, the construction of the grid model Fig. 9(c)①, and the extraction process of the final point cloud model Fig. 9(c)④ was between 1 and 2 min.

4.1.2. Modeling of the walnut tree skeleton

Based on the extracted point cloud model, the walnut tree skeleton is constructed, as depicted in Fig. 10. Fig. 10①-④ demonstrate the process of determining the optimal skeleton by employing 1000 randomly chosen skeleton points.. Based on the concept of tree water transport, such as Fig. 7, combined with the shortest path algorithms, the initial distribution of skeleton points is obtained (Fig. 10⑤). Subsequently, the point cloud skeleton is expanded and optimized based on the final point cloud model in Fig. 9(c)④. Further iterations are performed to examine the proximity between adjacent vertices, and neighboring vertices are merged to reconstruct a lightweight tree skeleton. The derived point cloud skeleton is depicted in Fig. 10⑥ and Fig. 10⑦.

Fig. 10⑧ represents the final fitted cylindrical skeleton model of the walnut tree. Based on the simplified skeleton, a cylindrical fitting method is employed to obtain the geometric model of the tree. Each cylindrical component representing a trunk or branch can be considered as a closed convex hull polyhedron. The top and bottom faces of the polyhedron are convex polygons. The reconstructed model is shown in Fig. 10⑧, while Fig. 10⑨ and ⑩ provide local detail enlarged graphs of the model.

4.2. Experimental result analysis

4.2.1. Point cloud registration accuracy analysis

The impact of the IWOA algorithm on the performance of the RANSAC-NDT algorithm was investigated, with a focus on recording its execution time. These experiments utilized data from fruit trees and were conducted on the graphics workstation depicted in Fig. 5. The implementation was carried out in C++ language. Quantitative analysis involved calculating the average Euclidean distance between matching points of two point cloud groups after precise registration. Ideally, the Euclidean distance should be zero when the point clouds are perfectly matched. This study compares the mean registration error and processing time between the RANSAC-NDT algorithm and the IWOA-enhanced RANSAC-NDT algorithm. As shown in Table 1, the addition of the IWOA algorithm did not significantly alter the accuracy of point cloud registration, maintaining it at 0.0152 m, but it notably reduced the registration time from 25.2 s to 18.1 s, demonstrating the efficiency of the IWOA algorithm.

Table 1. Comparison of RANSAC-NDT with IWOA-RANSAC-NDT.

Registration Method	Amount of point cloud	Registration time/s	Registration error/m
IWOA-RANSAC-NDT	69,563	18.1	0.0152
RANSAC-NDT	69,563	25.2	0.0153

The proposed IWOA-RANSAC-NDT algorithm is compared with the classical ICP algorithm, RANSAC algorithm, and Depth Filtering-ICP algorithm in terms of average registration distance error and registration processing time. The comparative results are presented in Table 2. The findings indicate that the IWOA-RANSAC-NDT algorithm achieves the highest registration accuracy with an average registration distance error of 0.0152 m. Furthermore, the registration time is significantly reduced compared to the other methods while maintaining accuracy.

Table 2. Comparison of different point cloud registration algorithms.

Registration Method	Amount of point cloud	Registration time/s	Registration error/m
ICP	69,563	14.7	0.0825
Depth Filtering-ICP	69,563	25.8	0.0313
IWOA-RANSAC-NDT	69,563	18.1	0.0152

4.2.2. Experimental analysis of skeleton extraction

The skeleton of fruit trees is extracted based on the constructed three-dimensional model, utilizing core algorithms such as 3D-DT (3D Delaunay Triangulation), DSP (Dijkstra's Shortest Path) analysis, and LM-CF (Levenberg-Marquardt Cylindrical Fitting). This experiment was conducted on the graphics workstation depicted in Fig. 5, with the code implemented in C++ language. Table 3 documents the time required for each key algorithm involved in skeleton extraction. The results indicate that the 3D-DT operation takes approximately 3.35 s to generate the triangular mesh of fruit trees, the DSP operation requires about 5.58 s to compute the minimum spanning tree, and the LM-CF operation concludes in about 2.23 s. These findings demonstrate that the skeleton extraction algorithm effectively balances computational efficiency and algorithmic complexity.

Table 3. Skeleton extraction algorithm.

Method	Amount of point cloud	Amount of faces	Registration time/s
3D-DT	164,520	2,192,821	3.35
DSP	164,520	298,860	5.58
LM-CF	164,520	298,860	2.23

4.3. Precision analysis of fruit tree branch reconstruction

To further enhance the usability of our research model and validate its accuracy, a method for calculating the diameter of cylindrical branches and tree height was designed. In the point cloud model, coordinate transformations were employed to obtain tree height data and diameter data for key branches.

To mitigate the impact of data extraction, point cloud registration, and modeling processes on accuracy, this study reconstructed the fruit tree point cloud model ten times. The diameters of five key branches and tree height were statistically analyzed in comparison with the values from the reconstructed models to assess their accuracy, as shown in Table 4.

Table 4. Statistical Tables for Tree Height and Diameter Data.

Parameter	Diameter of branch ①/mm	Diameter of branch ②/mm	Diameter of branch ③/mm	Diameter of branch ④/mm	Diameter of branch ⑤/mm	tree height/m
The real value	121.69	93.35	48.49	45.23	46.82	2693.00
Reconstructed model measurements1	126.85	96.81	47.12	47.36	49.12	2764.78
Reconstructed model measurements2	116.06	89.54	46.94	42.82	43.07	2583.12
Reconstructed model measurements3	124.62	96.43	52.19	48.95	45.64	2741.09
Reconstructed model measurements4	118.78	88.67	46.52	43.09	45.36	2609.25
Reconstructed model measurements5	119.37	91.18	47.72	43.61	44.71	2630.66
Reconstructed model measurements6	125.54	95.58	48.33	43.29	50.09	2793.87
Reconstructed model measurements7	118.29	90.76	51.64	41.85	42.78	2755.03
Reconstructed model measurements8	126.11	97.81	46.63	47.31	47.92	2765.41
Reconstructed model measurements9	127.25	96.12	52.98	49.45	49.83	2813.34
Reconstructed model measurements10	116.32	89.24	46.81	41.03	43.17	2594.91
Average reconstructed value	122.15	92.68	48.80	45.34	47.54	2702.43
Average deviation of reconstruction	4.16	3.34	2.07	2.78	2.59	82.94
The average error of reconstruction	3.42 %	3.58 %	4.27 %	6.15 %	5.54 %	3.08 %

According to the table above, after conducting ten reconstructions and calculating the mean values, the average deviations of the diameter and height of the five key branches were 4.16 mm, 3.34 mm, 2.07 mm, 2.78 mm, 2.59 mm, and 82.94 mm, respectively. The average errors were 3.42 %, 3.58 %, 4.27 %, 6.15 %, 5.54 %, and 3.08 %, respectively. This demonstrates that the quality of diameter and tree height parameters extracted based on the Kinect v2 sensor has good accuracy. It indicates that the diameter and tree height parameters based on the sensor reconstruction model are within the error tolerance range, which can meet the requirements of orchard measurement and pruning.

5. Discussion

5.1. Three-dimensional reconstruction system

Three-dimensional reconstruction is a crucial aspect of intelligent pruning of fruit trees. This study is based on the extraction and registration of multi-angle point cloud data. The registration process, which involves coarse registration and fine registration based on feature points, takes about 3 min and does not meet the needs of real-time three-dimensional reconstruction. To address this issue, the real-time 3D dense reconstruction algorithm could be considered to replace the current point cloud registration method, thereby achieving real-time 3D reconstruction. To address issues related to the quality of crown reconstruction, some existing studies have proposed new research ideas to tackle these challenges (Liang et al., 2022, Yao et al., 2021).

5.2. Design and analysis of experiments

This experiment is conducted based on collected multi-angle point cloud data of fruit trees. The data undergoes pre-processing and alignment to enable 3D reconstruction and skeleton extraction, thereby constructing a 3D reconstruction and skeleton extraction system for fruit trees. Considering the unstructured environment of the orchard, multiple acquisitions are employed for initial data acquisition to mitigate the accumulation of initial errors. The established initial point cloud data processing, 3D reconstruction, and skeleton extraction system serve as a reference for the development of vision models for smart orchards and pruning robots.

However, this experiment still has certain limitations. The process of initial target point cloud extraction in unstructured environments using sensors still requires further exploration. The methods employed in the data acquisition, 3D model reconstruction, and skeleton extraction processes still consume a significant amount of time, severely restricting the operational efficiency of the system and the real-time reconstruction requirements for fruit trees. This remains the primary issue to be addressed in future research.

6. Conclusion

The three-dimensional model of fruit trees is of great significance for the study of intelligent pruning of fruit trees. Our work demonstrates that it is possible to reconstruct the three-dimensional model of fruit trees with low cost and high efficiency based on a depth camera. In this paper, we designed the extraction process of RGB data and depth data of fruit trees and the self-registration and skeleton self-extraction system of point cloud data of fruit trees. The system can obtain a three-dimensional model and skeleton model of fruit trees and perform phenotypic structure measurement of fruit trees, providing support for intelligent pruning of fruit trees. Finally, a field experiment was carried out in a walnut industrial park. The experimental results show that the proposed three-dimensional reconstruction and skeleton extraction methods are reliable, meet the reconstruction error requirements of fruit tree intelligent pruning, and the reconstruction model can provide a reference for digital orchards.

Future work will continue to study the method of real-time 3D reconstruction, with a focus on improving the efficiency of 3D reconstruction and skeleton extraction. The goal is to provide technical support for robotic pruning systems and sophisticated orchard management. This could potentially involve exploring new algorithms, utilizing more advanced sensors, or optimizing the current processes. Additionally, further research could also look into improving the accuracy and robustness of the system in different environmental conditions.

CRediT authorship contribution statement

Xiaojuan Li: Funding acquisition, Investigation, Project administration, Supervision, Writing – review & editing, Conceptualization. Bo Liu: Conceptualization, Investigation, Software, Visualization, Writing – original draft, Data curation. Yinggang Shi: Data curation, Supervision. Mingming Xiong: Data curation, Supervision. Dongyu Ren: Conceptualization, Investigation, Software. Letian Wu: Data curation, Investigation. Xiangjun Zou: Project administration, Writing – review & editing, Funding acquisition.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgement

This research was funded by the Xinjiang Uygur Autonomous Region project ' Research on visual perception and adaptive control system of humanoid picking robot ' (20227SYCCX0061), the National Natural Science Foundation of China project ' Research on bionic vision and adaptive grasping method of Xinjiang long-staple cotton picking robot ' (52265003) and the Xinjiang Uygur Autonomous Region major project ' Research on automatic and intelligent mechanical equipment for vegetables around Tarim Basin ' (2022A02005-5).

Data availability

Data will be made available on request.

References

Alexa, 2020
M. Alexa
Conforming weighted delaunay triangulations
ACM Trans. Graph., 39 (2020)
Google Scholar
Besl and McKay, 1992
P.J. Besl, N.D. McKay
A method for registration of 3-D shapes
IEEE T. Pattern Anal., 14 (1992), pp. 239-256
View in Scopus Google Scholar
Du et al., 2019
Du, S., Lindenbergh, R., Ledoux, H., Stoter, J., Nan, L., 2019. AdTree: Accurate, Detailed, and Automatic Modelling of Laser-Scanned Trees.
Google Scholar
Fu et al., 2020
Y. Fu, C. Li, J. Zhu, B. Wang, B. Zhang, W. Fu
Three-dimensional model construction method and experiment of jujube tree point cloud using alpha-shape algorithm
Trans. Chin. Soc. Agric. Eng., 36 (2020), pp. 214-221
View in Scopus Google Scholar
Giang and Ryoo, 2023
T.T. Giang, Y.-J. Ryoo
Pruning points detection of sweet pepper plants using 3D point clouds and semantic segmentation neural network
Sensors (2023)
Google Scholar
Hu et al., 2020
G. Hu, Z. Zhou, J. Cao, H. Huang
Non-linear calibration optimisation based on the Levenberg–Marquardt algorithm
IET Image Proc., 14 (2020), pp. 1402-1414
Crossref View in Scopus Google Scholar
Itakura et al., 2019
K. Itakura, I. Kamakura, F. Hosoi
Three-dimensional monitoring of plant structural parameters and chlorophyll distribution
Sensors, 19 (2019), p. 413
Crossref View in Scopus Google Scholar
Kok et al., 2023
E. Kok, X. Wang, C. Chen
Obscured tree branches segmentation and 3D reconstruction using deep learning and geometrical constraints
Comput. Electron. Agric., 210 (2023), Article 107884
View PDF View article View in Scopus Google Scholar
Li et al., 2022a
G. Li, Y. Cui, L. Wang, L. Meng
Automatic registration algorithm for the point clouds based on the optimized RANSAC and IWOA algorithms for robotic manufacturing
Appl. Sci. (2022)
Google Scholar
Li et al., 2020
Li, Y., Wang, P., Sun, J., Gan, X., 2020. Simulation of tree point cloud based on the ray-tracing algorithm and three-dimensional tree model. biosystems engineering 200, 259-271.
Google Scholar
Li et al., 2022b
J. Li, W. Ma, Q. Li, C. Zhao, D. Tulpan, S. Yang, L. Ding, R. Gao, L. Yu, Z. Wang
Multi-view real-time acquisition and 3D reconstruction of point clouds for beef cattle
Comput. Electron. Agric., 197 (2022), Article 106987
View PDF View article View in Scopus Google Scholar
Li et al., 2017
L. Li, F. Yang, H. Zhu, D. Li, Y. Li, L. Tang
An improved RANSAC for 3D point cloud plane segmentation based on normal distribution transformation cells
Remote Sens. (Basel) (2017)
Google Scholar
Liang et al., 2022
P. Liang, W. Lin, G. Luo, C. Zhang
Research of hand–eye system with 3D vision towards flexible assembly application
Electronics, 11 (2022), p. 354
Crossref View in Scopus Google Scholar
Lowe, 2004
D.G. Lowe
Distinctive image features from scale-invariant keypoints
Int. J. Comput. Vis., 60 (2004), pp. 91-110
View in Scopus Google Scholar
Lu et al., 2020
S. Lu, G. Li, J. Wang
A novel method for extracting skeleton of fruit tree from 3D point clouds
Int. J. Model., Simul., Sci. Comput., 11 (2020)
Google Scholar
Ma et al., 2021
B. Ma, J. Du, L. Wang, H. Jiang, M. Zhou
Automatic branch detection of jujube trees based on 3D reconstruction for dormant pruning using the deep learning-based method
Comput. Electron. Agric., 190 (2021), Article 106484
View PDF View article View in Scopus Google Scholar
Maier et al., xxxx
Maier, R., Sturm, J., Cremers, D., Submap-based bundle adjustment for 3D reconstruction from RGB-D data. Springer, pp. 54-65.
Google Scholar
Mirjalili and Lewis, 2016
S. Mirjalili, A. Lewis
The whale optimization algorithm
Adv. Eng. Softw., 95 (2016), pp. 51-67
View PDF View article View in Scopus Google Scholar
Nguyen and Vu, 2023
T.H.D. Nguyen, D.C. Vu
A review on phytochemical composition and potential health-promoting properties of walnuts
Food Rev. Intl., 39 (2023), pp. 397-423
Crossref View in Scopus Google Scholar
Okura, 2022
F. Okura
3D modeling and reconstruction of plants and trees: A cross-cutting review across computer graphics, vision, and plant phenotyping
Breed. Sci., 72 (2022), pp. 31-47
Crossref View in Scopus Google Scholar
Rosten et al., xxxx
Rosten, E., Drummond, T., Fusing points and lines for high performance tracking. Ieee, pp. 1508-1515.
Google Scholar
Straub et al., 2022
J. Straub, D. Reiser, N. Lüling, A. Stana, H.W. Griepentrog
Approach for graph-based individual branch modelling of meadow orchard trees with 3D point clouds
Precis. Agric., 23 (2022), pp. 1967-1982
Crossref View in Scopus Google Scholar
Su et al., 2022
T. Su, W. Wang, H. Liu, Z. Liu, X. Li, Z. Jia, L. Zhou, Z. Song, M. Ding, A. Cui
An adaptive and rapid 3D Delaunay triangulation for randomly distributed point cloud data
Vis. Comput., 38 (2022), pp. 197-221
Crossref View in Scopus Google Scholar
Sun et al., 2021
J. Sun, Z. Shen, Y. Wang, H. Bao, X. Zhou
LoFTR: Detector-free local feature matching with transformers
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 8922-8931
Google Scholar
Triggs et al., xxxx
Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W., Bundle adjustment—a modern synthesis. Springer, pp. 298-372.
Google Scholar
Viola and Jones, xxxx
Viola, P., Jones, M., Rapid object detection using a boosted cascade of simple features. Ieee, pp. I-I.
Google Scholar
Wang et al., 2022
Wang, F., Ma, X., Liu, M., Wei, B., 2022. Three-Dimensional Reconstruction of Soybean Canopy Based on Multivision Technology for Calculation of Phenotypic Traits.
Google Scholar
Wang et al., xxxx
Wang, L., Duan, B., Guo, J., Du, S., Jiang, B., Li, Y., Zhou, Y., Liu, C., Zhang, Y., Pseudo-Huber Loss Function-Based Affine Registration Algorithm of Point Clouds. Available at SSRN 4159534.
Google Scholar
Wei et al., 2022
F. Wei, G. Xu, Q. Wu, J. Kuang, P. Tian, P. Qin, Z. Li
Azure kinect calibration and parameter recommendation in different scenarios
IEEE Sensors J., 22 (2022), pp. 9733-9742
Crossref View in Scopus Google Scholar
Wu et al., 2019
S. Wu, B. Xiao, W. Wen, X. Guo, L. Liu
Fruit trees 3D data acquisition and reconstruction based on multi-source
D. Li, C. Zhao (Eds.), Computer and Computing Technologies in Agriculture XI, Springer International Publishing, Cham (2019), pp. 162-172
Crossref Google Scholar
Yang et al., 2019
H. Yang, X. Wang, G. Sun
Three-dimensional morphological measurement method for a fruit tree canopy based on kinect sensor self-calibration
Agronomy, 9 (2019), p. 741
Google Scholar
Yao et al., 2021
Z. Yao, Q. Zhao, X. Li, Q. Bi
Point cloud registration algorithm based on curvature feature similarity
Measurement, 177 (2021), Article 109274
View PDF View article View in Scopus Google Scholar
Yu et al., 2019
H. Yu, Q. Fu, J. Sun, S.L. Wu, Y.M. Chen
Improved 3D-NDT point cloud registration algorithm for indoor mobile robot
Chin. J. Sci. Instrum, 40 (2019), pp. 154-164
View in Scopus Google Scholar
Zahid et al., 2021
A. Zahid, M.S. Mahmud, L. He, P. Heinemann, D. Choi, J. Schupp
Technological advancements towards developing a robotic pruner for apple trees: A review
Comput. Electron. Agric., 189 (2021), Article 106383
View PDF View article View in Scopus Google Scholar
Zahid et al., 2022
A. Zahid, M.S. Mahmud, L. He, J. Schupp, D. Choi, P. Heinemann
An apple tree branch pruning analysis
HortTechnology, 32 (2022), pp. 90-98
Crossref View in Scopus Google Scholar
Zeng et al., 2022
H. Zeng, J. Yang, N. Yang, J. Huang, H. Long, Y. Chen
A Review of the Research Progress of Pruning Robots
2022 IEEE 2nd International Conference on Data Science and Computer Application (ICDSCA) (2022), pp. 1069-1073
Crossref View in Scopus Google Scholar
Zhang et al., 2022
K. Zhang, H. Chen, H. Wu, X. Zhao, C. Zhou
Point cloud registration method for maize plants based on conical surface fitting-ICP
Sci Rep, 12 (2022), p. 6852
View in Scopus Google Scholar
Zhu et al., 2023
T. Zhu, X. Ma, H. Guan, X. Wu, F. Wang, C. Yang, Q. Jiang
A calculation method of phenotypic traits based on three-dimensional reconstruction of tomato canopy
Comput. Electron. Agric., 204 (2023), Article 107515
View PDF View article View in Scopus Google Scholar

Cited by (0)

View Abstract

[1] Alexa, 2020
M. Alexa
Conforming weighted delaunay triangulations
ACM Trans. Graph., 39 (2020)
Google Scholar

[2] Besl and McKay, 1992
P.J. Besl, N.D. McKay
A method for registration of 3-D shapes
IEEE T. Pattern Anal., 14 (1992), pp. 239-256
View in Scopus Google Scholar

[3] Du et al., 2019
Du, S., Lindenbergh, R., Ledoux, H., Stoter, J., Nan, L., 2019. AdTree: Accurate, Detailed, and Automatic Modelling of Laser-Scanned Trees.
Google Scholar

[4] Fu et al., 2020
Y. Fu, C. Li, J. Zhu, B. Wang, B. Zhang, W. Fu
Three-dimensional model construction method and experiment of jujube tree point cloud using alpha-shape algorithm
Trans. Chin. Soc. Agric. Eng., 36 (2020), pp. 214-221
View in Scopus Google Scholar

[5] Giang and Ryoo, 2023
T.T. Giang, Y.-J. Ryoo
Pruning points detection of sweet pepper plants using 3D point clouds and semantic segmentation neural network
Sensors (2023)
Google Scholar

[6] Hu et al., 2020
G. Hu, Z. Zhou, J. Cao, H. Huang
Non-linear calibration optimisation based on the Levenberg–Marquardt algorithm
IET Image Proc., 14 (2020), pp. 1402-1414
Crossref View in Scopus Google Scholar

[7] Itakura et al., 2019
K. Itakura, I. Kamakura, F. Hosoi
Three-dimensional monitoring of plant structural parameters and chlorophyll distribution
Sensors, 19 (2019), p. 413
Crossref View in Scopus Google Scholar

[8] Kok et al., 2023
E. Kok, X. Wang, C. Chen
Obscured tree branches segmentation and 3D reconstruction using deep learning and geometrical constraints
Comput. Electron. Agric., 210 (2023), Article 107884
View PDF View article View in Scopus Google Scholar

[9] Li et al., 2022a
G. Li, Y. Cui, L. Wang, L. Meng
Automatic registration algorithm for the point clouds based on the optimized RANSAC and IWOA algorithms for robotic manufacturing
Appl. Sci. (2022)
Google Scholar

[10] Li et al., 2020
Li, Y., Wang, P., Sun, J., Gan, X., 2020. Simulation of tree point cloud based on the ray-tracing algorithm and three-dimensional tree model. biosystems engineering 200, 259-271.
Google Scholar

[11] Li et al., 2022b
J. Li, W. Ma, Q. Li, C. Zhao, D. Tulpan, S. Yang, L. Ding, R. Gao, L. Yu, Z. Wang
Multi-view real-time acquisition and 3D reconstruction of point clouds for beef cattle
Comput. Electron. Agric., 197 (2022), Article 106987
View PDF View article View in Scopus Google Scholar

[12] Li et al., 2017
L. Li, F. Yang, H. Zhu, D. Li, Y. Li, L. Tang
An improved RANSAC for 3D point cloud plane segmentation based on normal distribution transformation cells
Remote Sens. (Basel) (2017)
Google Scholar

[13] Liang et al., 2022
P. Liang, W. Lin, G. Luo, C. Zhang
Research of hand–eye system with 3D vision towards flexible assembly application
Electronics, 11 (2022), p. 354
Crossref View in Scopus Google Scholar

[14] Lowe, 2004
D.G. Lowe
Distinctive image features from scale-invariant keypoints
Int. J. Comput. Vis., 60 (2004), pp. 91-110
View in Scopus Google Scholar

[15] Lu et al., 2020
S. Lu, G. Li, J. Wang
A novel method for extracting skeleton of fruit tree from 3D point clouds
Int. J. Model., Simul., Sci. Comput., 11 (2020)
Google Scholar

[16] Ma et al., 2021
B. Ma, J. Du, L. Wang, H. Jiang, M. Zhou
Automatic branch detection of jujube trees based on 3D reconstruction for dormant pruning using the deep learning-based method
Comput. Electron. Agric., 190 (2021), Article 106484
View PDF View article View in Scopus Google Scholar

[17] Maier et al., xxxx
Maier, R., Sturm, J., Cremers, D., Submap-based bundle adjustment for 3D reconstruction from RGB-D data. Springer, pp. 54-65.
Google Scholar

[18] Mirjalili and Lewis, 2016
S. Mirjalili, A. Lewis
The whale optimization algorithm
Adv. Eng. Softw., 95 (2016), pp. 51-67
View PDF View article View in Scopus Google Scholar

[19] Nguyen and Vu, 2023
T.H.D. Nguyen, D.C. Vu
A review on phytochemical composition and potential health-promoting properties of walnuts
Food Rev. Intl., 39 (2023), pp. 397-423
Crossref View in Scopus Google Scholar

[20] Okura, 2022
F. Okura
3D modeling and reconstruction of plants and trees: A cross-cutting review across computer graphics, vision, and plant phenotyping
Breed. Sci., 72 (2022), pp. 31-47
Crossref View in Scopus Google Scholar

[21] Rosten et al., xxxx
Rosten, E., Drummond, T., Fusing points and lines for high performance tracking. Ieee, pp. 1508-1515.
Google Scholar

[22] Straub et al., 2022
J. Straub, D. Reiser, N. Lüling, A. Stana, H.W. Griepentrog
Approach for graph-based individual branch modelling of meadow orchard trees with 3D point clouds
Precis. Agric., 23 (2022), pp. 1967-1982
Crossref View in Scopus Google Scholar

[23] Su et al., 2022
T. Su, W. Wang, H. Liu, Z. Liu, X. Li, Z. Jia, L. Zhou, Z. Song, M. Ding, A. Cui
An adaptive and rapid 3D Delaunay triangulation for randomly distributed point cloud data
Vis. Comput., 38 (2022), pp. 197-221
Crossref View in Scopus Google Scholar

[24] Sun et al., 2021
J. Sun, Z. Shen, Y. Wang, H. Bao, X. Zhou
LoFTR: Detector-free local feature matching with transformers
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021), pp. 8922-8931
Google Scholar

[25] Triggs et al., xxxx
Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W., Bundle adjustment—a modern synthesis. Springer, pp. 298-372.
Google Scholar

[26] Viola and Jones, xxxx
Viola, P., Jones, M., Rapid object detection using a boosted cascade of simple features. Ieee, pp. I-I.
Google Scholar

[27] Wang et al., 2022
Wang, F., Ma, X., Liu, M., Wei, B., 2022. Three-Dimensional Reconstruction of Soybean Canopy Based on Multivision Technology for Calculation of Phenotypic Traits.
Google Scholar

[28] Wang et al., xxxx
Wang, L., Duan, B., Guo, J., Du, S., Jiang, B., Li, Y., Zhou, Y., Liu, C., Zhang, Y., Pseudo-Huber Loss Function-Based Affine Registration Algorithm of Point Clouds. Available at SSRN 4159534.
Google Scholar

[29] Wei et al., 2022
F. Wei, G. Xu, Q. Wu, J. Kuang, P. Tian, P. Qin, Z. Li
Azure kinect calibration and parameter recommendation in different scenarios
IEEE Sensors J., 22 (2022), pp. 9733-9742
Crossref View in Scopus Google Scholar

[30] Wu et al., 2019
S. Wu, B. Xiao, W. Wen, X. Guo, L. Liu
Fruit trees 3D data acquisition and reconstruction based on multi-source
D. Li, C. Zhao (Eds.), Computer and Computing Technologies in Agriculture XI, Springer International Publishing, Cham (2019), pp. 162-172
Crossref Google Scholar

[31] Yang et al., 2019
H. Yang, X. Wang, G. Sun
Three-dimensional morphological measurement method for a fruit tree canopy based on kinect sensor self-calibration
Agronomy, 9 (2019), p. 741
Google Scholar

[32] Yao et al., 2021
Z. Yao, Q. Zhao, X. Li, Q. Bi
Point cloud registration algorithm based on curvature feature similarity
Measurement, 177 (2021), Article 109274
View PDF View article View in Scopus Google Scholar

[33] Yu et al., 2019
H. Yu, Q. Fu, J. Sun, S.L. Wu, Y.M. Chen
Improved 3D-NDT point cloud registration algorithm for indoor mobile robot
Chin. J. Sci. Instrum, 40 (2019), pp. 154-164
View in Scopus Google Scholar

[34] Zahid et al., 2021
A. Zahid, M.S. Mahmud, L. He, P. Heinemann, D. Choi, J. Schupp
Technological advancements towards developing a robotic pruner for apple trees: A review
Comput. Electron. Agric., 189 (2021), Article 106383
View PDF View article View in Scopus Google Scholar

[35] Zahid et al., 2022
A. Zahid, M.S. Mahmud, L. He, J. Schupp, D. Choi, P. Heinemann
An apple tree branch pruning analysis
HortTechnology, 32 (2022), pp. 90-98
Crossref View in Scopus Google Scholar

[36] Zeng et al., 2022
H. Zeng, J. Yang, N. Yang, J. Huang, H. Long, Y. Chen
A Review of the Research Progress of Pruning Robots
2022 IEEE 2nd International Conference on Data Science and Computer Application (ICDSCA) (2022), pp. 1069-1073
Crossref View in Scopus Google Scholar

[37] Zhang et al., 2022
K. Zhang, H. Chen, H. Wu, X. Zhao, C. Zhou
Point cloud registration method for maize plants based on conical surface fitting-ICP
Sci Rep, 12 (2022), p. 6852
View in Scopus Google Scholar

[38] Zhu et al., 2023
T. Zhu, X. Ma, H. Guan, X. Wu, F. Wang, C. Yang, Q. Jiang
A calculation method of phenotypic traits based on three-dimensional reconstruction of tomato canopy
Comput. Electron. Agric., 204 (2023), Article 107515
View PDF View article View in Scopus Google Scholar

Outline 概要

Figures (10) 图（10）

Tables (4)

Computers and Electronics in Agriculture
计算机与电子在农业中

Highlights 亮点

Abstract 摘要

Keywords 关键词

1. Introduction 1. 引言

2. Three-dimensional reconstruction and skeleton extraction principle

2.1. Multi-view data acquisition based on kinect sensor

2.2. Rough registration method based on IWOA-RANSAC algorithm

2.2.1. Random Sample Consensus (RANSAC) algorithm

2.2.2. Nonlinear constraint-based whale algorithm

2.3. Accurate registration method based on 3D-NDT algorithm

2.4. Skeleton extraction method based on Delaunay triangulation and Dijkstra algorithm

3. Materials and methods

3.1. Walnut tree 3D reconstruction platform

3.2. Walnut tree skeleton construction system

3.3. Data acquisition and experimental design

4. Results

4.1. Establishment of walnut tree point cloud model and accuracy analysis

4.1.1. Point cloud modeling of walnut trees

4.1.2. Modeling of the walnut tree skeleton

4.2. Experimental result analysis

4.2.1. Point cloud registration accuracy analysis

4.2.2. Experimental analysis of skeleton extraction

4.3. Precision analysis of fruit tree branch reconstruction

5. Discussion

5.1. Three-dimensional reconstruction system

5.2. Design and analysis of experiments

6. Conclusion

CRediT authorship contribution statement

Declaration of competing interest

Acknowledgement

Data availability

References

Cited by (0)

Computers and Electronics in Agriculture计算机与电子在农业中

Efficient three-dimensional reconstruction and skeleton extraction for intelligent pruning of fruit trees高效的三维重建和果树骨架提取，以实现智能修剪

Highlights 亮点

Abstract 摘要

Keywords 关键词

1. Introduction 1. 引言

2. Three-dimensional reconstruction and skeleton extraction principle

2.1. Multi-view data acquisition based on kinect sensor

2.2. Rough registration method based on IWOA-RANSAC algorithm

2.2.1. Random Sample Consensus (RANSAC) algorithm

2.2.2. Nonlinear constraint-based whale algorithm

2.3. Accurate registration method based on 3D-NDT algorithm

2.4. Skeleton extraction method based on Delaunay triangulation and Dijkstra algorithm

3. Materials and methods

3.1. Walnut tree 3D reconstruction platform

3.2. Walnut tree skeleton construction system

3.3. Data acquisition and experimental design

4. Results

4.1. Establishment of walnut tree point cloud model and accuracy analysis

4.1.1. Point cloud modeling of walnut trees

4.1.2. Modeling of the walnut tree skeleton

4.2. Experimental result analysis

4.2.1. Point cloud registration accuracy analysis

4.2.2. Experimental analysis of skeleton extraction

4.3. Precision analysis of fruit tree branch reconstruction

5. Discussion

5.1. Three-dimensional reconstruction system

5.2. Design and analysis of experiments

6. Conclusion

CRediT authorship contribution statement

Declaration of competing interest

Acknowledgement

Data availability

References

Computers and Electronics in Agriculture
计算机与电子在农业中

Efficient three-dimensional reconstruction and skeleton extraction for intelligent pruning of fruit trees
高效的三维重建和果树骨架提取，以实现智能修剪