这是用户在 2024-5-20 24:22 为 https://app.immersivetranslate.com/pdf-pro/4f9d8054-2b83-4889-a768-edb8d5132183 保存的双语快照页面,由 沉浸式翻译 提供双语支持。了解如何保存?
2024_05_20_6272fd3eea1461074c11g

Maize stem-leaf segmentation framework based on deformable point clouds
基于可变形点云的玉米茎叶分割框架


Tao Yang , Tongyu
陶阳 、童宇

College of Information and Electrical Engineering, Shenyang Agricultural University, Shenyang 110161, China
沉阳农业大学信息与电气工程学院,沉阳 110161

School of Mathematics and Computer Science, Zhejiang Agriculture and Forestry University, Hangzhou 311300,China
浙江农林大学数学与计算机科学学院,浙江 杭州 311300

School of Information and Intelligence Engineering, University of Sanya, Sanya 572022, China
三亚大学信息与智能工程学院, 三亚 572022

}

A R T I C L E I N F O
文章信息

Keywords: 关键词:

Maize Point Cloud Dataset
玉米点云数据集
Physically based deformation
基于物理的变形
Point Cloud Data Augmentation
点云数据增强
Stem-Leaf Segmentation 茎叶分割

Abstract 抽象的

A B S T R A C T The efficacy of three-dimensional (3D) point clouds in studying crop morphological structures is based on their direct and accurate data presentation ability. With deep-learning integration, organ segmentation from point clouds could serve as the basis for tremendous advancements in organ-level phenotyping. However, despite the potential, the acquisition of a sufficient number of annotated plant point clouds for practical model training remains a major hurdle. To help overcome this limitation, we constructed a 3D point-cloud dataset specifically for maize stem-leaf segmentation encompassing 428 maize plants ranging from 2 to 12 leaves. We also developed a point cloud enhancement strategy that uses highly controllable deformations to improve the morphological diversity of the training set significantly, while preserving the local geometric features of organs. Our dataset supports the generation of abundant training data from a limited number of labelled data, and we also provide a segmentation framework based on the augmented data to validate the efficiency of our enhancement technique. Two labelled data items were randomly chosen from our plant dataset based on every leaf number, yielding 22 labelled data items total, to produce several deformed point clouds for training the PointNet++ semantic segmentation model, as well as the hierarchical aggregation for the 3D instant segmentation (HAIS) model. These models were tested on 406 datasets, where the PointNet++ model secured a mean intersection-over-union (mIoU) in semantic segmentation and the HAIS model obtained an mean average precision (mAP) in instance segmentation. Following post-processing, an instance segmentation result of mAP was achieved with the HAIS model. These findings demonstrate that our method allows for the efficient training of organ segmentation models with minimal labelled data input in a reduced timeframe. Moreover, it offers an effective tool for point-cloud parsing in maize phenotyping research. Our Maize dataset is available from https://github.com/syau-miao/SignleMaizePointCloudDataSet.git, and the source code of our method can be found at https://github.com/yangxin6/Deformation3D.git.
摘要 三维(3D)点云在研究作物形态结构方面的功效基于其直接、准确的数据呈现能力。通过深度学习集成,点云的器官分割可以作为器官水平表型分析巨大进步的基础。然而,尽管有潜力,获取足够数量的带注释的植物点云用于实际模型训练仍然是一个主要障碍。为了帮助克服这一限制,我们构建了一个专门用于玉米茎叶分割的 3D 点云数据集,其中包含 428 株玉米植株,叶子数量从 2 到 12 片不等。我们还开发了一种点云增强策略,该策略使用高度可控的变形来显着提高训练集的形态多样性,同时保留器官的局部几何特征。我们的数据集支持从有限数量的标记数据生成丰富的训练数据,并且我们还提供基于增强数据的分割框架来验证我们增强技术的效率。根据每个叶子的数量,从我们的植物数据集中随机选择两个标记数据项,总共产生 22 个标记数据项,以生成多个变形点云,用于训练 PointNet++ 语义分割模型,以及用于 3D 即时分割的分层聚合( HAIS)模型。这些模型在 406 个数据集上进行了测试,其中 PointNet++ 模型在语义分割中获得了 平均交集 (mIoU),HAIS 模型获得了 平均平均精度( mAP)在实例分割中。经过后处理,使用HAIS模型获得了 mAP的实例分割结果。 这些发现表明,我们的方法可以在更短的时间内以最少的标记数据输入有效地训练器官分割模型。此外,它为玉米表型研究中的点云解析提供了有效的工具。我们的玉米数据集可从 https://github.com/syau-miao/SignleMaizePointCloudDataSet.git 获取,我们方法的源代码可以在 https://github.com/yangxin6/Deformation3D.git 找到。

1. Introduction 一、简介

Maize plays a crucial role in sustaining the global food supply, and high-throughput phenotyping is required for future varietal improvements in the species. In recent years, the ubiquitous use of threedimensional (3D) sensing technologies, such as laser scanning (Su et al., 2018), multi-angle imaging (Wu et al., 2020), and LIDAR (Jin et al., 2019), has resulted in revolutionary advancements in the measurement of plant phenotype parameters. Precise organ-level segmentation is essential for measuring plant phenotypic features (e.g. the leaf length and width), and substantial research has been conducted on this issue.
玉米在维持全球粮食供应方面发挥着至关重要的作用,未来该物种的品种改良需要高通量表型分析。近年来,三维(3D)传感技术的广泛使用,例如激光扫描(Su et al., 2018)、多角度成像(Wu et al., 2020)和激光雷达(Jin et al., 2019) ),在植物表型参数的测量方面带来了革命性的进步。精确的器官水平分割对于测量植物表型特征(例如叶长和宽度)至关重要,并且已经针对这个问题进行了大量研究。
Traditional approaches primarily target plants with specific morphologies using the topological and morphological features of these plants as a priori knowledge to create hand-crafted segmentation features (Elnashef et al., 2019; Jin et al., 2019). These methods do not require data annotation and can yield effective results for plants with certain morphologies. However, they exhibit limited generalisability and present challenges when they are applied to different entities. In addition, their high sensitivity to parameters significantly complicates the determination of optimal parameters that are suitable for a wider range of data.
传统方法主要针对具有特定形态的植物,使用这些植物的拓扑和形态特征作为先验知识来创建手工分割特征(Elnashef 等,2019;Jin 等,2019)。这些方法不需要数据注释,并且可以对具有某些形态的植物产生有效的结果。然而,它们的通用性有限,并且在应用于不同实体时面临挑战。此外,它们对参数的高敏感性使确定适合更广泛数据的最佳参数变得非常复杂。
Deep-learning methods use data-driven strategies for the direct learning of point cloud feature extraction and combination schemes that aid in segmentation tasks. Methods for applying deep learning to pointcloud segmentation are generally categorised as supervised, semisupervised, and unsupervised learning. Recent 3D-based supervised deep-learning methods have exhibited great potential in improving the generality and accuracy of organ segmentation. Supervised learningbased models require a vast amount of labelled data for training, and labelling point-cloud data is extremely time consuming and labour intensive. Despite the continual innovations and improvements of neural network structures, the problem of missing annotated plant 3D data has not been effectively solved, which has become a key factor restricting progress in this field. Although point-cloud data augmentation can alleviate the data dependency pressure to a certain extent, existing methods still have room for improvement in increasing plant morphological diversity and improving method controllability. Several unsupervised and semi-supervised poin-cloud segmentation deeplearning models have been proposed in recent years (Zhang et al., 2023b; Wang and Wang, 2020; de Gélis et al., 2023). These methods require little or no annotated data, thereby offering new perspectives for plant organ segmentation. However, the segmentation accuracy of these current methods still lags behind that of supervised learning approaches and their applicability in plant point clouds necessitates substantial work for validation.
深度学习方法使用数据驱动策略来直接学习点云特征提取和有助于分割任务的组合方案。将深度学习应用于点云分割的方法通常分为监督学习、半监督学习和无监督学习。最近基于 3D 的监督深度学习方法在提高器官分割的通用性和准确性方面表现出了巨大的潜力。基于监督学习的模型需要大量的标记数据进行训练,而标记点云数据极其耗时且耗费人力。尽管神经网络结构不断创新和改进,但植物3D标注数据缺失的问题仍未得到有效解决,成为制约该领域进展的关键因素。虽然点云数据增强可以在一定程度上缓解数据依赖压力,但现有方法在增加植物形态多样性和提高方法可控性方面仍有改进空间。近年来提出了几种无监督和半监督的点云分割深度学习模型(Zhang et al., 2023b; Wang and Wang, 2020; de Gélis et al., 2023)。这些方法需要很少或不需要注释数据,从而为植物器官分割提供新的视角。然而,这些当前方法的分割精度仍然落后于监督学习方法,并且它们在植物点云中的适用性需要大量的验证工作。
This study aims to design a set of methodologies that can mitigate the dependence on labelled data of supervised learning segmentation models. The contributions of our current study are threefold: we provide a maize point-cloud dataset for organ segmentation; we improve the point-cloud augmentation based on physical deformations to increase the morphological variety of the maize point-cloud training data while maintaining the local geometric features of organs; and we develop a maize stem-leaf segmentation framework based on deformable pointcloud data to enable deep-learning model training with minimal annotated data items.
本研究旨在设计一套方法,可以减轻监督学习分割模型对标记数据的依赖。我们当前研究的贡献有三个:我们提供了用于器官分割的玉米点云数据集;我们改进了基于物理变形的点云增强,以增加玉米点云训练数据的形态多样性,同时保持器官的局部几何特征;我们开发了一个基于可变形点云数据的玉米茎叶分割框架,以便能够使用最少的注释数据项进行深度学习模型训练。

2.1. Traditional plant point-cloud segmentation
2.1. 传统植物点云分割

Numerous traditional methods have been proposed for the segmentation of 3D plant organs, which commonly utilise hand-crafted features such as local geometric features and global topological structures. Elnashef et al. (2019) employed tensor features of local point clouds to segment the stems and leaves of maize plants during the seedling stage. Jin et al. (2019) introduced a median normalisation vector growth algorithm that rapidly segments maize stems and leaves using the directional features of local point clouds. Li et al. (Li et al., 2018a) achieved the regional growth leaf segmentation of dense plants using the directional and curvature features of local point clouds. The global topological structure of plants delineates the spatial and connective relationships between various organs such as stems and leaves, thereby serving as pivotal prior knowledge for segmentation. Several studies have adopted point-cloud skeletons to represent these global topological structures (Chaudhury and Godin, 2020; Miao et al., 2021b; Xiang et al., 2019). These skeleton methods consolidate a specific range of point clouds into skeleton vertices, which are subsequently interconnected to form edges based on their relational bonds, ultimately culminating in a graph structure (Wu et al., 2019). Thereafter, plant segmentation is realised by extracting the shortest paths between these skeleton vertices from the graph structure, guided by the topological relationships between the plant stems and leaves (Zermas et al., 2020). Although these traditional methods eliminate the need for model pre-training, they are limited in their segmentation efficiency and highly sensitive to parameter changes.
人们提出了许多传统的 3D 植物器官分割方法,这些方法通常利用手工制作的特征,例如局部几何特征和全局拓扑结构。埃尔纳谢夫等人。 (2019)利用局部点云的张量特征来分割幼苗阶段玉米植株的茎和叶。金等人。 (2019)引入了一种中值归一化向量增长算法,该算法利用局部点云的方向特征快速分割玉米茎和叶。李等人。 (Li et al., 2018a)利用局部点云的方向性和曲率特征实现了密集植物的区域生长叶片分割。植物的整体拓扑结构描绘了茎、叶等各个器官之间的空间和连接关系,从而成为分割的关键先验知识。一些研究采用点云骨架来表示这些全局拓扑结构(Chaudhury and Godin,2020;Miao et al.,2021b;Xiang et al.,2019)。这些骨架方法将特定范围的点云合并为骨架顶点,这些顶点随后相互连接以基于其关系键形成边,最终形成图结构(Wu et al., 2019)。此后,在植物茎和叶之间的拓扑关系的指导下,通过从图结构中提取这些骨架顶点之间的最短路径来实现植物分割(Zermas et al., 2020)。尽管这些传统方法消除了模型预训练的需要,但它们的分割效率有限并且对参数变化高度敏感。

2.2. Plant point-cloud segmentation via deep learning
2.2.通过深度学习进行植物点云分割

Among the supervised, semi-supervised, and unsupervised learning paradigms for point-cloud segmentation, supervised learning exhibits significant effectiveness. For example, Jin et al. (2020) developed a voxel-based convolutional neural network (VCNN) that classifies and segments maize at different growth stages. Turgut et al. (Turgut et al., 2022) used the PointNet++ model (Qi et al., 2017) to achieve the semantic segmentation of rose bushes. Li et al., (2022a) similarly developed simultaneous semantic and instance segmentation for plant point clouds that differentiates stems and leaves, as well as single leaf instances. Li et al., (2022b) used the legacy PointNet network in conjunction with a mean-shift algorithm to perform semantic and instance segmentation of stems and leaves. Ao et al. (2022) used the point cloud-based feature learning model (Li et al., 2018b) to segment maize stems and leaves successfully, and Zhang et al. (Zhang et al., 2023a) developed a PointNet++ variant that segments maize tassels based on their ear tip features. Du et al. (2023) developed PSTPointGroup, which is a novel deep-learning approach for highresolution instance segmentation of siliques in rapeseed plant point clouds, using dynamic voxel encoding and attention mechanisms. Sun et al. (Sun et al., 2023b) used point and voxel representations and a point-VCNN (Liu et al., 2019) to segment the main stems, branches, and bolls of cotton plants in an end-to-end manner. Guo et al. (2023) presented a novel feature-fusion network for semantic segmentation that consists of voxel- and point-branches, and demonstrated its efficacy through comprehensive experiments on large-scale maize and tomato point clouds. Unsupervised and semi-supervised methods that are grounded in deep learning offer novel perspectives for plant point-cloud organ segmentation, primarily owing to their reduced reliance on extensive (or minimal) manual annotations for neural network training. Luo et al. (2023) proposed a weakly supervised 3D organ-level plant shoot segmentation model using an annotation-efficient point-cloud framework. Nonetheless, the practical applicability of unsupervised and semi-supervised methods in plant point-cloud segmentation remains a ripe area for further exploration.
在点云分割的监督、半监督和无监督学习范式中,监督学习表现出显着的有效性。例如,金等人。 (2020)开发了一种基于体素的卷积神经网络(VCNN),可以对不同生长阶段的玉米进行分类和分割。图尔古特等人。 (Turgut et al., 2022)使用PointNet++模型(Qi et al., 2017)实现了玫瑰花丛的语义分割。 Li 等人(2022a)同样开发了植物点云的同步语义和实例分割,可以区分茎和叶以及单叶实例。 Li 等人 (2022b) 使用传统的 PointNet 网络与均值平移算法结合来执行茎和叶的语义和实例分割。敖等人。 (2022)使用基于点云的特征学习模型(Li et al., 2018b)成功分割了玉米茎叶,Zhang et al. (Zhang et al., 2023a) 开发了一种 PointNet++ 变体,可以根据耳尖特征分割玉米穗。杜等人。 (2023) 开发了 PSTPointGroup,这是一种新颖的深度学习方法,使用动态体素编码和注意机制,对油菜植物点云中的长角果进行高分辨率实例分割。孙等人。 (Sun et al., 2023b) 使用点和体素表示以及点 VCNN (Liu et al., 2019) 以端到端的方式分割棉花植物的主茎、分枝和棉铃。郭等人。 (2023)提出了一种由体素和点分支组成的用于语义分割的新型特征融合网络,并通过对大规模玉米和番茄点云的综合实验证明了其功效。 基于深度学习的无监督和半监督方法为植物点云器官分割提供了新颖的视角,主要是因为它们减少了对神经网络训练的广泛(或最少)手动注释的依赖。罗等人。 (2023) 使用注释高效的点云框架提出了一种弱监督的 3D 器官级植物芽分割模型。尽管如此,无监督和半监督方法在植物点云分割中的实际适用性仍然是进一步探索的成熟领域。

2.3. Point-cloud data augmentation
2.3.点云数据增强

The current lack of sufficient annotated plant-based point-cloud data is a severely limiting factor. As a result, several scholars have presented affine transformations to enhance and expand existing data collections. For example, Shi et al. (2020) and Chen et al.(2021) introduced random flipping, global scaling, global rotation, and global translation methods to enhance the training set, resulting in improved 3D object detection performance in street scenes. Xin et al. (2023) validated the effectiveness of these affine transformations, particularly translation and rotation, for the 3D semantic segmentation of tomato plants using point clouds. Affine transformation can be applied to the point cloud of the entire plant or a single organ, thereby increasing the diversity of plant pose and organ distribution in 3D space. However, these methods cannot deal with deformation such as bending and twisting, and thus, their ability to enhance the diversity of geometric forms of plant organs is limited. Jitter (Yu and Grauman, 2017) and elastic transformations (Gabrani and Tretiak, 1996) alter the morphological features of data, resulting in greater data variability (Chen et al., 2021; Jiang et al., 2020; Ngo et al., 2023). However, the predominant focus of these methods on the deformation of entire point-cloud scenes, coupled with the largely random and challenging-to-fine-tune nature of these deformations, means that the gain in data diversity still needs to be strengthened.
目前缺乏足够的带注释的基于植物的点云数据是一个严重的限制因素。因此,一些学者提出了仿射变换来增强和扩展现有的数据收集。例如,石等人。 (2020) 和 Chen 等人 (2021) 引入了随机翻转、全局缩放、全局旋转和全局平移方法来增强训练集,从而提高了街道场景中的 3D 对象检测性能。辛等人。 (2023) 验证了这些仿射变换(特别是平移和旋转)对于使用点云对番茄植株进行 3D 语义分割的有效性。仿射变换可以应用于整个植物或单个器官的点云,从而增加3D空间中植物姿态和器官分布的多样性。然而,这些方法无法处理诸如弯曲和扭曲等变形,因此它们增强植物器官几何形态多样性的能力有限。抖动(Yu 和 Grauman,2017)和弹性变换(Gabrani 和 Tretiak,1996)改变数据的形态特征,导致更大的数据变异性(Chen 等,2021;Jiang 等,2020;Ngo 等, 2023)。然而,这些方法的主要关注点是整个点云场景的变形,再加上这些变形的很大程度上随机性和难以微调的性质,意味着数据多样性的增益仍然需要加强。

3. Materials and methods
3 材料与方法

3.1. Dataset construction
3.1.数据集构建

Field experiments were conducted in the experimental maize field at
田间试验在玉米试验田进行
Shenyang Agricultural University between May and July 2019, 2020 and 2021. Five maize varieties (Xian Yu 335, LD145, LD502, LD586 and LD 1281) were planted in plots with a total area of and row-row and plant-plant distances of 60 and , respectively. Maize samples were randomly selected and transplanted to pots in an indoor laboratory. Raw point clouds were obtained using a 3D laser scanner (FreeScan X3; Tianyuan Inc., Beijing, China).
2019年5月至7月、2020年和2021年沉阳农业大学 。在总面积为 。随机选择玉米样品 并移植到室内实验室的盆中。使用 3D 激光扫描仪(FreeScan X3;天元公司,北京,中国)获得原始点云。
The unprocessed dataset comprised point clouds representing both the pot and various surrounding objects. Using CloudCompare Stereo software, these extraneous points were manually eradicated, thereby isolating only the point cloud associated with the plant. Further refinements to eliminate noise involved the use of pass-through and statistical outlier-removal filters. To preserve the morphological characteristics of each plant while minimising the data scale of the point-cloud database, the farthest point-sampling approach was employed. Hence, point clouds exceeding 20480 points were downsampled to 20 480, whereas those with fewer than 20480 points were maintained.
未处理的数据集包含代表罐子和各种周围物体的点云。使用 CloudCompare Stereo 软件,手动消除这些无关点,从而仅隔离与植物相关的点云。消除噪声的进一步改进涉及使用直通和统计异常值去除滤波器。为了保留每种植物的形态特征,同时最小化点云数据库的数据规模,采用了最远点采样方法。因此,超过20480点的点云被下采样到20480,而少于20480点的点云被保留。
The number of point clouds from various maize varieties having different numbers of leaves is listed in Table 1. In 2019, owing to manual operations, a batch of stem point-cloud data only retained half of the full point cloud. We included these data in our dataset to test the power of our method to handle incomplete point clouds.
不同叶数的各玉米品种的点云数量如表1所示。2019年,由于人工操作,一批茎点云数据仅保留了完整点云的一半。我们将这些数据包含在数据集中,以测试我们的方法处理不完整点云的能力。
In this study, our own Label3DMaize(Miao et al., 2021a) toolkit (Fig. 1) was used for semantic and instance annotations of maize point clouds. This toolkit achieves semiautomatic point cloud segmentation and annotation at different growth stages through a series of operations, including stem, coarse, fine and sample-based segmentations. Users only need to interactively select some key areas in the core point cloud (e.g. bottom or top of stems and the highest point of each non-stem organ), and the software automatically segments those instances using a series of unsupervised segmentation algorithms. If some areas are not accurately annotated, the toolkit provides a fine segmentation function that enables the correction of misclassified points through simple manual interactions. After using Label3DMaize to label the point cloud of the maize plants, the central axis of the stem point cloud coincided with the Z-axis, and the growth direction of the plant pointed towards the positive direction of the Z-axis. Each organ instance was then assigned an integer label (Fig. 1C), and the label of the stem instance was set to zero. As such, the leaf instances were assigned labels, beginning at one from the bottom to the top of the plant. The semantic label ("stem" or "leaf") of the point cloud can be determined directly from the instance label. Tests have shown that the toolkit takes approximately 4-10 min to segment a maize shoot, and the obtained annotation results are satisfactory for model training.
在本研究中,我们自己的 Label3DMaize(Miao et al., 2021a)工具包(图 1)用于玉米点云的语义和实例注释。该工具包通过茎分割、粗分割、细分割和基于样本的分割等一系列操作,实现不同生长阶段的半自动点云分割和标注。用户只需交互地选择核心点云中的一些关键区域(例如茎的底部或顶部以及每个非茎器官的最高点),软件就会使用一系列无监督分割算法自动分割这些实例。如果某些区域没有准确标注,该工具包提供了精细的分割功能,可以通过简单的手动交互来纠正错误分类的点。使用Label3DMaize对玉米植株的点云进行标注后,茎点云的中心轴与Z轴重合,且植株的生长方向指向Z轴的正方向。然后为每个器官实例分配一个整数标签(图1C),并将茎实例的标签设置为零。因此,叶子实例被分配了标签,从植物的底部到顶部从一个标签开始。点云的语义标签(“茎”或“叶”)可以直接从实例标签确定。测试表明,该工具包对玉米芽进行分割大约需要4-10分钟,获得的注释结果满足模型训练的要求。

3.2. The segmentation pipeline
3.2.分割管道

Our point-cloud enhancement pipeline includes four parts: plant annotation, physics-based plant deformation, point-cloud segmentation model training and post-processing of instance segmentation results.
我们的点云增强流程包括四个部分:植物注释、基于物理的植物变形、点云分割模型训练和实例分割结果的后处理。

3.2.1. Plant point-cloud deformation
3.2.1.植物点云变形

A physics-based elastic deformation simulation framework was used to perform point-cloud data augmentation. Our framework was developed based on the open-source VegFEM simulation library(Sin et al., 2013), which is computationally efficient.
基于物理的弹性变形模拟框架用于执行点云数据增强。我们的框架是基于开源 VegFEM 模拟库(Sin et al., 2013)开发的,计算效率很高。
The finite element method (FEM) was used to discretise the partial differential equations of solid continuum mechanics. The deformable maize plant was represented by a volumetric mesh consisting of 3D hexahedral elements. As such, the plant's point cloud was used to generate a hexahedral volumetric mesh. First, the minimum bounding box of the point cloud was calculated and segmented at a certain resolution to obtain a series of adjacent hexahedra; the volumes lacking point clouds were removed. In this study, the length of each hexahedron edge was set to 0.02 times the longest edge length of the bounding box. The semantic and instance labels of point clouds within each hexahedron were counted, and the semantic and instance labels of the hexahedron were designated as labels with the highest number of point clouds.
有限元法(FEM)用于离散固体连续介质力学的偏微分方程。可变形玉米植株由 3D 六面体元素组成的体积网格表示。因此,植物的点云用于生成六面体体积网格。首先计算点云的最小包围盒,并以一定的分辨率进行分割,得到一系列相邻的六面体;缺少点云的体积被删除。在本研究中,每个六面体边的长度设置为边界框最长边长度的0.02倍。统计每个六面体内点云的语义和实例标签,并将六面体的语义和实例标签指定为点云数量最多的标签。
In solid mechanics, three-dimensional deformable objects can be modelled using nonlinear elastic partial differential equations, which are represented by ordinary differential equations (Fig. 1):
在固体力学中,三维可变形物体可以使用非线性弹性偏微分方程进行建模,用常微分方程表示(图1):
where and represent 3D displacement, velocity and acceleration of vertices in the volume mesh, respectively. is the mass matrix, which is formed by stacking the mass of each element. In this study, the mass of each element was set to 1000. designate the internal elastic forces, and designate the external forces specified by users to drive maize plant volumetric mesh deformations. is the Rayleigh damping matrix, which is expressed by the following formula:
其中 分别表示体积网格中 顶点的3D位移、速度和加速度。 是质量矩阵,由各个元素的质量叠加而成。在本研究中,每个单元的质量设置为1000。 表示内部弹力, 表示用户指定的驱动玉米植株体积网格变形的外力。 为瑞利阻尼矩阵,用以下公式表示:
where is the tangent stiffness matrix, and are scalar parameters that can be adjusted by the user to control damping effects, is used to slow down global deformation and is used to dampen the relative deformation velocity differences, which is useful in removing temporal high-frequency instabilities. In this study, these parameters are and .
其中 是切线刚度矩阵, 是用户可以调整以控制阻尼效果的标量参数, 用于减慢全局变形, 用于抑制相对变形速度差异,这对于消除时间高频不稳定性很有用。在本研究中,这些参数是
The internal elastic forces depend on the chosen elastic material properties (i.e. isotropic, anisotropic, linear and nonlinear). In this study, the maize plant was set as a St. Venant Kirchhoff (StVK) material. Note that StVK uses the Green-Lagrange strain tensor, , and is defined by the energy density function, :
内部弹性力 取决于所选弹性材料特性(即各向同性、各向异性、线性和非线性)。在这项研究中,玉米植株被设定为圣维南基尔霍夫(StVK)材料。请注意,StVK 使用格林-拉格朗日应变张量 ,并由能量密度函数 定义:
Table 1 表格1
Number of maize sample varieties with different numbers of leaves.
具有不同叶子数量的玉米样本品种的数量。
Number of
leaves
Number of
samples
XY335 data
bulk
LD145 data
bulk
LD502 data
bulk
LD586 data
bulk
LD1281
databulk
Number of stem- 茎数-
missingdata
Number of stem- 茎数-
completedata
2 12 12 0 0 0 0 8 4
3 17 17 0 0 0 0 7 10
4 38 31 3 0 1 3 10 27
5 98 85 2 3 4 4 28 70
6 100 86 4 3 5 2 25 75
7 58 44 3 3 4 4 12 47
8 36 33 1 0 1 1 6 30
9 27 20 1 5 1 0 7 20
10 24 13 3 4 3 1 5 19
11 12 6 2 2 0 2 5 7
12 6 2 0 0 3 1 4 2
Fig. 1. Maize plant point-cloud annotation: (A) plant point cloud, (B) Label3DMaize annotation software, and (C) visualisation of point-cloud annotation results. Each organ instance is assigned an integer label and is represented by a colour.
图 1. 玉米植株点云标注:(A) 植株点云,(B) Label3DMaize 标注软件,(C) 点云标注结果可视化。每个器官实例都分配有一个整数标签并用颜色表示。
where is the identity matrix, is the vertex of the undeformed volumetric mesh and is the position of the vertex, , after deformation (i.e. ). is the Young's modulus, and is the Poisson ratio. In this study, for the stem material, and were set to and 0.01 , respectively. For the leaf material, and were set to and 0.45 , respectively. Symbol represents the trace of the strain tensor matrix, , and the colon (:) denotes the tensor contraction (i.e. element-wise dot product). The strain energy, , of any given deformation is computed by integrating the energy density, , over the entire body, :
其中 单位矩阵, 是未变形体积网格的顶点, 是顶点的位置, ,变形后(即 )。 是杨氏模量, 是泊松比。在本研究中,对于茎材料, 分别设置为 和0.01。对于叶子材质, 分别设置为 和 0.45 。符号 表示应变张量矩阵 的迹,冒号 (:) 表示张量收缩(即逐元素点积)。任何给定变形的应变能 是通过对整个主体 上的能量密度 进行积分来计算的:
The internal elastic forces, , equal to the gradient of strain energy, , with respect to the components of the deformation vector, . For STVK materials, the internal forces and tangent stiffness matrix are cubic and quadratic polynomials, respectively. The coefficients of these polynomials can then be derived analytically using the integration of the FEM shape functions over each hexahedron element.
内部弹性力 等于应变能梯度 ,相对于变形矢量的分量 。对于 STVK 材料,内力和切线刚度矩阵分别是三次多项式和二次多项式。然后可以使用每个六面体元素上的 FEM 形状函数的积分来分析导出这些多项式的系数。
To time step the deformation simulation at runtime, we numerically integrated the system from Eq. (1). In this study, an implicit Newmark integrator was used for numerical integration. Given current deformations at time , the implicit Newmark method obtained a nonlinear equation for the next deformations, , at time :
为了在运行时对变形模拟进行时间步长,我们对方程式中的系统进行了数值积分。 (1).在本研究中,使用隐式纽马克积分器进行数值积分。给定时间 的当前变形 ,隐式 Newmark 方法获得了时间 的下一个变形 的非线性方程:
where and are user-chosen parameters. In practice, these values are often set to and . To solve for , the Newton-Raphson procedure was performed iteratively, beginning with to generate the updated guesses, , using the and tangent stiffness matrix, , at the current guess, :
其中 是用户选择的参数。在实践中,这些值通常设置为 。为了求解 ,迭代执行 Newton-Raphson 过程,从 开始,使用 ,当前猜测
when falls below a certain threshold or number of iterations exceeds a certain number, the iteration process is terminated.
低于一定阈值或迭代次数超过一定次数时,迭代过程终止。
When external forces are applied to the vertices of the maize plant's volumetric mesh, the mesh undergoes elastic deformation. Using a simple trilinear interpolation, the displacement of the vertices of each hexahedral element can be transferred to a point cloud inside the element. Through deformation, we only changed the coordinates of each point, without changing its other properties such as the index, colour, and intensity. As the normal vector of a point will change following deformation, it needs to be recalculated.
当外力施加到玉米植株体积网格的顶点时,网格会发生弹性变形。使用简单的三线性插值,每个六面体单元的顶点的位移可以转移到单元内的点云。通过变形,我们只改变了每个点的坐标,而没有改变其其他属性,例如索引、颜色和强度。由于变形后点的法向量会发生变化,因此需要重新计算。
To make the deformed maize plants more diverse in terms of plant shape, we carefully designed a set of methods for applying the necessary external forces. External forces were applied to key areas in the volumetric mesh, including the tip vertices of each leaf and the bottom vertices of the stem. The stem vertices were sorted from small to large according to the Z-coordinate, and the smallest eight vertices were set as the bottom vertices of the stem. To identify the tip vertices of each leaf, we computed the Euclidean distances of the volumetric mesh vertex of each leaf instance from the Z-axis. Vertices that were greater than fourfifths of the maximum distance were defined as the leaf tip area vertices.
为了使变形玉米植株的株形更加多样化,我们精心设计了一套施加必要外力的方法。外力被施加到体积网格中的关键区域,包括每片叶子的尖端顶点和茎的底部顶点。将茎顶点按照Z坐标从小到大排序,最小的8个顶点设置为茎的底部顶点。为了识别每个叶子的尖端顶点,我们计算了每个叶子实例的体积网格顶点距 Z 轴的欧几里德距离。大于最大距离五分之四的顶点被定义为叶尖面积顶点。
During deformation, we first applied a force to the stem. We randomly selected one vertex from the bottom vertices of the stem as the leader point for applying force and designated the five neighbourhood vertices of this point as auxiliary points according to the Euclidean distance. The external force exerted on the leader point was randomly assigned values ranging from to in the -direction, from to in the Y-direction, and from to in the direction, where to represent user-defined parameters. The magnitude of these parameters directly influences the random variability of the external forces applied, leading to more pronounced deformations in the point clouds. We calculated the distances from each auxiliary point to the leader point and sorted them in ascending order. If the random force at the leader point was , then the force applied at the auxiliary point ranked was . This method of applying force causes changes in stem length and curvature.
在变形过程中,我们首先向茎施加力。我们从茎的底部顶点中随机选择一个顶点作为施力的引导点,并根据欧氏距离指定该点的五个邻域顶点作为辅助点。施加在引导点上的外力在 方向上随机分配从 的值,从 在 Y 方向上,以及在 方向上从 ,其中 代表用户定义的参数。这些参数的大小直接影响所施加的外力的随机变化,导致点云的变形更加明显。我们计算每个辅助点到引导点的距离,并按升序对它们进行排序。如果引导点处的随机力为 ,则排名为 的辅助点处施加的力为 。这种施力方法会导致阀杆长度和曲率发生变化。
As a result of stem deformation, leaf instances were deformed. Before deformation, we increased the material parameters of the stem (i.e. mass density, Young's modulus and Poisson's ratio) to and 0.45 , respectively. We set the vertices of the fixed stem top and bottom points to displacements of zero to make the stem difficult to change. We then applied a force to deform the leaves. Like the stem, we selected leader and auxiliary points from the leaf tip vertices. The method of adding external force to the auxiliary point was the same as that of the stem.
由于茎变形,叶子实例也发生了变形。在变形之前,我们将茎的材料参数(即质量密度、杨氏模量和泊松比)分别增加到 和 0.45 。我们将固定阀杆顶部和底部点的顶点设置为零位移,以使阀杆难以改变。然后我们施加力使叶子变形。与茎一样,我们从叶尖顶点选择引导点和辅助点。辅助穴位加外力的方法与茎相同。
These effects are illustrated in Fig. 2.
这些效果如图 2 所示。

3.2.2. Point-cloud segmentation model
3.2.2.点云分割模型

The deep-learning models applied to point-cloud segmentation tasks can be broadly classified into point- and voxel-based methods. To verify the effectiveness of enhanced data through their deformations in actual training models, this study used PointNet++ (Qi et al., 2017), a pointbased model, and HAIS(Chen et al., 2021), a voxel-based model) to train semantic and instance segmentation models, respectively.
应用于点云分割任务的深度学习模型可以大致分为基于点的方法和基于体素的方法。为了通过实际训练模型中的变形来验证增强数据的有效性,本研究使用基于点的模型PointNet++(Qi et al., 2017)和基于体素的模型HAIS(Chen et al., 2021)来分别训练语义和实例分割模型。
PointNet++, which was developed by Qi et al. (2017), is a deeplearning framework that is tailored for point-cloud processing. This model extends the original PointNet (Charles et al., 2017) by introducing hierarchical neural network structures to capture local features at multiple scales. Its architecture is characterised by set abstraction layers, which recursively apply PointNet to partitions of the input point set. Each layer comprises sampling, grouping, and PointNet layers. The sampling process employs farthest-point sampling to determine the centroid points, around which grouping forms local regions. Subsequently, the PointNet layers process these regions, extracting features that are invariant to the order of points. These features are aggregated across the network, enabling PointNet ++ to learn complex geometric structures within point clouds of varying densities.
PointNet++,由 Qi 等人开发。 (2017)是一个专为点云处理量身定制的深度学习框架。该模型通过引入分层神经网络结构来捕获多个尺度的局部特征,从而扩展了原始的 PointNet(Charles 等人,2017)。其架构的特点是集合抽象层,递归地将 PointNet 应用于输入点集的分区。每层都包含采样层、分组层和 PointNet 层。采样过程采用最远点采样来确定质心点,围绕该质心点分组形成局部区域。随后,PointNet 层处理这些区域,提取与点顺序不变的特征。这些特征在网络中聚合,使 PointNet ++ 能够学习不同密度的点云中的复杂几何结构。
HAIS, which was initially proposed by Chen et al. (2021), is a voxelbased model for instance segmentation in point clouds. It converts point clouds into a voxel grid to simplify the data structure for subsequent processing. At the core of HAIS is a hierarchical agglomerative clustering algorithm that incrementally merges voxels or voxel groups based on their spatial proximity and feature similarity. The process begins with individual voxels and progresses to larger clusters, with the identification of distinct object instances. A key feature of HAIS is the use of a learned feature embedding for each voxel, which ensures that clustering decisions are informed by relevant spatial and contextual data. This approach enables HAIS to segment individual instances in complex point clouds effectively.
HAIS最初由Chen等人提出。 (2021),是一种基于体素的模型,用于点云中的实例分割。它将点云转换为体素网格,以简化后续处理的数据结构。 HAIS 的核心是一种分层凝聚聚类算法,该算法根据体素或体素组的空间邻近性和特征相似性增量合并体素或体素组。该过程从单个体素开始,并逐渐发展到更大的簇,并识别不同的对象实例。 HAIS 的一个关键特征是使用每个体素的学习特征嵌入,这确保聚类决策由相关空间和上下文数据通知。这种方法使 HAIS 能够有效地分割复杂点云中的各个实例。

3.2.3. Post-processing of instance segmentation results
3.2.3.实例分割结果的后处理

The HAIS model is prone to under-segmentation, often merging adjacent leaves into a single instance. The confidence in these undersegmented areas may be low, resulting in them being marked as unsegmented instances (a label value of 0 ); however, the model may also mistakenly identify these areas as a single leaf instance. The first scenario can be directly identified by verifying whether the label value is 0 . For the second scenario, the number of point clouds for each leaf instance is counted, and the point cloud is considered as undersegmented if the count exceeds a certain threshold.
HAIS 模型容易出现分割不足的情况,通常会将相邻的叶子合并到单个实例中。这些未分割区域的置信度可能较低,导致它们被标记为未分割实例(标签值为 0 );然而,模型也可能错误地将这些区域识别为单个叶实例。第一种场景可以通过验证标签值是否为 0 来直接识别。对于第二种情况,对每个叶实例的点云数量进行计数,如果计数超过某个阈值,则认为点云未分割。
After identifying all under-segmented point clouds, we applied the distance-filed based segmentation pipeline (DFSP) (Wang et al., 2023) that was previously developed by our team for further segmentation. DFSP can rapidly locate and segment organs by encoding the global spatial structure and local connections in maize plants. Upon entering a base point (set to in this study), DFSP constructs a Minkowski distance field from other point clouds to the base point. QuickShift++ then estimates the local maximum regions from the distance field for graph segmentation.
在识别出所有未分割的点云后,我们应用了我们团队之前开发的基于距离场的分割管道(DFSP)(Wang et al., 2023)进行进一步分割。 DFSP 可以通过编码玉米植株的全局空间结构和局部连接来快速定位和分割器官。输入基点(在本研究中设置为 )后,DFSP 构造从其他点云到基点的 Minkowski 距离场。 QuickShift++ 然后根据距离场估计局部最大区域以进行图分割。
Both the HAIS model and DFSP may also result in over-segmentation. In this study, over-segmentation was identified simply by the number of point clouds contained within an instance, and over-segmented instances were directly merged into the nearest instance. It's important to note that the distance between two instances is measured as the Euclidean distance between the closest points of the instances.
HAIS模型和DFSP也可能导致过度分割。在本研究中,过度分割仅通过实例中包含的点云数量来识别,并且过度分割的实例被直接合并到最近的实例中。值得注意的是,两个实例之间的距离是通过实例最近点之间的欧几里得距离来测量的。

4. Results 4. 结果

4.1. Point-cloud deformation effect
4.1.点云变形效果

Fig. 3 presents a selection of plant point clouds that were randomly generated by the methods outlined in this paper using a physics-based deformation framework for plant point clouds. This approach enables the application of varying force locations, directions, and magnitudes to each organ, thereby allowing for completely distinct deformation trends and extents for each organ. Such control over the localised morphology of organs significantly enhances the morphological diversity of plant point clouds. In addition, as the deformation technique is rooted in elastic deformation, the application of force to localised regions of each organ affects the corresponding organ and the entire plant through the transmission of force within the medium, which results in overall plant deformation. Conversely, the deformation of an organ is also subject to the internal forces of the entire plant. This interplay between the local organ and global plant forces ensures that the stem-leaf connections, phyllotaxy, and other topological features remain accurate post-
图 3 展示了通过本文概述的方法使用基于物理的植物点云变形框架随机生成的植物点云选择。这种方法能够对每个器官施加不同的力位置、方向和大小,从而允许每个器官完全不同的变形趋势和程度。这种对器官局部形态的控制显着增强了植物点云的形态多样性。另外,由于变形技术植根于弹性变形,对每个器官的局部区域施加力,通过介质内的力传递影响相应的器官和整个植物,从而导致植物整体变形。反之,器官的变形也受到整个植物的内力的作用。局部器官和整体植物力之间的相互作用确保了茎叶连接、叶序和其他拓扑特征在事后保持准确。
A  一个
D
Fig. 2. Leaf and stem deformation process and its effects: (A) volumetric mesh of the maize plant point cloud from Fig. 1, (B) instance labels and key areas of the volumetric mesh, (C) force applied to the stem, (D) force applied to the leaves, and (E) deformed volumetric mesh and corresponding point clouds.
图 2. 叶和茎变形过程及其影响:(A) 图 1 中玉米植株点云的体积网格,(B) 体积网格的实例标签和关键区域,(C) 施加到茎上的力,(D)施加到叶子上的力,以及(E)变形的体积网格和相应的点云。
Fig. 3. Randomly generated plant point clouds using the method described. (A) and (C) show the original annotated point clouds. (B) and (D) display the point clouds deformed using the presented method. Note: Within each plant, points belonging to the same organ instance are depicted in identical colours, with different organs represented by different colours.
图 3.使用所描述的方法随机生成植物点云。 (A) 和 (C) 显示原始注释点云。 (B) 和 (D) 显示使用所提出的方法变形的点云。注意:在每株植物中,属于同一器官实例的点用相同的颜色表示,不同的器官用不同的颜色表示。
Fig. 4. Visualization of deformation results for stem and leaf organs: (A) original point cloud of the stem before deformation, (B) point clouds of the stem after five types of deformation, (C) unaltered leaf point cloud, and (D) point clouds of leaves after five types of deformation. Note: The top sub-figures in (A) and (B) show the frontal views of the stem point clouds, and the bottom sub-figures display the bottom views; in (C) and (D), green and yellow represent different leaf point clouds.
图4.茎叶器官变形结果可视化:(A)变形前茎的原始点云,(B)五种变形后茎的点云,(C)未改变的叶子点云,( D)五种变形后的树叶点云。注:(A)和(B)中顶部子图显示茎点云的正视图,底部子图显示底部视图; (C)和(D)中,绿色和黄色代表不同的叶子点云。

deformation 形变

Although some external forces of excessive magnitude may render the plants less 'realistic', the physics-based deformation preserves the local geometric characteristics of the organs well (as shown in Fig. 4). For instance, the cylindrical features of a bent stem remained intact and the undulating surface features of bent leaves are preserved, which is beneficial for the model to learn the local geometric features of organs.
尽管一些过大的外力可能会使植物不太“真实”,但基于物理的变形很好地保留了器官的局部几何特征(如图4所示)。例如,弯曲茎的圆柱形特征保持完整,弯曲叶子的起伏表面特征被保留,这有利于模型学习器官的局部几何特征。
This method provides significant flexibility, enabling the creation of maize plants with unique shapes through various external force application strategies. By adjusting the magnitude and application points of the forces, we simulated scenarios such as maize plant lodging (Fig. 5 (A)), variations in leaf inclination (Fig. 5(B)), and the state of the leaves after being bent or broken (Fig. 5(C)).
该方法提供了显着的灵活性,能够通过各种外力应用策略创造出具有独特形状的玉米植株。通过调整力的大小和作用点,我们模拟了玉米植株倒伏(图5(A))、叶片倾斜度变化(图5(B))以及叶片弯曲后的状态等情况或破损(图5(C))。

4.2. Data augmentation effects
4.2.数据增强效应

The data deformation capability of the presented method establishes it as a novel data-augmentation tool that is suitable for plant point-cloud segmentation. We compared the effects on enhancing the segmentation model performance using our method with eight commonly used pointcloud augmentation techniques (Table 5). We employed these dataaugmentation techniques to enlarge our dataset and train the segmentation models. The implementation details of these eight enhancement algorithms are provided in Appendix A.1. The setup of the training parameters is documented in Appendix A.2, while the strategy for data preparation for model training and testing is described in Appendix A.3. Table 2 details the number of training data points generated through these augmentation techniques and the segmentation accuracy of the PointNet++ and HAIS models trained with these augmented datasets. In all tables in this paper, the best result is in boldface, whereas the secondbest result is underlined.
该方法的数据变形能力使其成为一种适用于植物点云分割的新型数据增强工具。我们将使用我们的方法与八种常用的点云增强技术对增强分割模型性能的效果进行了比较(表 5)。我们采用这些数据增强技术来扩大数据集并训练分割模型。附录 A.1 提供了这八种增强算法的实现细节。训练参数的设置记录在附录 A.2 中,而模型训练和测试的数据准备策略在附录 A.3 中描述。表 2 详细介绍了通过这些增强技术生成的训练数据点的数量以及使用这些增强数据集训练的 PointNet++ 和 HAIS 模型的分割精度。在本文的所有表格中,最好的结果用黑体字表示,而次好的结果用下划线表示。
Visual representations of stem-leaf segmentation by PointNet++ and HAIS models trained with different augmentation methods are presented in Figs. 6 and 7, respectively.
图 2 和图 3 给出了使用不同增强方法训练的 PointNet++ 和 HAIS 模型的茎叶分割的视觉表示。分别为6和7。
An analysis of the segmentation metrics for PointNet , as outlined in Table 2, reveals significant findings. All augmentation methods substantially enhanced the accuracy of PointNet. Specifically, the method presented in this study achieved the highest segmentation accuracy with 22,000 data points and the second-highest accuracy with 2200 data points. Overall, the organ-level augmentation methods outperformed the plant-level methods in semantic segmentation tasks, especially with a smaller training dataset (2200 data points), where the segmentation enhancement, particularly of the stems, was markedly better for organ-level augmentations. This superiority was notably evident in the visual outcomes, especially in the stem segmentation, where the plant-level methods often resulted in under-segmentation, as illustrated in Fig. 6.
如表 2 所示,对 PointNet 分段指标的分析揭示了重要的发现。所有增强方法都大大提高了 PointNet 的准确性。具体来说,本研究中提出的方法在 22,000 个数据点上实现了最高的分割精度,在 2200 个数据点上实现了第二高的精度。总体而言,器官级增强方法在语义分割任务中优于植物级方法,特别是在训练数据集较小(2200 个数据点)的情况下,其中分割增强(尤其是茎的分割增强)对于器官级增强明显更好。这种优越性在视觉结果中尤其明显,特别是在茎分割中,植物级方法经常导致分割不足,如图 6 所示。
Instance segmentation, which is a more complex task than semantic segmentation, exhibited considerable improvement with our method, particularly in segmenting individual leaf instances, according to the HAIS instance segmentation metrics. The HAIS model trained with 22 real label data points tended to produce under-segmentation results, merging adjacent leaves into a single instance. All augmentation methods can address under-segmentation by increasing the diversity of the data. Our method with a model trained on 2200 data points surpassed models trained with other methods that were expanded to 22,000 data points. This tenfold difference in the data volume highlights the effectiveness of our approach in instance segmentation tasks. The capability of our method to deform organs in a diverse manner increases the geometric variability between adjacent organs, thereby enhancing the leaf segmentation outcomes. As can be observed from Fig. 7, the HAIS models trained with all augmentation methods still struggled to resolve under-segmentation completely, with adjacent leaf areas often being mistakenly segmented as a single instance, particularly in the lower or upper leaf regions. Compared to other augmentation methods, models trained with our method and rotation augmentation can handle under-segmentation of the upper leaves better. Although the model may not effectively segment closely packed upper leaves, identifying them as an unsegmented instance lays the groundwork for further postprocessing in these areas. The trained model of our method also significantly outperformed other augmentation techniques for the bottom leaves. As illustrated in Fig. 7, other methods mistakenly segmented multiple bottom leaves as one instance, whereas our method segmented these leaves more effectively. The experimental results indicate that the deformed data of our method enable the model to learn individual leaf features better than other augmentation methods, thereby improving the model segmentation accuracy.
实例分割是比语义分割更复杂的任务,根据 HAIS 实例分割指标,我们的方法显示出相当大的改进,特别是在分割单个叶实例方面。使用 22 个真实标签数据点训练的 HAIS 模型往往会产生分割不足的结果,将相邻的叶子合并到单个实例中。所有增强方法都可以通过增加数据的多样性来解决分割不足的问题。我们使用在 2200 个数据点上训练的模型的方法超越了使用扩展到 22,000 个数据点的其他方法训练的模型。数据量的十倍差异凸显了我们的方法在实例分割任务中的有效性。我们的方法以多种方式使器官变形的能力增加了相邻器官之间的几何变异性,从而增强了叶子分割结果。从图 7 中可以看出,使用所有增强方法训练的 HAIS 模型仍然难以完全解决分割不足的问题,相邻叶区域经常被错误地分割为单个实例,特别是在下部或上部叶区域。与其他增强方法相比,使用我们的方法和旋转增强训练的模型可以更好地处理上部叶子的欠分割。尽管该模型可能无法有效地分割紧密排列的上部叶子,但将它们识别为未分割的实例为这些区域的进一步后处理奠定了基础。我们方法的训练模型也显着优于其他底部叶子增强技术。如图所示 如图7所示,其他方法错误地将多个底部叶子分割为一个实例,而我们的方法更有效地分割这些叶子。实验结果表明,我们方法的变形数据使模型能够比其他增强方法更好地学习单个叶子特征,从而提高模型分割精度。

4.3. Segmentation results
4.3.分割结果

The maize stem-leaf semantic segmentation PointNet++ model trained in this study achieved an mIOU of , and the instance segmentation HAIS model achieved a mAP of . We also compared these two models with some of the latest unsupervised segmentation methods, including the unsupervised deep-learning semantic segmentation method GrowpSP (Z. Zhang et al., 2023), and the traditional unsupervised instance segmentation algorithms SS (Miao et al., 2021b) and DFSP (Wang et al., 2023), which are specifically designed for maize stem-leaf segmentation.
本研究训练的玉米茎叶语义分割PointNet++模型取得了 的mIOU,实例分割HAIS模型取得了 的mAP。我们还将这两个模型与一些最新的无监督分割方法进行了比较,包括无监督深度学习语义分割方法 GrowpSP (Z.Zhang et al., 2023) 和传统的无监督实例分割算法 SS (Miao et al., 2023)。 2021b) 和 DFSP (Wang et al., 2023),它们是专门为玉米茎叶分割而设计的。
Table 3 presents a comparison of the accuracy achieved in the maize stem-leaf semantic segmentation tasks between the PointNet++ model trained in this study and the GrowSP method. As a self-supervised deeplearning method, GrowSP saves time on data annotation but still requires time for model training. The accuracy of GrowSP slightly exceeded that of PointNet in indoor semantic segmentation ntasks; however, its accuracy was significantly lower than that of PointNet ++ in maize stem-leaf semantic segmentation tasks. It is evident from Fig. 8 that the segmentation results of GrowSP differed significantly from the ground truth, indicating that its self-learned features offer little aid in recognising plant organ semantics. The application pattern of unsupervised deep learning methods in plant point-cloud processing tasks remains to be explored.
表 3 比较了本研究中训练的 PointNet++ 模型与 GrowSP 方法在玉米茎叶语义分割任务中所实现的精度。作为一种自监督深度学习方法,GrowSP 节省了数据标注时间,但仍然需要时间进行模型训练。 GrowSP在室内语义分割ntasks中的准确率略高于PointNet;但在玉米茎叶语义分割任务中其准确率明显低于PointNet++。从图8中可以明显看出,GrowSP的分割结果与地面真实值存在显着差异,表明其自学习特征对识别植物器官语义几乎没有帮助。无监督深度学习方法在植物点云处理任务中的应用模式还有待探索。
Table 4 shows the accuracy comparison of the maize stem-leaf instance segmentation tasks between the HAIS model trained in this study and the SS and DFSP methods. SS and DFSP do not require learning features from data but incorporate extensive prior botanical knowledge in their algorithms, such as the topological connections between stems and leaves and the geometric features of organs. The segmentation performance of both methods was inferior to that of the the HAIS model (Fig. 9). Although DFSP and SS save time on learning from data, tuning parameters is a time-consuming task, and more importantly, their efficiency, accuracy, and stability in point-cloud segmentation could not compare with those of the HAIS model.
表4显示了本研究训练的HAIS模型与SS和DFSP方法之间的玉米茎叶实例分割任务的准确性比较。 SS 和 DFSP 不需要从数据中学习特征,而是在算法中融入广泛的先验植物学知识,例如茎和叶之间的拓扑连接以及器官的几何特征。两种方法的分割性能均低于HAIS模型(图9)。虽然DFSP和SS节省了从数据中学习的时间,但调整参数是一项耗时的任务,更重要的是,它们在点云分割方面的效率、准确性和稳定性无法与HAIS模型相比。
We used DFSP for post-processing the segmentation results of the
我们使用DFSP对分割结果进行后处理
Fig. 5. Simulation of corn plant morphologies: (A) lodging posture, (B) specially changing the inclination angle of one leaf, and (C) transforming a leaf into a broken one. Note: Within each plant, the points belonging to the same organ instance are depicted in identical colours, with different organs represented by different colours.
图5 玉米植株形态模拟:(A)倒伏姿势,(B)专门改变一片叶子的倾斜角度,(C)将一片叶子变成折断的叶子。注意:在每株植物中,属于同一器官实例的点用相同的颜色表示,不同的器官用不同的颜色表示。
Table 2 表2
Test results on PointNet ++ and HAIS.
PointNet++ 和 HAIS 上的测试结果。
Note: This study involved selecting 22 template data points from the original dataset, which were then expanded to 2200 and 22,000 data points, excluding the flipping enhancement, for training purposes. Both the semantic and instance segmentation models were trained using these enlarged datasets and evaluated against a set of 406 real-world data points. Additionally, a baseline model was trained using only the selected 22 template data points to assess the impact of various data enhancement methods.
注:本研究涉及从原始数据集中选择 22 个模板数据点,然后将其扩展到 2200 和 22,000 个数据点(不包括翻转增强)用于训练目的。语义分割模型和实例分割模型均使用这些扩大的数据集进行训练,并针对一组 406 个真实数据点进行评估。此外,仅使用选定的 22 个模板数据点来训练基线模型,以评估各种数据增强方法的影响。
Fig. 6. Visualisation of semantic segmentation results using data-augmentation methods on PointNet++: (A) ground truth, (B) flip, (C) rotation, (D) noise, (E) crop, (F) jitter, (G) elastic deformation, (H) leaf crossover, (I) LRDT, (J) our method, and (K) baseline. Note: The red and blud point clouds represent the stem and leaves, respectively.
图 6. 在 PointNet++ 上使用数据增强方法的语义分割结果可视化:(A) 地面实况,(B) 翻转,(C) 旋转,(D) 噪声,(E) 裁剪,(F) 抖动,(G) )弹性变形,(H)叶交叉,(I)LRDT,(J)我们的方法,以及(K)基线。注:红色点云和蓝色点云分别代表茎和叶。
HAIS model to address the under-segmentation of leaves, which improved the leaf mAP by . It is clear from Fig. 9 that the undersegmentation problems of the top and bottom leaves were essentially resolved. Thus, incorporating DFSP post-processing significantly enhances the segmentation capability of the HAIS model.
HAIS模型解决了叶子分割不足的问题,将叶子mAP提高了 。从图9可以清楚地看出,顶部和底部叶子的欠分割问题得到了根本解决。因此,结合DFSP后处理显着增强了HAIS模型的分割能力。

5. Discussion 5. 讨论

5.1. Plant organ segmentation dataset
5.1.植物器官分割数据集

It has been evident for a long time that the construction of a copiously labelled 3D dataset would facilitate the development of powerful 3D crop models for point-cloud feature segmentation. Consequently, the demand for such datasets containing organ-level labels has increased rapidly in recent years. Nevertheless, the efforts to produce such datasets have been quite forbidding owing to their complex structures and the
长期以来,显而易见的是,构建大量标记的 3D 数据集将有助于开发用于点云特征分割的强大 3D 裁剪模型。因此,近年来对此类包含器官级别标签的数据集的需求迅速增加。然而,由于其复杂的结构和数据,生成此类数据集的努力非常困难。
Fig. 7. Visualisation of instance segmentation results using data-augmentation methods on HAIS: (A) ground truth, (B) flip, (C) rotation, (D) noise, (E) crop, (F) jitter, (G) elastic deformation, (H) leaf crossover, (I) LRDT, (J) our method, and (K) baseline. Note: The different colours in the figure represent different instances, and black indicates unsegmented point clouds.
图 7. 在 HAIS 上使用数据增强方法的实例分割结果可视化:(A) 地面实况,(B) 翻转,(C) 旋转,(D) 噪声,(E) 裁剪,(F) 抖动,(G) )弹性变形,(H)叶交叉,(I)LRDT,(J)我们的方法,以及(K)基线。注:图中不同颜色代表不同实例,黑色表示未分割的点云。
Table 3 表3
Semantic segmentation accuracies.
语义分割准确性。
Name mIoU (%) Stem mIoU (%) 投票率 (%) Leaf mIoU (%) 叶面积 (%)
GrowSP 54.64 20.06
PointNet++
substantial labour resources needed for data labelling. Nevertheless, several useful annotation point-cloud databases have been provided, such as ROSE-X(Dutagaci et al., 2020), Rice Panicles(Gong et al., 2021), Cucumbers (Boogaard et al., 2021) and Soybean(Sun et al., 2023a). To replace the intensive labour requirements with a smart physics-based data augmentation method, this study constructed a maize annotation point-cloud database beginning with 428 data points and updated it with an innovative maize stem-leaf segmentation framework based on deformable point-cloud data to allow deep-learning model training with minimal annotated data items, which, to the best of our knowledge, is the most comprehensive open-source database currently available. We expect that this database and its augmentation technique can be used in many other agricultural study areas needing type segmentation.
数据标记需要大量劳动力资源。尽管如此,已经提供了一些有用的注释点云数据库,例如ROSE-X(Dutagaci et al., 2020)、Rice Panicles(Gong et al., 2021)、Cucumbers(Boogaard et al., 2021)和Soybean( Sun 等人,2023a)。为了用基于智能物理的数据增强方法取代密集的劳动力需求,本研究构建了一个从 428 个数据点开始的玉米注释点云数据库,并使用基于可变形点云数据的创新玉米茎叶分割框架对其进行了更新允许使用最少的注释数据项进行深度学习模型训练,据我们所知,这是目前可用的最全面的开源数据库。我们期望该数据库及其增强技术可用于许多其他需要类型分割的农业研究领域。

5.2. The diversity of our deformation method
5.2.我们变形方法的多样性

The methodology introduced in this study relies on a physics-based deformation framework to deform plant point clouds by applying external forces to organs. This approach allows for precise control over the forces, thereby enabling the setting of different application positions, directions, and magnitudes to facilitate deformation control of the
本研究中引入的方法依赖于基于物理的变形框架,通过对器官施加外力来使植物点云变形。这种方法可以精确控制力,从而能够设置不同的应用位置、方向和大小,以促进变形控制
Table 4 表4
Instance segmentation accuracy and efficiency.
实例分割的准确性和效率。
Name mAP (%) Stem mAP (%) Leaf mAP (%) Time per plant (s)
每株植物的时间(秒)
SS 33.04 13.65 52.43 143.4
DFSP 56.61 39.96 73.25 3.2
HAIS