Flow-MAE: Leveraging Masked AutoEncoder for Accurate, Efficient and Robust Malicious Traffic Classification
Flow-MAE：利用屏蔽的自动编码器进行准确、高效和强大的恶意流量分类

2.1.1 Fingerprint Methods. Fingerprint methodologies, such as FlowPrint [47], initially build a database of spatial and temporal features extracted from numerous malicious flows. Consequently, hidden malicious flows can be detected by matching fingerprints with those in the database. The effectiveness of flow classification depends on the integrity of the fingerprint database and the quality of real-time flow fingerprinting.
2.1.1 指纹识别方法。指纹方法，如FlowPrint [ 47]，最初建立了一个从众多恶意流中提取的空间和时间特征的数据库。因此，可以通过将指纹与数据库中的指纹进行匹配来检测隐藏的恶意流。流量分类的有效性取决于指纹数据库的完整性和实时流量指纹的质量。

2.1.2 Statistical Machine Learning Methods. Statistical Machine Learning (ML) approaches extract a range of spatial or temporal statistical features to form the feature vector, then employ ML algorithms to model the feature distributions of network flows. Over the past decade, the intersection of network security and ML methodologies has produced groundbreaking outcomes, including AppScanner [45], CUMUL [33], BIND [1] and k-fingerprinting (K-fp) [14].
2.1.2 统计机器学习方法。统计机器学习 （ML） 方法提取一系列空间或时间统计特征以形成特征向量，然后使用 ML 算法对网络流的特征分布进行建模。在过去的十年中，网络安全和机器学习方法的交叉产生了突破性的成果，包括 AppScanner [ 45]、CUMUL [ 33]、BIND [ 1] 和 k-fingerprinting （K-fp） [ 14]。

However, traditional ML-based methods rely on accurate feature extraction and a significant amount of labeled malicious data. Given that only a small portion of all traffic constitutes real malicious traffic, manually extracting well-labeled malicious traffic is a daunting task, let alone obtaining valuable and accurate features.
然而，传统的基于ML的方法依赖于准确的特征提取和大量标记的恶意数据。鉴于所有流量中只有一小部分构成真正的恶意流量，手动提取标记良好的恶意流量是一项艰巨的任务，更不用说获得有价值和准确的特征了。

2.1.3 Deep Learning Methods. Deep Learning (DL) constitutes an emerging paradigm to malicious traffic detection, offering end-to-end solutions that surpass traditional statistical Machine Learning (ML) techniques. These models facilitate automatic feature extraction by employing raw flow data as input and integrate feature learning and classification within a unified pipeline. Existing DL models include DF [41], which employs Convolutional Neural Networks (CNNs), FS-Net [29] using AutoEncoders, TSCRNN [26] utilizing Recurrent Neural Networks (RNNs), as well as Deeppacket [31] relying on AutoGolun [11].
2.1.3 深度学习方法。深度学习 （DL） 是恶意流量检测的新兴范式，提供超越传统统计机器学习 （ML） 技术的端到端解决方案。这些模型通过使用原始流数据作为输入来促进自动特征提取，并将特征学习和分类集成到统一的管道中。现有的深度学习模型包括采用卷积神经网络（CNNs）的DF [ 41]、使用自动编码器的FS-Net [ 29]、使用递归神经网络（RNN）的TSCRNN [ 26]，以及依赖于AutoGolun [ 11] 的Deeppacket [ 31]。

However, DL models require a considerable amount of well-labeled malicious traffic for convergence. Moreover, these models may face difficulties when applied to different flow schemas. As a result, a complete redesign or re-training of the model is needed when adapting the model to a new task or dataset, leading to sub-optimal robustness.
但是，DL 模型需要大量标记良好的恶意流量才能进行收敛。此外，这些模型在应用于不同的流程模式时可能会遇到困难。因此，在使模型适应新任务或数据集时，需要对模型进行完全的重新设计或重新训练，从而导致次优鲁棒性。

In summary, the problems of efficacy, efficiency, and robustness in the aforementioned methods remain unresolved.
总之，上述方法的功效、效率和稳健性问题仍未解决。

2.2.1 Benefits of Pre-training Models. Recently, transcending traditional ML-based approaches, pre-training techniques [53] for DL models have emerged. Pre-training models can autonomously learn unbiased data representations from vast amounts of unlabeled data and subsequently transfer to downstream tasks through fine-tuning over limited epochs on minimal labeled data. During fine-tuning, the models adapt the previously learned representations to the specific task at hand, incorporating the labeled data to refine their performance. Since the pre-training phase is independent of any specific task or labels, the representations obtained from Flow-MAE are considered unbiased, as they are learned solely from the raw data without any specific task-related bias.
2.2.1 预训练模型的好处。最近，超越了传统的基于ML的方法，出现了用于深度学习模型的预训练技术[53]。预训练模型可以从大量未标记的数据中自主学习无偏数据表示，然后通过对最少标记数据的有限时间进行微调，将数据传输到下游任务。在微调过程中，模型将先前学习的表示调整为手头的特定任务，并结合标记的数据来优化其性能。由于预训练阶段独立于任何特定任务或标签，因此从 Flow-MAE 获得的表示被认为是无偏的，因为它们仅从原始数据中学习，没有任何与任务相关的特定偏差。

These unbiased representations enable Flow-MAE to generalize well to different downstream tasks, as they capture the essential features and patterns present in the data. This transferability of representations from pre-training to fine-tuning allows to achieve high performance even with limited labeled data. Numerous pre-training models favor the transformer [48] backbone, a DL model based on a self-attention mechanism. The transformer has two primary branches: the BERT [8] (Bidirectional Encoder Representations from Transformers) Model in NLP and the MAE (Masked AutoEncoder) model [16] in CV.
这些无偏表示使 Flow-MAE 能够很好地泛化到不同的下游任务，因为它们捕获了数据中存在的基本特征和模式。这种从预训练到微调的表示的可转移性允许即使在有限的标记数据下也能实现高性能。许多预训练模型都支持转换器 [ 48] 主干，这是一种基于自注意力机制的 DL 模型。转换器有两个主要分支：NLP 中的 BERT [ 8]（来自转换器的双向编码器表示）模型和 CV 中的 MAE（掩蔽自动编码器）模型 [ 16]。

2.2.2 BERT Model in NLP. Contemporary researches have implemented BERT-like pre-training architectures for traffic classification, yielding substantial advancements. PERT [15] adapts the ALBERT [22] pre-training model for traffic classification, attaining a 93.23% F1 score on the ISCX-VPN-2016 dataset [10]. The state-of-the-art ET-BERT [27] incorporates a bespoke Byte Pair Encoding representation for encrypted traffic and two pre-training tasks based on the BERT [8] model, enhancing the F1 score to 99%. Both PERT and ET-BERT extend BERT-like models to the networking domain, exemplifying the merits of pre-training architectures when capitalizing on copious unlabeled traffic data.
2.2.2 NLP中的BERT模型。当代研究已经实现了类似 BERT 的预训练架构来进行流量分类，取得了实质性的进展。PERT [ 15] 将 ALBERT [ 22] 预训练模型用于流量分类，在 ISCX-VPN-2016 数据集上获得了 93.23% 的 F1 分数 [ 10]。最先进的 ET-BERT [ 27] 结合了用于加密流量的定制字节对编码表示和两个基于 BERT [ 8] 模型的预训练任务，将 F1 得分提高到 99%。PERT 和 ET-BERT 都将类似 BERT 的模型扩展到网络领域，在利用大量未标记的流量数据时，举例说明了预训练架构的优点。

Despite the remarkable accomplishments attained so far, it is evident that two refinements can be implemented for ET-BERT: (a) Constrained by the BERT model's input dimensions, ET-BERT employs a 128-byte sequence to represent network traffic; this is substantially shorter than the majority of network traffic (ranging from kilobytes to megabytes) and might inadequately represent extensive network traffic. Furthermore, by adopting solely packet payloads, the packet headers containing temporal and spatial features of the traffic are discarded, leading to accuracy loss. Nonetheless, the 128-byte input length still incurs elevated computation and memory overheads for the ET-BERT model. (b) ET-BERT relies upon the Byte Pair Encoding (BPE) [36] technique to convert the network traffic byte stream into BERT input tokens, utilizing a fixed dictionary, which consequently results in diminished performance under disparate traffic patterns.
尽管迄今为止取得了显著的成就，但很明显，ET-BERT可以实现两个改进：（a）受BERT模型输入维度的约束，ET-BERT采用128字节的序列来表示网络流量;这比大多数网络流量（从千字节到兆字节不等）要短得多，并且可能不足以表示广泛的网络流量。此外，通过仅采用数据包有效负载，包含流量时间和空间特征的数据包标头将被丢弃，从而导致准确性损失。尽管如此，128字节的输入长度仍然会给ET-BERT模型带来更高的计算和内存开销。（b） ET-BERT依靠字节对编码（BPE）[36]技术将网络流量字节流转换为BERT输入令牌，使用固定字典，从而导致不同流量模式下的性能下降。

2.2.3 MAE-like Model in CV. The Masked AutoEncoder (MAE) [16] constitutes a pre-training architecture with a transformer [48] backbone in the Computer Vision (CV) domain. This model employs a transformer autoencoder to learn deep latent state representations by randomly masking input image patches and subsequently reconstructing the masked regions. The image size in the MAE model surpasses the sentence length in the BERT model by orders of magnitude, thereby facilitating the processing of extended input sequences. Furthermore, MAE incorporates a versatile convolutional embedding layer as opposed to the fixed BPE encoding layer, which encodes sequences utilizing a rigid vocabulary. This feature enables the model to adapt seamlessly to varying input patterns.
2.2.3 CV中类似MAE的模型。掩码自动编码器（MAE）[16]构成了计算机视觉（CV）领域中具有转换器[48]骨干的预训练架构。该模型采用转换器自动编码器，通过随机屏蔽输入图像块并随后重建屏蔽区域来学习深度潜伏状态表示。MAE 模型中的图像大小比 BERT 模型中的句子长度高出几个数量级，从而便于处理扩展的输入序列。此外，MAE 包含一个通用的卷积嵌入层，而不是固定的 BPE 编码层，后者使用严格的词汇表对序列进行编码。此功能使模型能够无缝适应不同的输入模式。

3.1.1 Preprocessing. Flow-MAE adopts burst-level traffic representation. An initial preprocessing phase, termed burst representation (§ 3.2), converts the traffic datagram into bursts. Subsequently, the patch embedding step (§ 3.3) projects these bursts into patches, conforming to the input dimensions of the MAE model.
3.1.1 预处理。Flow-MAE采用突发级流量表示。初始预处理阶段称为突发表示 （§ 3.2），将流量数据报转换为突发数据报。随后，补丁嵌入步骤（§ 3.3）将这些突发投影到补丁中，符合 MAE 模型的输入维度。

3.1.2 Pre-training Procedure (Section 3.4). The Flow-MAE model undergoes pre-training on unlabeled background traffic, facilitating the acquisition of deep contextualized burst-level unbiased representations from large-scale unlabeled network data.
3.1.2 预训练程序（第 3.4 节）。Flow-MAE 模型对未标记的后台流量进行了预训练，有助于从大规模未标记的网络数据中获取深度上下文化的突发级无偏表示。

Flow-MAE employs a transformer autoencoder architecture, consisting of multiple encoder and decoder layers, each containing multi-head self-attention blocks. During the pre-training process, randomly selected input patches are masked. The encoder focuses exclusively on visible patches, capturing latent relationships and generating an output representation for the burst. The decoder then attempts to reconstruct the masked patches using the encoder output. The optimization goal is to minimize the reconstruction loss.
Flow-MAE 采用 transformer 自动编码器架构，由多个编码器和解码器层组成，每个层都包含多头自注意力模块。在预训练过程中，随机选择的输入补丁被屏蔽。编码器专门关注可见斑块，捕获潜在关系并生成突发的输出表示。然后，解码器尝试使用编码器输出重建屏蔽的补丁。优化目标是将重建损耗降至最低。

3.1.3 Fine-tuning Procedure (§ 3.5). Upon completing the pre-training phase, the Flow-MAE encoder is capable of producing a deep latent representation for any given burst. The classifier model inherits the encoder component of the transformer autoencoder (including its architecture and weights) and replaces the decoder with a fully connected linear classifier.
3.1.3 微调程序 （§ 3.5）。在完成预训练阶段后，Flow-MAE编码器能够为任何给定的突发生成深度潜在表示。分类器模型继承了 transformer 自动编码器的编码器组件（包括其架构和权重），并将解码器替换为全连接的线性分类器。

By fine-tuning the classifier model on a limited amount of task-specific labeled traffic, it can be customized for downstream applications. When exposed to target-specific traffic, the classifier model can accurately determine its classification.
通过在有限数量的特定于任务的标记流量上微调分类器模型，可以针对下游应用程序对其进行自定义。当暴露于特定于目标的流量时，分类器模型可以准确地确定其分类。

3.2.1 Session split. Initially, we separate the entire traffic into sessions, according to the 5-tuple: source IP address, destination IP address, protocol type, source port, and destination port. The definition of a session is as follows:
3.2.1 会话拆分。最初，我们根据 5 元组将整个流量分成会话：源 IP 地址、目标 IP 地址、协议类型、源端口和目标端口。会话的定义如下：

As session packets are bidirectional, a session includes request and response packets of a specific application exchanged between two hosts. For example, a standard network application communicates via the TCP protocol, establishing a connection between ports on two hosts. A virus might also create a connection to enable lateral movement within a local network. A session can characterize an application since different sessions may possess unique and intrinsic features that facilitate their differentiation.
由于会话数据包是双向的，因此会话包括在两个主机之间交换的特定应用程序的请求和响应数据包。例如，标准网络应用程序通过 TCP 协议进行通信，在两台主机上的端口之间建立连接。病毒还可能创建连接，以便在本地网络内实现横向移动。会话可以表征应用程序，因为不同的会话可能具有独特的内在特征，有助于区分它们。

However, in certain scenarios (e.g., P2P downloading, video calling, and live streaming), the session length can be considerable (reaching MBs or even more). Therefore, further splitting is necessary to accommodate the model input length.
但是，在某些情况下（例如，P2P 下载、视频通话和直播），会话长度可能相当长（达到 MB 甚至更多）。因此，需要进一步拆分以适应模型输入长度。

3.2.2 Burst Split. We subsequently divide each session into bursts. Formally, the definition of a burst is:
3.2.2 突发拆分。随后，我们将每个会话分成几个突发。从形式上讲，突发的定义是：

A burst can characterize the pattern of network flow transmission from the application layer perspective; while it is as descriptive as a session, it is more focused on an interval.
突发可以从应用层的角度表征网络流传输的模式;虽然它与会话一样具有描述性，但它更侧重于间隔。

We use the raw bytes in an anonymous burst B to construct an input sequence, where each burst B is represented by a sequence of ℓ_B bytes. Concretely, we anonymize the burst by removing the MAC address, IP address, and port fields from each packet header. Anonymity is essential, as bursts from the same flow category may share identical addresses or ports; these can be used as explicit identifiers and prevent models from learning latent and unbiased representations of different flow categories. Subsequently, we concatenate the remaining bytes from the burst in the packet order so that the temporal relations in the burst is preserved. We also limit the maximum instances of bursts generated from the same session to guarantee unbiased burst generation by randomly sampling from these bursts.
我们使用匿名突发 B 中的原始字节来构造一个输入序列，其中每个突发 B 由 l _B 字节序列表示。具体来说，我们通过从每个数据包标头中删除 MAC 地址、IP 地址和端口字段来匿名化突发。匿名性是必不可少的，因为来自同一流类别的突发可能共享相同的地址或端口;这些可以用作显式标识符，并防止模型学习不同流类别的潜在和无偏表示。随后，我们将数据包中突发的剩余字节连接起来，order以便保留突发中的时间关系。我们还限制了从同一会话生成的突发的最大实例，以通过从这些突发中随机采样来保证无偏突发的生成。

A burst is a temporal, application-level representation that facilitates stable, fine-grained, and representative feature extraction for various traffic types. Another advantage of burst representation is that it allows for organizing numerous flows into smaller groups, significantly reducing the sequence length for model input.
突发是一种时间、应用程序级的表示，有助于对各种流量类型进行稳定、细粒度和代表性的特征提取。突发表示的另一个优点是，它允许将多个流组织成更小的组，从而显著减少模型输入的序列长度。

However, a disparity remains between the burst and the MAE model input. Different bursts can have varying burst lengths, while the MAE model requires fixed-sized image inputs. Consequently, we introduce a burst cropping and padding scheme, as described below.
然而，突发和 MAE 模型输入之间仍然存在差异。不同的连拍可以具有不同的连拍长度，而 MAE 模型需要固定大小的图像输入。因此，我们引入了一种突发裁剪和填充方案，如下所述。

3.2.3 Burst Cropping and Padding. Cropping and padding are common techniques in transformer models for NLP and time series analysis (e.g., BERT [8], ALBERT [22], and Informer [54]). Since sentences and time series have varying lengths, input sequences are cropped or padded to a predefined maximum length ${\ell }_{\text{MAX}}$ that the model can process. A mask sequence M of equivalent length (${\ell }_M={\ell }_{\text{MAX}}$) is introduced to indicate the valid sequence from the original input in the cropped or padded sequence.

Specifically, for a burst B with a length ℓ_B smaller than the maximum length ${\ell }_{\text{MAX}}$, we pad ${\ell }_{\text{MAX}}-{\ell }_B$ zeros at its end to reach the maximum length. The corresponding positions in the mask M[ℓ_B: ℓ_MAX] are filled with 0s, while the remaining positions M[0: ℓ_B − 1] corresponding to the original sequence are filled with 1s. Otherwise, if a burst B is longer than the maximum length ${\ell }_{\text{MAX}}$, we crop it to length ${\ell }_{\text{MAX}}$ by discarding the posterior ${\ell }_B-{\ell }_{\text{MAX}}$ bytes and fill all ${\ell }_{\text{MAX}}$ positions of the mask with 1s.
具体来说，对于长度 l _B 小于最大长度 ${\ell }_{\text{MAX}}$ 的突发 B，我们在其末端填充 ${\ell }_{\text{MAX}}-{\ell }_B$ 零以达到最大长度。掩码 M[l _B ： l _MAX ] 中的相应位置填充为 0，而与原始序列相对应的其余位置 M[0： l _B − 1] 填充为 1s。否则，如果突发 B 的长度超过最大长度 ${\ell }_{\text{MAX}}$ ，我们通过丢弃后向 ${\ell }_B-{\ell }_{\text{MAX}}$ 字节将其裁剪为长度 ${\ell }_{\text{MAX}}$ ，并用 1 填充掩码的所有 ${\ell }_{\text{MAX}}$ 位置。

It is important to note that the mask used in the padding preprocessing is different from the one applied in Masked Patch Model pre-training. The bytes masked in preprocessing are not calculated or updated during model pre-training. Instead, they remain untouched throughout the pre-training, as explained in § 3.4.
需要注意的是，填充预处理中使用的掩码与掩码贴片模型预训练中使用的掩码不同。在模型预训练期间，不会计算或更新预处理中屏蔽的字节。相反，它们在整个预训练过程中保持不变，如§3.4所述。

3.3.1 Burst to Patch Embedding. In the MAE model, an operation called patch embedding initially uses a convolutional layer to encode the input image. The input image is partitioned uniformly into numerous contiguous, non-overlapping segments, referred to as patches. Following this, each patch is embedded into $R^d$, where d represents the embedding dimensionality of the MAE model. For example, within the MAE-Base model, an input image measuring 224x224 with a patch dimension of 16x16 yields 14x14=196 pacthes, each possessing d = 768 dimensions. The input image is down-sampled by 256 times. This process allows MAE to efficiently process and analyze larger inputs. On the other hand, BERT employs BPE (Byte Pair Encoding) encoding, which maps pairs of bytes to a single 768-dimensional vector. BPE encoding differs from patch embedding and does not directly support the incorporation of patch embedding techniques. Therefore, applying patch embedding to the BERT model, as used in MAE, may not be straightforward or feasible due to the fundamental differences in their encoding mechanisms.
3.3.1 突发到补丁嵌入。在 MAE 模型中，称为补丁嵌入的操作最初使用卷积层对输入图像进行编码。输入图像被均匀地划分为许多连续的、不重叠的段，称为补丁。在此之后，每个面片都嵌入到 $R^d$ 中，其中 d 表示 MAE 模型的嵌入维数。例如，在 MAE-Base 模型中，尺寸为 224x224、面片尺寸为 16x16 的输入图像生成 14x14=196 个 pacthe，每个 pacthes 具有 d = 768 维度。输入图像被下采样 256 倍。此过程使 MAE 能够有效地处理和分析更大的输入。另一方面，BERT 采用 BPE（字节对编码）编码，将字节对映射到单个 768 维向量。BPE 编码不同于补丁嵌入，不直接支持合并补丁嵌入技术。因此，将贴片嵌入应用于 MAE 中使用的 BERT 模型可能并不简单或可行，因为它们的编码机制存在根本差异。

Patch embedding is performed for two key reasons in Flow-MAE:
在 Flow-MAE 中执行补丁嵌入有两个关键原因：

(1) First, as input length increases, the computational and storage requirements for the transformer MAE grow quadratically. As a result, the transformer models typically employs a maximum input length of 128 or 256 to minimize computation and memory overhead. For extended input bursts, patch embedding significantly reduces the input sequence length, making it compatible with the MAE's input dimensions. Consequently, more comprehensive burst data, including packet headers, can be incorporated.
（1）首先，随着输入长度的增加，变压器MAE的计算和存储需求呈二次增长。因此，Transformer 模型通常采用 128 或 256 的最大输入长度，以最大限度地减少计算和内存开销。对于扩展的输入突发，贴片嵌入显著缩短了输入序列长度，使其与 MAE 的输入尺寸兼容。因此，可以合并更全面的突发数据，包括数据包标头。

(2) Second, bursts exhibit spatially localized sparsity. The convolutional patch embedding layer combines the sparse local data originating from the patches. Considering that the downstream task is traffic classification, which shows lower sensitivity to local semantics, patch embedding can be employed to extract a comprehensive high-level representation of traffic. The sparsity attribute enables the integration of the down-sampling embedding layer, yielding compact encoding and efficient high-level representation learning, while maintaining minimal computation and memory overheads.
（2）其次，暴发表现出空间局部稀疏性。卷积补丁嵌入层结合了源自补丁的稀疏局部数据。考虑到下游任务是流量分类，对局部语义的敏感度较低，因此可以使用补丁嵌入来提取流量的综合高级表示。稀疏性属性支持下采样嵌入层的集成，产生紧凑的编码和高效的高级表示学习，同时保持最小的计算和内存开销。

In the Flow-MAE framework, each input burst B constitutes a byte stream preprocessed to a fixed length ℓ_B as delineated in § 3.2. Treating each burst B as a one-dimensional input sequence $B\in N^{1\times {\ell }_B}$, a one-dimensional convolution layer with a singular input channel, d output channels, a kernel size of K = (1, k), and stride S = (1, k) is employed for patch embedding. This process yields $\lceil \frac{{\ell }_B}{k}\rceil$ patches $p_i\in R^d$. Consequently, patch embedding uniformly subdivides a burst B into $\left|P\right|=\lceil \frac{{\ell }_B}{k}\rceil$ non-overlapping segments $P$.
在 Flow-MAE 框架中，每个输入突发 B 构成一个字节流，预处理为固定长度的 l _B ，如第 3.2 节所述。将每个突发 B 视为一维输入序列 $B\in N^{1\times {\ell }_B}$ ，采用具有奇异输入通道、d 输出通道、核大小为 K = （1， k） 和步幅 S = （1， k） 的一维卷积层进行补丁嵌入。此过程生成 $\lceil \frac{{\ell }_B}{k}\rceil$ 补丁 $p_i\in R^d$ 。因此，贴片嵌入均匀地将突发 B 细分为 $\left|P\right|=\lceil \frac{{\ell }_B}{k}\rceil$ 非重叠段 $P$ 。

Owing to the potential padding of bytes to burst B during preprocessing, it becomes essential to ascertain whether a patch is derived from the padded or original bytes in burst B. This distinction enables the calculation of attention scores solely from the original bytes, avoiding disruptions from padded bytes. An attention mask $M$ is generated from M:
由于在预处理过程中可能会填充字节到突发 B，因此必须确定补丁是从突发 B 中的填充字节还是原始字节派生而来的。这种区别使得仅从原始字节计算注意力分数，避免了填充字节的中断。从 M 生成一个注意力掩码 $M$ ：

Per eq. (4), the patch mask m_i assumes a value of 1 provided that at least one original byte is present within the corresponding patch p_i. Conversely, the patch mask m_i is assigned a value of 0 if all bytes within patch p_i are padded.
根据方程（4），如果相应的补丁 p _i 中至少存在一个原始字节，则补丁掩码 m _i 假定值为 1。相反，如果修补程序 p _i 中的所有字节都已填充，则为修补程序掩码 m _i 分配值 0。

3.3.2 Positional Embedding. Datagram transmission order may serve as a distinguishing characteristic for certain malware activities. Given that the self-attention block within the transformer layer treats each patch as equivalent, positional embedding is implemented to introduce positional biases into patches. This approach represents the positional information within the sequence and enables the model to discern the spatial relationships among patches.
3.3.2 位置嵌入。数据报传输order可以作为某些恶意软件活动的区别特征。鉴于变压器层内的自注意力模块将每个贴片视为等价，因此实现了位置嵌入以将位置偏差引入贴片中。这种方法表示序列中的位置信息，并使模型能够辨别斑块之间的空间关系。

The positional embedding value for patch p_i is represented by $\text{Pos}\left(i\right)\in R^h$, depending on the relative position of p_i within the patches $P$. $\text{Pos}(\cdot )$ signifies the embedding function.
patch p _i 的位置嵌入值由 $\text{Pos}\left(i\right)\in R^h$ 表示，具体取决于 p _i 在 中的相对position值 $P$ 。 $\text{Pos}(\cdot )$ 表示嵌入函数。

3.4.1 Patch Masking. Patch masking randomly masks each patch with a probability of $r(0<r<1)$. A Shuffle operation firstly permutes all patches $P$ at random. Following this, the shuffled patches are divided into two subsets: the final $\lfloor r\left|P\right|\rfloor$ elements constitute the masked patches $P_m$, while the remaining patches form the visible patches $P_v$.
3.4.1 补丁掩蔽。补丁掩码随机屏蔽每个补丁，概率为 $r(0<r<1)$ 。操作 Shuffle 首先 $P$ 随机排列所有补丁。在此之后，洗牌后的补丁分为两个子集：最终 $\lfloor r\left|P\right|\rfloor$ 元素构成掩码补丁 $P_m$ ，而其余补丁形成可见补丁 $P_v$ 。

Only the visible patches $P_v$ are incorporated into the encoder for representation learning, with the masked patches $P_m$ being disregarded. The shuffle operation randomly permutes all the patches in the dataset. This step introduces randomness and diversity into the training process. By shuffling and masking the patches, the model is exposed to different patch arrangements, ensuring that it learns robust representations that are invariant to the specific ordering of patches. This helps prevent the model from relying solely on the spatial arrangement of patches and encourages it to capture more meaningful and generalizable features.
只有可见的补丁 $P_v$ 被合并到编码器中，用于表示学习，而屏蔽的补丁 $P_m$ 将被忽略。随机排列操作随机排列数据集中的所有补丁。此步骤将随机性和多样性引入训练过程。通过对补丁进行洗牌和屏蔽，模型将暴露于不同的补丁排列中，从而确保它学习与补丁的特定顺序不变的鲁棒表示。这有助于防止模型仅依赖于面块的空间排列，并鼓励它捕获更有意义和可泛化的特征。

Afterwards, a class patch p_cls = 0^d is concatenated preceding the visible patches, facilitating the fusion of information from all visible patches at a later stage.
之后，在可见斑块之前连接一个类补丁 p _cls = 0 ^d ，从而促进后期所有可见斑块的信息融合。

3.4.2 Masked Encoder. The transformer MAE capture latent inter-patch relationships within visible patches $P_{cv}$, subsequently reconstructing masked patches $P_m$. These bidirectional transformer autoencoder blocks facilitate information capture from both antecedent and subsequent visible patches. The self-attention mechanism autonomously derives suitable representations from input patches, expediting the recovery of masked patches.
3.4.2 蒙面编码器。转换器MAE捕获可见斑块内潜在的斑块间关系 $P_{cv}$ ，随后重建掩蔽斑块 $P_m$ 。这些双向变压器自动编码器模块有助于从前一个和后续可见斑块中捕获信息。自注意力机制自主地从输入补丁中导出合适的表示，从而加快了屏蔽补丁的恢复。

Delving deeper, the transformer encoder $E=\left\{{\text{Encoder}}_i(\cdot )\mid i\in N\right\}$ incorporates |E| layers, with each layer ${\text{Encoder}}_i(\cdot )$ discerning latent relationships among input patches $P_i$ and outputting an deep latent representation $P_{i+1}$.
更深入地研究，变压器编码器 $E=\left\{{\text{Encoder}}_i(\cdot )\mid i\in N\right\}$ 集成了 |E|层，每一层 ${\text{Encoder}}_i(\cdot )$ 都辨别输入补丁之间的潜在关系 $P_i$ ，并输出深度潜在表示 $P_{i+1}$ 。

While maintaining the same shape as $P_i$, each patch $p_{i+1,j}\in P_{i+1}$ fuses information from all patches in $P_i$ via the self-attention encoder block.
在保持与 $P_i$ 相同的形状的同时，每个 Patch $p_{i+1,j}\in P_{i+1}$ 通过自注意力编码器模块融合 $P_i$ 来自所有 Patch 的信息。

Note that the attention mask $M_v$ is provided to the self-attention block within each layer of transformer encoders ${\text{Encoder}}_i$, serving to mask the padded patches. $M_v$ conceals the attention scores of padded patches, ensuring that only the original patches will be attended to and precluding interference from padded patches.
请注意，注意力掩码 $M_v$ 提供给每层变压器编码器 ${\text{Encoder}}_i$ 内的自注意力模块，用于掩蔽填充的补丁。 $M_v$ 隐藏填充补丁的注意力分数，确保只关注原始补丁，并排除来自填充补丁的干扰。

3.4.3 Decoder. With the final layer's latent representation of the visible patches, denoted as $P_{\left|E\right|}$, we can reconstruct the masked patches using the transformer decoder. The decoder's architecture resembles that of the encoder, consisting of multiple layers with self-attention blocks. There are two key differences: (1) the decoder is relatively lightweight, and (2) the number of patches is $\left|P\right|$.
3.4.3 解码器。通过最后一层可见斑块的潜在表示，表示为 $P_{\left|E\right|}$ ，我们可以使用转换器解码器重建掩蔽的斑块。解码器的架构类似于编码器的架构，由具有自注意力块的多层组成。有两个关键区别：（1）解码器相对轻量级，（2）补丁数量为 $\left|P\right|$ 。

To reconstruct the masked patches, we initially insert placeholders at the masked patch positions and return the visible patches to their original locations. This operation, referred to as Unshuffle, produces $P_0^{\prime }$.
为了重建蒙版补丁，我们首先在蒙版补丁位置插入占位符，并将可见补丁返回到其原始位置。此操作称为 Unshuffle ，生成 $P_0^{\prime }$ 。

The placeholder patches, represented by $\left|P_m\right|$ zero vectors 0^d, are soon replaced by the decoder with recovered masked patches. The Unshuffle operation restores the patch count to $\left|P\right|=\left|P_v\right|+\left|P_m\right|$. Subsequently, a positional embedding is added to $P^{\prime }$. The decoder layers, denoted as $D=\left\{{\text{Decoder}}_i(\cdot )\mid i\in N\right\}$, are then employed to replace the placeholders with recovered masked patches. The decoding process proceeds as follows:
由 $\left|P_m\right|$ 零向量 0 ^d 表示的占位符补丁很快被具有恢复的屏蔽补丁的解码器替换。该 Unshuffle 操作将修补程序计数恢复到 $\left|P\right|=\left|P_v\right|+\left|P_m\right|$ 。随后，将 $P^{\prime }$ 位置嵌入添加到 中。然后使用解码器层（表示为 $D=\left\{{\text{Decoder}}_i(\cdot )\mid i\in N\right\}$ ）将占位符替换为恢复的掩码补丁。解码过程按如下方式进行：

The final decoder layer's latent representation, $P_{\left|D\right|}^{\prime }$, has its placeholder patches extracted and projected to conform to the input burst's original shape, represented as $P_r^{\prime }$.
最终解码器层的潜在表示形式 $P_{\left|D\right|}^{\prime }$ ，其占位符块被提取并投影为符合输入突发的原始形状，表示为 $P_r^{\prime }$ 。

The pre-training loss is calculated as the Mean Square Error (MSE) between the restored patches $P_r^{\prime }$ and the original masked patches $P_m$:
训练前的损失计算为恢复的补丁 $P_r^{\prime }$ 和原始屏蔽的补丁之间的均方误差 （MSE $P_m$ ）：

4.1.1 Flow-MAE Implementation.
4.1.1 Flow-MAE 实现。

We adopt the Vision Transformer base (ViT-base) [9] model as the masked autoencoder backbone in Flow-MAE. Vit-base incorporates 12 encoder layers, and 12 decoder layers, and adopts a 768-dimension embedding for each input patch.
我们采用Vision Transformer基站（ViT-base）[9]模型作为Flow-MAE中的掩蔽自动编码器主干。Vit-base 包含 12 个编码层和 12 个解码层，每个输入补丁采用 768 维嵌入。

Table 1 presents the parameter configurations for the pre-training and fine-tuning experiments. The Flow-MAE model comprises |E| = 12 transformer encoder layers, each featuring a self-attention block with h_E = 2 heads. The pre-training Masked Patch Model decoder comprises |D| =12 layers and h_D=12 self-attention heads. The input burst length l_MAX is set at 1024 bytes. The convolution embedding layer, employing a stride of s=32 and kernel size k=32, projects the burst into |P| =32 patches of d=768 dimensions. During pre-training, with a mask ratio r_m=0.15, |P_m| =4 of the 32 total patches are masked, leaving the remaining |P_v| =28 patches visible. The attention drop rate r_a and forward drop rate r_f are assigned values of 0 during pre-training and 0.1 in fine-tuning, respectively. In total, the Flow-MAE model contains 85.12 M parameters and a model size of 450 MB.
表 1 显示了预训练和微调实验的参数配置。Flow-MAE 模型包括 |E|= 12 个变压器编码器层，每个层都有一个 h _E = 2 个磁头的自注意力块。预训练掩码补丁模型解码器包括 |D|=12 层和 h _D =12 个自我关注头。输入突发长度 l _MAX 设置为 1024 字节。卷积嵌入层采用 s=32 的步幅和核大小 k=32，将突发投影到 |P|=32 个 d=768 维的补丁。在预训练期间，掩模比 r _m =0.15，|P _m |=32 个补丁中有 4 个被屏蔽，剩下的 |P _v |=28 个可见的补丁。在预训练期间，注意力下降率 r _a 和前向下降率 r _f 的值分别为 0，在微调时分别为 0.1。Flow-MAE 模型总共包含 85.12 M 参数，模型大小为 450 MB。

All experiments were conducted on a testbed equipped with an i7-12700K CPU (8 P-cores @4.7GHz and 4 E-cores @3.6GHz), 64 GB DDR5 DRAM (@6000MT/s), and two NVIDIA GeForce 3090Ti GPUs (24 GB of GDDR6X memory each). The software environment of the testbed includes Ubuntu 22.04.1 LTS (kernel 5.15.0-50), Python 3.8.13, PyTorch 1.12.0, and CUDA 11.6.
所有实验均在配备 i7-12700K CPU（8 个 P 核 @4.7GHz 和 4 个 E 核 @3.6GHz）、64 GB DDR5 DRAM （@6000MT/s） 和两个 NVIDIA GeForce 3090Ti GPU（每个 24 GB GDDR6X 内存）的测试平台上进行。测试平台的软件环境包括 Ubuntu 22.04.1 LTS（内核 5.15.0-50）、Python 3.8.13、PyTorch 1.12.0 和 CUDA 11.6。

4.1.2 Datasets. 4.1.2 数据集。

The experiments are conducted using the following datasets, as summarized in Table 2:
实验使用以下数据集进行，如表2所示：

1. CIC-IDS2018 [38]: An intrusion detection system evaluation dataset collected by the Canadian Institute for Cybersecurity (CIC). The CIC-IDS2018 dataset comprises benign background traffic and seven distinct malicious scenarios: Brute-force, Heartbleed, Botnet, DoS, DDoS, Web attacks, and infiltration, collected over a 10-day period. The network infrastructure includes 50 malicious machines in a local network, as well as 420 victim machines and 30 victim servers distributed across five departments.
   CIC-IDS2018 [ 38]：加拿大网络安全研究所（CIC）收集的入侵检测系统评估数据集。CIC-IDS2018 数据集包括良性后台流量和七种不同的恶意场景：暴力破解、Heartbleed、僵尸网络、DoS、DDoS、Web 攻击和渗透，收集时间为 10 天。网络基础设施包括本地网络中的 50 台恶意计算机，以及分布在五个部门的 420 台受害计算机和 30 台受害服务器。
   
2. USTC-TFC-2016 [50]: A dataset containing encrypted traffic from 10 malware and 10 benign applications. This well-recognized dataset was published by a researcher from the University of Science and Technology of China (USTC) and has been extensively adopted by researchers.
   USTC-TFC-2016 [ 50]：包含来自 10 个恶意软件和 10 个良性应用程序的加密流量的数据集。这个广受认可的数据集由中国科学技术大学（USTC）的研究人员发表，并已被研究人员广泛采用。
   
3. ISCX-VPN-2016-App [10]: ISCX-VPN-App is a dataset that contains encrypted network traffic collected by the Canadian Institute for Cybersecurity (CIC). The dataset focuses on Virtual Private Networks (VPNs) used for network communication. VPNs are widely used for intrusion purposes as they allow users to bypass censorship and hide their location through protocol obfuscation and proxy mode. The ISCX-VPN-App dataset is classified based on applications and consists of 17 different applications.
   ISCX-VPN-2016-App [ 10]： ISCX-VPN-App 是一个数据集，其中包含加拿大网络安全研究所 （CIC） 收集的加密网络流量。该数据集侧重于用于网络通信的虚拟专用网络 （VPN）。VPN被广泛用于入侵目的，因为它们允许用户绕过审查并通过协议混淆和代理模式隐藏他们的位置。ISCX-VPN-App 数据集根据应用程序进行分类，由 17 个不同的应用程序组成。
   
4. ISCX-VPN-2016-Service [10] ISCX-VPN-Service is another dataset within the ISCX-VPN-2016 collection. Similar to ISCX-VPN-App, this dataset contains encrypted network traffic collected by the CIC. It specifically focuses on the service level of VPNs. The ISCX-VPN-Service dataset is classified based on 12 different services related to VPN usage. By studying this dataset, researchers can gain insights into the characteristics and behavior of various VPN services.
   ISCX-VPN-2016-服务 [ 10] ISCX-VPN-Service 是 ISCX-VPN-2016 集合中的另一个数据集。与 ISCX-VPN-App 类似，此数据集包含 CIC 收集的加密网络流量。它特别关注VPN的服务水平。ISCX-VPN-Service 数据集根据与 VPN 使用相关的 12 种不同服务进行分类。通过研究这个数据集，研究人员可以深入了解各种VPN服务的特征和行为。
   
5. ISCX-Tor-2016 [23]: This dataset comprises traffic from 16 applications utilizing the Onion Router (Tor) for encrypted communications. Tor obscures data between the sender and the receiver through a distributed routing network. Some intruders may use Tor to conceal their identity and activities, as the resulting obfuscation makes tracking and traffic classification challenging.
   ISCX-Tor-2016 [ 23]：该数据集包括来自 16 个应用程序的流量，这些应用程序使用洋葱路由器 （Tor） 进行加密通信。Tor 通过分布式路由网络掩盖发送方和接收方之间的数据。一些入侵者可能会使用 Tor 来隐藏他们的身份和活动，因为由此产生的混淆使跟踪和流量分类具有挑战性。
   
6. Cross-Platform [46]: This dataset includes encrypted traffic from Android applications, featuring 215 apps from the top 100 Apps in the US, China, and India. It is representative of encrypted traffic from current applications, as it covers a wide range of applications worldwide. Moreover, it has the highest number of classes among all datasets used in the experiment, making it more challenging than the others.
   跨平台 [ 46]：该数据集包括来自 Android 应用程序的加密流量，包含来自美国、中国和印度前 100 名应用程序的 215 个应用程序。它代表了来自当前应用程序的加密流量，因为它涵盖了全球广泛的应用程序。此外，在实验中使用的所有数据集中，它的类数量最多，使其比其他数据集更具挑战性。
   

Flow-MAE is pre-trained on a subset of unlabeled background traffic from CIC-IDS2018 [38], specifically the traffic collected on Tuesday, 20-02-2018, after removing malicious traffic directed towards the victim host 172.31.69.25. This subset dataset from 20-02-2018 consists of 451 raw pcap files, totaling 57.34 GB, and generates a pre-training dataset of 3.86 million bursts. To evaluate Flow-MAE's effectiveness and robustness, the model is fine-tuned for malicious traffic classification experiments on the first five public datasets mentioned above. The pre-training dataset is significantly larger in size compared to the fine-tuning dataset. For instance, the fine-tuning data on CIC-IDS2018 consists of approximately 10GB of malicious traffic. Additionally, the Cross-Platform (Android) dataset [46], which includes only benign traffic from 215 apps, is employed to validate Flow-MAE's transferability and robustness in the encrypted flow classification task involving large categories. However, to ensure an adequate amount of data for analysis, a data size threshold of at least 20MB was imposed. As a result, only 71 out of the 215 apps were included in the evaluation. This task demonstrates the model's capacity to adapt to different classification scenarios, even when handling a larger number of classes and diverse data types.
Flow-MAE 在 CIC-IDS2018 [ 38] 的未标记后台流量子集上进行了预训练，特别是在 2018 年 2 月 20 日星期二删除指向受害者主机 172.31.69.25 的恶意流量后收集的流量。2018 年 2 月 20 日的子集数据集由 451 个原始 pcap 文件组成，总计 57.34 GB，并生成 386 万次突发的预训练数据集。为了评估 Flow-MAE 的有效性和鲁棒性，该模型针对上述前五个公共数据集上的恶意流量分类实验进行了微调。与微调数据集相比，预训练数据集的大小要大得多。例如，CIC-IDS2018 上的微调数据由大约 10GB 的恶意流量组成。此外，采用跨平台（Android）数据集[46]，仅包含来自215个应用的良性流量，用于验证Flow-MAE在涉及大型类别的加密流量分类任务中的可转移性和鲁棒性。但是，为了确保有足够的数据量进行分析，施加了至少 20MB 的数据大小阈值。因此，在 215 个应用程序中，只有 71 个被纳入评估。此任务演示了模型适应不同分类方案的能力，即使在处理大量类和不同的数据类型时也是如此。

As outlined in the preprocessing procedure section (§ 3.2.2), the burst's IP address and port fields are anonymized to prevent explicit identification information from interfering with representation learning. Anonymous TCP and UDP sessions from each class are selected to create labeled burst datasets, which are then divided into training and testing sets in an 9:1 ratio.
如预处理过程部分（§ 3.2.2）所述，突发的 IP 地址和端口字段是匿名的，以防止显式识别信息干扰表示学习。从每个类中选择匿名 TCP 和 UDP 会话以创建标记的突发数据集，然后以 9：1 的比例将其划分为训练集和测试集。

4.1.3 Baselines. Twelve state-of-the-art generic malicious traffic detection methods are utilized as baselines: AppScanner [45], CUMUL [33], BIND [1], K-fp [14], FlowPrint [47], DF [41], FS-Net [29], GraphDApp [40], TSCRNN [26], Deeppacket [31], PERT [15], and ET-BERT [27]. These baseline methods represent a diverse range of approaches in the literature: (1) they encompass both traditional machine learning-based and deep learning-based methods; (2) they employ a wide variety of features, such as flow statistics, time series, raw packet headers, and raw payloads.
4.1.3 基线。12 种最先进的通用恶意流量检测方法被用作基线：AppScanner [ 45]、CUMUL [ 33]、BIND [ 1]、K-fp [ 14]、FlowPrint [ 47]、DF [ 41]、FS-Net [ 29]、GraphDApp [ 40]、TSCRNN [ 26]、Deeppacket [ 31]、PERT [ 15] 和 ET-BERT [ 27]。这些基线方法代表了文献中的各种方法：（1）它们包括传统的基于机器学习和基于深度学习的方法;（2） 它们采用了多种功能，例如流量统计、时间序列、原始数据包标头和原始有效负载。

4.1.4 Evaluation Metrics. We evaluate the performance of Flow-MAE using four typical metrics in the literature: Accuracy (AC), Precision (PR), Recall (RC), and F₁-score (F₁). These metrics are defined as follows:
4.1.4 评估指标。我们使用文献中的四个典型指标来评估 Flow-MAE 的性能：准确率 （AC）、精确度 （PR）、召回率 （RC） 和 F ₁ 分数 ₁ （F）。这些指标定义如下：

• Accuracy (AC): $AC=\frac{T_P+T_N}{T_P+T_N+F_P+F_N}$, the ratio of the number of correctly classified instances (True Positives T_p and True Negatives T_N) to all instances in the test dataset, which comprises all input bursts T_P + T_N + F_P + F_N.
  准确度 （AC）： $AC=\frac{T_P+T_N}{T_P+T_N+F_P+F_N}$ ，正确分类的实例数（真阳性 T _p 和真阴性 T _N ）与测试数据集中所有实例的比率，包括所有输入突发 T _P + T _N + F _{P N} + F。
  
• Precision rate: $PR=\frac{T_P}{T_P+F_P}$, the ratio of the number of correctly reported instances (True Positives T_p) to the number of reported instances T_P + F_P.
  准确率： $PR=\frac{T_P}{T_P+F_P}$ ，正确报告的实例数（真阳性 T _p ）与报告的实例数 T _P + F 的比值 _P 。
  
• Recall Rate: $RC=\frac{T_P}{T_P+F_N}$, the ratio of the number of correctly reported instances to the number of correct instances T_p + F_N.
  召回率： $RC=\frac{T_P}{T_P+F_N}$ ，正确报告的实例数与正确实例数的比值 _N T _p + F 。
  
• F₁-score: $F_1=\frac{2\times PR\times RC}{PR+RC}$, where PR is the Precision rate and RC is the Recall rate.
  F ₁ -score： $F_1=\frac{2\times PR\times RC}{PR+RC}$ ，其中 PR 是 Precision 率，RC 是 Recall 率。
  

To account for imbalanced traffic categories and avoid biased results, Macro Average [44] is employed to calculate the mean values of AC, PR, RC, and F1 for all traffic categories. Moreover, to further mitigate potential biases, a five-fold validation is performed on each dataset.
为了解释不平衡的流量类别并避免有偏差的结果，采用宏观平均值 [ 44] 来计算所有流量类别的 AC、PR、RC 和 F1 的平均值。此外，为了进一步减少潜在的偏差，对每个数据集进行了五重验证。

4.2.1 CIC-IDS-2018. As depicted in Table 3, in the largest dataset, CIC-IDS-2018, Flow-MAE attains up to 69.9%, 61.1%, 60.3%, and 60.7% improvements in accuracy, precision, recall, and F₁ compared to existing solutions, respectively.
4.2.1 CIC-IDS-2018年。如表3所示，在最大的数据集CIC-IDS-2018中， ₁ 与现有解决方案相比，Flow-MAE在准确度、精密度、召回率和F方面分别提高了69.9%、61.1%、60.3%和60.7%。

Compared to the state-of-the-art method, ET-BERT, Flow-MAE improves accuracy, precision, recall, and F₁ by 0.03%, 0.26%, 0.26%, and 0.25%, respectively. This improvement can be primarily attributed to the patch embedding layer, which allows for a longer input length (1024 bytes) than in ET-BERT (128 bytes), resulting in more comprehensive flow information being used. Moreover, the Flow-MAE model utilizes both packet header and payload to generate the bursts as input, capturing unique interaction patterns in the packet headers of distinct flows.
与最先进的ET-BERT方法相比，Flow-MAE的准确度、精密度、召回率和F率分别 ₁ 提高了0.03%、0.26%、0.26%和0.25%。这种改进主要归因于补丁嵌入层，它允许比 ET-BERT（128 字节）更长的输入长度（1024 字节），从而使用更全面的流信息。此外，Flow-MAE 模型利用数据包标头和有效负载来生成突发作为输入，从而捕获不同流的数据包标头中的独特交互模式。

Flow-MAE also surpasses several other existing solutions. For example, CUMUL achieves an accuracy of merely 0.586 and performs as poorly as a random guess model, making it unsuitable for malicious traffic detection in the CIC-IDS-2018 dataset. Although FlowPrint and DF produce slightly better results than CUMUL, their accuracy remains below 0.80, exhibiting a 19.8%-21.1% performance decrease compared to Flow-MAE. Other solutions display intermediate results, with accuracy ranging from a minimum of 0.8661 (BIND) to a maximum of 0.9661 (Deeppacket), indicating a gap of 12.9% and 2.97% when compared with Flow-MAE.
Flow-MAE 还超越了其他几种现有解决方案。例如，CUMUL的准确率仅为0.586，其性能与随机猜测模型一样差，因此不适合CIC-IDS-2018数据集中的恶意流量检测。尽管 FlowPrint 和 DF 产生的结果略好于 CUMUL，但它们的准确度仍低于 0.80，与 Flow-MAE 相比，性能下降了 19.8%-21.1%。其他解决方案显示中间结果，精度范围从最小 0.8661 （BIND） 到最大 0.9661 （Deeppacket），与 Flow-MAE 相比，差距分别为 12.9% 和 2.97%。

FS-Net, similar to ET-BERT, utilizes only the payload as input but employs lightweight AutoEncoders without pre-training, rendering it inferior to both ET-BERT and Flow-MAE. This comparison demonstrates that a larger model is more robust in fitting the downstream classification task than a lightweight model. Furthermore, a pre-trained model benefits the fine-tuning process on the classification task with higher convergence than a randomly initialized model.
FS-Net与ET-BERT类似，仅使用有效载荷作为输入，但使用轻量级自动编码器，无需预训练，因此不如ET-BERT和Flow-MAE。这种比较表明，较大的模型在拟合下游分类任务方面比轻量级模型更强大。此外，与随机初始化的模型相比，预训练模型有利于分类任务的微调过程，收敛性更高。

4.2.2 USTC-TFC-2016. The performance of Flow-MAE on the USTC-TFC-2016 [50] dataset, as displayed in Table 3, demonstrates the highest performance across all four metrics. All metrics exceed 0.998, leading to a 0.54%-0.59% improvement compared to the state-of-the-art solution, ET-BERT.
4.2.2 USTC-TFC-2016.如表3所示，Flow-MAE在USTC-TFC-2016 [ 50] 数据集上的性能在所有四个指标上都表现出最高的性能。所有指标均超过 0.998，与最先进的解决方案 ET-BERT 相比，提高了 0.54%-0.59%。

Although the dataset comprises malicious traffic with unencrypted application layer data, most models exhibit no significant enhancement over the CIC-IDS-2018 encrypted dataset, except for PERT, which improves by 6.5%. This result is counter-intuitive, as plaintext payload might be expected to simplify classification for most models. However, this can be explained by the fact that the models treat both encrypted and unencrypted data as a data stream, and the encrypted byte sequence also contains inherent patterns of malicious traffic.
尽管该数据集包含带有未加密应用层数据的恶意流量，但大多数模型与 CIC-IDS-2018 加密数据集相比没有显着增强，但 PERT 提高了 6.5%。这个结果是违反直觉的，因为明文有效负载可能会简化大多数模型的分类。但是，这可以通过以下事实来解释：模型将加密和未加密的数据都视为数据流，并且加密的字节序列还包含恶意流量的固有模式。

Flow-MAE is capable of capture critical patterns of malicious traffic, whether encrypted or not.
Flow-MAE 能够捕获恶意流量的关键模式，无论是否加密。

4.2.3 ISCX-VPN-2016-App. Firstly, as presented Table 3, Flow-MAE achieves high scores for accuracy (0.9987), precision (0.9991), recall (0.9989), and F₁ (0.9990), with percentile advantages of 0.25%-46.28%, 0.55%-54.6%, and 0.53%-57.60% compared to existing schemes.
4.2.3 ISCX-VPN-2016-应用程序。首先，如表3所示，Flow-MAE在准确率（0.9987）、准确率（0.9991）、召回率（0.9989）和F ₁ （0.9990）方面取得了高分，与现有方案相比，百分位优势为0.25%-46.28%、0.55%-54.6%和0.53%-57.60%。

Second, Flow-MAE demonstrates robustness to variation in traffic patterns, as it does not experience any decrease in accuracy compared to USTC-TFC-2016. In contrast, other solutions, such as Appscanner, BIND, DF, and FS-Net, show a clear decline in accuracy, ranging from 21.5% to 30.0%, indicating poor robustness to traffic variations. Flow-MAE is less sensitive to dataset variation and generalizes better, indicating superior robustness compared to the state-of-the-art scheme.
其次，Flow-MAE表现出对流量模式变化的鲁棒性，因为与USTC-TFC-2016相比，它的准确性没有下降。相比之下，Appscanner、BIND、DF 和 FS-Net 等其他解决方案的准确率明显下降，从 21.5% 到 30.0% 不等，表明对流量变化的鲁棒性较差。Flow-MAE对数据集变化的敏感度较低，泛化性更好，表明与最先进的方案相比具有更好的鲁棒性。

Thirdly, the diminished accuracy in existing solutions when applied to the ISCX-VPN-2016-App dataset indicates that it is less distinguishable compared to the CIC-IDS-2018 and USTC-TFC-2016 datasets. AppScanner exhibits the poorest performance, with accuracy, precision, recall, and F1 scores registering at 0.6266, 0.4864, 0.5198, and 0.4935, respectively. This represents a decline of 30%, 45.9%, 42%, and 44.5% in comparison to the USTC-TFC-2016 dataset. Moreover, the accuracy of ET-BERT (flow) also experiences a reduction from 0.9929 to 0.8519, amounting to a 14.1% decrease.
第三，将现有解决方案应用于 ISCX-VPN-2016-App 数据集时的准确性降低表明，与 CIC-IDS-2018 和 USTC-TFC-2016 数据集相比，该数据集的可区分性较差。AppScanner 的性能最差，准确度、精确度、召回率和 F1 分数分别为 0.6266、0.4864、0.5198 和 0.4935。与 USTC-TFC-2016 数据集相比，这分别下降了 30%、45.9%、42% 和 44.5%。此外，ET-BERT（流量）的精度也从0.9929降低到0.8519，下降了14.1%。

4.2.4 ISCX-VPN-2016-Service. The ISCX-VPN-2016-Service dataset is categorized by service, which differs from the application-based categorization of the ISCX-VPN-2016-App dataset.
4.2.4 ISCX-VPN-2016-服务。ISCX-VPN-2016-Service 数据集按服务分类，这与 ISCX-VPN-2016-App 数据集的基于应用程序的分类不同。

First, as shown in Table 4, Flow-MAE achieved an accuracy of 0.9915, precision of 0.9924, recall of 0.9915, and F₁ score of 0.9917, which represents an improvement of 0.25%-76.7%, 0.33%-68.5%, 0.25%-74.6%, and 0.27%-74.9%, respectively, over the existing solutions.
首先，如表4所示，Flow-MAE的准确度为0.9915，精密度为0.9924，召回率为0.9915，F ₁ 得分为0.9917，与现有解决方案相比，分别提高了0.25%-76.7%、0.33%-68.5%、0.25%-74.6%和0.27%-74.9%。

Second, while Flow-MAE attained the best results on all four metrics, its performance on the ISCX-VPN-2016-Service was lower than on the ISCX-VPN-2016-App, with the metrics dropping from above 0.998 to below 0.992. The state-of-the-art solutions also experienced a decline in accuracy compared to their performance on the ISCX-VPN-2016-App dataset. This can be attributed to the mixing of traffic from multiple applications within each service category, and the fact that traffic from each application may appear in several services, which makes it difficult to distinguish between different categories. For example, the file transfer service category includes traffic from both sftp and skype applications.
其次，虽然Flow-MAE在所有四个指标上都取得了最好的结果，但它在ISCX-VPN-2016-Service上的表现低于ISCX-VPN-2016-App，指标从0.998以上下降到0.992以下。与ISCX-VPN-2016-App数据集上的性能相比，最先进的解决方案的准确性也有所下降。这可以归因于每个服务类别中来自多个应用程序的流量混合，以及来自每个应用程序的流量可能出现在多个服务中，这使得很难区分不同的类别。例如，文件传输服务类别包括来自 sftp 和 skype 应用程序的流量。

4.2.5 ISCX-Tor-2016. The results obtained on the ISCX-Tor-2016 Dataset are noteworthy. Flow-MAE outperforms the literature schemes with a precision, recall, and F₁ of 0.9944, 0.9941, and 0.9942, respectively, which are up to 2.6x, 2.7x, and 2.7x higher. Conversely, ET-BERT (flow) underperforms with metrics of 0.5564, 0.6448, and 0.5886, respectively, compared to its performance on the previous four datasets. The poor performance can be attributed to the multi-layer encryption and adversarial obfuscation used by Tor to conceal malicious behavior or identity, which makes it harder for other schemes to distinguish different flows. Flow-MAE, on the other hand, is not affected by encryption and obfuscation in Tor and can capture the latent property of different flows, thereby achieving higher accuracy in malicious traffic detection.
4.2.5 ISCX-Tor-2016 中。在 ISCX-Tor-2016 数据集上获得的结果值得注意。Flow-MAE 的精度、召回率和 F ₁ 分别为 0.9944、0.9941 和 0.9942，分别高出 2.6 倍、2.7 倍和 2.7 倍。相反，ET-BERT（流量）的表现不如前四个数据集的指标分别为 0.5564、0.6448 和 0.5886。性能不佳可归因于 Tor 用于隐藏恶意行为或身份的多层加密和对抗性混淆，这使得其他方案更难区分不同的流量。另一方面，Flow-MAE 不受 Tor 加密和混淆的影响，可以捕获不同流量的潜在属性，从而在恶意流量检测中实现更高的准确性。

4.2.6 Cross-Platform (Android). We employed the Flow-MAE model to tackle the challenging task of encrypted flow classification using the Cross-Platform (Android) dataset consisting of traffic from 73 Android applications. Due to the complexity of the encryption protocol and the high number of classes, this task is particularly difficult.
4.2.6 跨平台 （Android）。我们采用 Flow-MAE 模型，使用跨平台 （Android） 数据集（来自 73 个 Android 应用程序的流量）来解决加密流分类的挑战性任务。由于加密协议的复杂性和类数众多，这项任务特别困难。

Despite not being specifically designed for encrypted flow classification, the Flow-MAE model exhibited excellent transferability. Experimental results on the Cross-Platform (Android) dataset, as shown in Table 4, indicate that Flow-MAE outperforms existing solutions by at least 4.4%, 6.3%, and 6.6% in terms of precision, recall, and F₁, respectively, with values of 0.9855, 0.9853, and 0.9853. The most significant improvements are 3.7x, 3.7x, and 3.8x, respectively.
尽管不是专门为加密流分类而设计的，但 Flow-MAE 模型表现出出色的可转移性。跨平台（Android）数据集的实验结果（如表4所示）表明，Flow-MAE在精度、召回率和F ₁ 方面分别比现有解决方案高出至少4.4%、6.3%和6.6%，值分别为0.9855、0.9853和0.9853。最显着的改进分别是 3.7 倍、3.7 倍和 3.8 倍。

Moreover, Flow-MAE achieved near-optimal accuracy compared to state-of-the-art ET-BERT (flow), with an absolute difference of only 0.13%. The model's accuracy is not affected by the diverse encrypted traffic from different applications, making it robust to everyday encryption and large classes.
此外，与最先进的ET-BERT（流量）相比，Flow-MAE实现了近乎最佳的精度，绝对差异仅为0.13%。该模型的准确性不受来自不同应用程序的各种加密流量的影响，使其对日常加密和大型类具有鲁棒性。

To conclude, the experiments conducted on the aforementioned datasets substantiate that Flow-MAE exhibits remarkable accuracy in identifying malicious traffic and robustness to disparate traffic patterns. Furthermore, in t-SNE and PCA visualizations (refer to Section A.2), the latent feature representations acquired by the Flow-MAE model are efficacious.
总而言之，在上述数据集上进行的实验证实，Flow-MAE在识别恶意流量方面表现出了非凡的准确性和对不同流量模式的鲁棒性。此外，在 t-SNE 和 PCA 可视化中（参见第 A.2 节），Flow-MAE 模型获取的潜在特征表示是有效的。

4.5.1 Positional Embedding. Our ablation analysis highlighted the importance of positional embedding within Flow-MAE, as its absence led to a decline in accuracy. Across the six datasets, the mean reduction in accuracy amounted to 2.03%, ranging from 0.458% to 18.611%. By incorporating positional embedding, the model effectively captured the temporal sequence of packets, resulting in improved accuracy. These insights emphasize the significance of considering the temporal dimensions of network traffic when developing models for traffic classification purposes.
4.5.1 位置嵌入。我们的消融分析强调了 Flow-MAE 中位置嵌入的重要性，因为缺少位置嵌入会导致准确性下降。在六个数据集中，准确度的平均降低为 2.03%，范围从 0.458% 到 18.611%。通过结合位置嵌入，该模型有效地捕获了数据包的时间序列，从而提高了准确性。这些见解强调了在开发用于流量分类目的的模型时考虑网络流量的时间维度的重要性。

4.5.2 Masked Patch Model Pre-training Task. The findings revealed that the Masked Patch Model pre-training task positively influenced Flow-MAE's performance. Specifically, the model pre-trained on the downstream classification task achieved higher accuracy compared to its non-pre-trained counterpart. The improvements in accuracy were notable, with increases of 0.125%, 1.125%, 0.89%, 1.53%, 75.1%, and 21.3% across the six datasets, respectively. The most significant enhancement occurred in the ISCX-Tor-2016 dataset, rising from 0.24285 to 0.9945, indicating a remarkable improvement of 75.1%. This observation emphasizes the necessity of pre-training the model to strengthen its ability to learn essential representations for the classification task.
4.5.2 Masked Patch Model 预训练任务。研究结果表明，Masked Patch Model 预训练任务对 Flow-MAE 的性能产生了积极影响。具体而言，与未预训练的模型相比，在下游分类任务上预训练的模型获得了更高的准确性。准确率的提高是显着的，六个数据集的准确率分别提高了0.125%、1.125%、0.89%、1.53%、75.1%和21.3%。最显着的增强发生在 ISCX-Tor-2016 数据集中，从 0.24285 上升到 0.9945，表明显着改善了 75.1%。这一观察结果强调了对模型进行预训练的必要性，以增强其学习分类任务基本表示的能力。

4.5.3 Patch Mask. The patch mask serves as a technique that enhances Flow-MAE's robustness by masking padded patches of input data. However, its effectiveness displayed variation across the six datasets utilized in our study.
4.5.3 贴片掩码。补丁掩码是一种技术，通过屏蔽输入数据的填充补丁来增强 Flow-MAE 的鲁棒性。然而，它的有效性在我们研究中使用的六个数据集中显示出差异。

Removing the patch mask had a negligible effect on the model's accuracy for the CIC-IDS-2018, USTC-TFC-2016, and Cross-Platform (Android) datasets, with changes below 0.06%. In contrast, for datasets such as ISCX-VPN-App and ISCX-Tor-2016, the patch mask exhibited a more significant impact, increasing accuracy by up to 18%.
移除贴片掩模对模型对 CIC-IDS-2018、USTC-TFC-2016 和跨平台 （Android） 数据集的准确性影响可以忽略不计，变化低于 0.06%。相比之下，对于 ISCX-VPN-App 和 ISCX-Tor-2016 等数据集，贴片掩码表现出更显着的影响，将准确性提高了 18%。

These findings suggest that the patch mask can improve the model's performance for specific datasets, while having minimal effect on others. The patch mask's effectiveness may also depend on distinct dataset attributes, such as sequence length. For example, in the CIC-IDS-2018 dataset, characterized by relatively long input sequence lengths, the patch mask's influence may be diminished.
这些发现表明，贴片掩码可以提高模型对特定数据集的性能，同时对其他数据集的影响最小。贴片掩码的有效性还可能取决于不同的数据集属性，例如序列长度。例如，在 CIC-IDS-2018 数据集中，其特征是输入序列长度相对较长，贴片掩模的影响可能会减弱。

4.5.4 Packet Headers. In experiment, we investigated the impact of truncating the first 40/76 bytes of packet headers to generate bursts, as the ET-BERT model solely utilizes the payload within packets by discarding the initial 76 bytes.
4.5.4 数据包头。在实验中，我们研究了截断数据包标头的前 40/76 字节以生成突发的影响，因为 ET-BERT 模型通过丢弃最初的 76 字节来仅利用数据包中的有效负载。

Our findings indicated that the first 40 bytes had a marginal effect on accuracy, with reductions below 1.66% across all datasets. However, the initial 76 bytes had a more significant impact on accuracy, with decreases of 16.46%, 17.116%, and 31.68% in ISCX-VPN-App, ISCX-VPN-Service, and ISCX-Tor-2016, respectively. Interestingly, we discerned that incorporating packet headers had no significant consequences for datasets featuring elongated packets, such as CIC-IDS-2018 and USTC-TFC-2016. This observation aligns with our antecedent findings concerning the patch mask's impact, as delineated in § 4.5.3. Additionally, the integration of packet headers can provide valuable insights into network traffic, including timestamps, window size, and protocol type, which may assist in differentiating between various network activities. These findings emphasize the importance of incorporating domain-specific knowledge into the model to enhance its effectiveness for specific tasks.
我们的研究结果表明，前 40 个字节对准确性的影响很小，所有数据集的减少率低于 1.66%。然而，最初的 76 个字节对准确性的影响更为显著，在 ISCX-VPN-App、ISCX-VPN-Service 和 ISCX-Tor-2016 中分别下降了 16.46%、17.116% 和 31.68%。有趣的是，我们发现合并数据包标头对具有细长数据包的数据集（例如 CIC-IDS-2018 和 USTC-TFC-2016）没有显着影响。这一观察结果与我们之前关于面罩影响的发现一致，如§4.5.3所述。此外，数据包标头的集成可以提供对网络流量的宝贵见解，包括时间戳、窗口大小和协议类型，这可能有助于区分各种网络活动。这些发现强调了将特定领域的知识纳入模型的重要性，以提高其对特定任务的有效性。

In conclusion, the ablation analysis demonstrated that the inclusion of positional embedding, MPM pre-training, and packet headers, can significantly improve Flow-MAE's performance in the classification task. On the other hand, the patch mask did not notably influence performance; however, it may still be beneficial in scenarios involving shorter packets where regularization is necessary.
综上所述，消融分析表明，包含位置嵌入、MPM预训练和数据包标头可以显著提高Flow-MAE在分类任务中的性能。另一方面，贴片掩模对性能没有明显影响;但是，在涉及需要正则化的较短数据包的方案中，它可能仍然有用。