RMIB: Representation Matching Information Bottleneck for Matching Text Representations
RMIB：匹配文本表示的表示匹配信息瓶颈

Text matching is a basic task in natural language processing (NLP). The purpose of this task is to predict the semantic relationship between two input texts. Text matching can be applied to many natural language processing scenarios. For example, in question-and-answer matching, it is necessary to judge whether the given candidate answers match the given question (Tan et al., 2016); In natural language reasoning, for a given hypothesis and premise, it is necessary to judge whether the hypothesis and premise are entailment, contradiction or neutral (Bowman et al., 2015); In information retrieval, it is necessary to determine the correlation between the searched information and the returned candidate (Reimers & Gurevych, 2019).
文本匹配是自然语言处理 （NLP） 中的一项基本任务。此任务的目的是预测两个输入文本之间的语义关系。文本匹配可以应用于许多自然语言处理方案。例如，在问答匹配中，有必要判断给定的候选答案是否与给定的问题匹配（Tan et al.， 2016）;在自然语言推理中，对于给定的假设和前提，有必要判断假设和前提是蕴涵、矛盾还是中立（Bowman et al.， 2015）;在信息检索中，有必要确定搜索信息与返回的候选者之间的相关性（Reimers&Gurevych，2019）。

Asymmetrical text matching refers to two input texts from different domains. For example, the question and the candidate answers can be viewed as being sampled from two different distributions in question-and-answer matching. One approach of asymmetrical domains text matching is to process text from different domains into text representation through deep learning models. In the common semantic space, text representations from different domains have the same dimensions, so it is easy to define semantic matching functions. The current mainstream approaches mainly focus on how to design more effective models to increase the interaction between input texts from different domains to obtain better text representations (Chen et al., 2017; Wang et al., 2017; Gong et al.; Kim et al., 2019; Yang et al., 2019). In the actual text matching, (Yu et al., 2022) finds an interesting phenomenon that the text representations from different domains are usually separated in the early stage of training. However, with the deepening of training, the text representations from different domains gradually begin to mix with each other. The explanation for this phenomenon is that there is a lexical gap between texts from different domains (Tay et al., 2018), so it is necessary to bridge this gap in the process of learning, which is similar to cross-modal matching. Existing researches (Li et al., 2020; Hazarika et al., 2020) shows that the matching model should generate domain invariant features, that is, the global distribution of text representations from different domains in the common semantic space should be as similar as possible, so as to
非对称文本匹配是指来自不同域的两个输入文本。例如，在问答匹配中，问题和候选答案可以被视为从两个不同的分布中抽样。非对称域文本匹配的一种方法是通过深度学习模型将来自不同域的文本处理为文本表示。在公共语义空间中，来自不同域的文本表示具有相同的维度，因此很容易定义语义匹配函数。目前的主流方法主要集中在如何设计更有效的模型，以增加来自不同领域的输入文本之间的交互，以获得更好的文本表示（Chen et al.， 2017;Wang等人，2017;龚等人;Kim 等人，2019 年;Yang 等人，2019 年）。在实际的文本匹配中，（Yu et al.， 2022）发现了一个有趣的现象，即来自不同领域的文本表示通常在训练的早期阶段是分开的。然而，随着训练的深入，来自不同领域的文本表示逐渐开始相互混合。对这种现象的解释是，来自不同领域的文本之间存在词汇鸿沟（Tay et al.， 2018），因此有必要在学习过程中弥合这一鸿沟，这类似于跨模态匹配。现有研究（Li et al.， 2020;Hazarika et al.， 2020）表明，匹配模型应生成域不变特征，即公共语义空间中不同域的文本表示的全局分布应尽可能相似，以便
capture the commonality between potential text representations. However, existing text matching models lack explicit constraints to make text representations domain invariant. It is still an open question to make it clear how to align the text representation distribution of different domains in the common semantic space. Meanwhile, the role of aligning the distribution of text representation from different domains remains unclear. (Yu et al., 2022) designs a regularizer based on distributed distance to explicitly promote domain invariant features. Unlike (Yu et al., 2022), which directly optimizes the distribution of text representations from different domains, we reduce the distance between the distribution of text representations from different domains by explicitly aligning the distribution of text representations from different domains with the same prior distribution. We theoretically prove that narrowing the distribution of text representations in asymmetrical domains text matching is equivalent to optimizing the information bottleneck (IB). For text matching, the interaction between texts has an important impact on the matching effect (Chen et al., 2017; Wang et al., 2017; Gong et al.; Kim et al., 2019; Yang et al., 2019). However, IB does not restrict the interaction between text representations. Therefore, we propose the adequacy of interaction and the incompleteness of a single text representation on the basis of IB and obtain the representation matching information bottleneck (RMIB). We theoretically prove that the constraints on text representations in RMIB is equivalent to maximizing the mutual information between text representations on the premise that the task information is given. The contributions of this work can be summarized as follows:
捕获潜在文本表示之间的共性。但是，现有的文本匹配模型缺乏显式约束，无法使文本表示成为域不变。要弄清楚如何在公共语义空间中对齐不同域的文本表示分布，仍然是一个悬而未决的问题。同时，协调来自不同领域的文本表示分布的作用仍不清楚。（Yu et al.， 2022）设计了一个基于分布距离的正则化器，以显式促进域不变特征。与直接优化来自不同域的文本表示分布的（Yu et al.， 2022）不同，我们通过将来自不同域的文本表示的分布与相同的先验分布显式对齐来缩短来自不同域的文本表示之间的距离。从理论上证明，缩小非对称域文本匹配中文本表示的分布等同于优化信息瓶颈（IB）。对于文本匹配，文本之间的交互对匹配效果有重要影响（Chen et al.， 2017;Wang等人，2017;龚等人;Kim 等人，2019 年;Yang 等人，2019 年）。但是，IB不限制文本表示之间的交互。因此，我们提出了基于IB的交互充分性和单个文本表示的不完备性，并得到了表示匹配信息瓶颈（RMIB）。从理论上证明，RMIB中对文本表示的约束等价于在给定任务信息的前提下，最大化文本表示之间的互信息。这项工作的贡献可以总结如下：

We use capital letters for random variables and lowercase letters for realizations. Let and be random variables, then the information entropy of random variable is defined as . The definition of conditional entropy is . The mutual information between and is defined as . The Kullback-Leibler divergence between two distributions is defined as .
我们使用大写字母表示随机变量，使用小写字母表示实现。设 和 为随机变量，则随机变量 的信息熵定义为 。条件熵的定义是 。和 之间的 互信息定义为 。两个分布之间的 Kullback-Leibler 散度定义为 。

Information bottleneck (IB) (Tishby et al., 2000), also known as representation information bottleneck (RIB), regards the feedforward calculation process of deep neural network as a Markov chain. Let be the model input, then is the task label, is the prediction result of the model and is the output of the middle layer, that is, the representation of the model input. If the above variables are regarded as random variables, the following Markov chain is formed as:
信息瓶颈（IB）（Tishby et al.， 2000），也称为表示信息瓶颈（RIB），将深度神经网络的前馈计算过程视为马尔可夫链。设 为模型输入，则 为任务标签， 为模型的预测结果， 为中间层的输出，即模型输入的表示。如果将上述变量视为随机变量，则形成以下马尔可夫链：

This means that or is satisfied, that is, under the premise that is known, the existence of will not affect , indicating that and are independent of each other. IB put forward the following two requirements for the representation Z learned by the deep learning model.
这意味着 或 被满足，即在已知的前提 下，存在 不会影响 ，表示 和 是相互独立的。IB对深度学习模型学习到的表示Z提出了以下两个要求。

Sufficient: The representation should contain as much information related to the target label as possible.
充分：表示 形式应包含与目标标签相关的尽可能多的信息。

Minimal: The representation should contain as little information as possible about input .
最小：表示 应尽可能少地包含有关输入 的信息。

On the premise of meeting the sufficient, IB requires that the representation should contain as little information as possible in input , which means that it contains as little redundant information as possible. Combining sufficient and minimal, we can get the optimization objective of IB as follows:
在满足充分条件的前提下，IB要求表示 在输入 中应包含尽可能少的信息，这意味着它包含尽可能少的冗余信息。结合充分和最小，我们可以得到IB的优化目标如下：

By using the Lagrange multiplier method, we can remove the constraint and equivalently express the above formula
通过使用拉格朗日乘数方法，我们可以消除约束并等效地表达上述公式
as: 如：

where is a positive constant. The optimization objective of Equation (5) is also called IB Lagrangian.
其中 为正常数。方程（5）的优化目标也称为IB拉格朗日量。

In the asymmetrical domains text matching task, the input of the model contains two texts, which are represented by two random variables, and . Usually and are sampled from different domains. Then the asymmetrical domains text matching task can be expressed as:
在非对称域文本匹配任务中，模型的输入包含两个文本，分别由两个随机变量表示， 并且 .通常 从 不同的域中采样。那么非对称域文本匹配任务可以表示为：

where represents the parameters of the constructed text matching model, and . The samples are assumed to come from an unknown distribution . The current text matching mainly includes two types of matching models: text encoding model and text interaction model. The text encoding model will encode the input texts and respectively, and there will be no interaction between and during the encoding process. A typical model of text encoding model is siamese neural network (Koch et al., 2015). The text interaction model will interact with the text representation in the feedforward calculation, and usually get better results. A typical text interaction model is ESIM (Chen et al., 2017). However, no matter which text matching model is used, two input text representations will be obtained in the feedforward calculation process. Therefore, the Markov chain in the process of text matching can be expressed as:
其中 表示构造的文本匹配模型的参数， 并表示 。假设样本来自未知分布 。目前的文本匹配主要包括两种类型的匹配模型：文本编码模型和文本交互模型。文本编码模型将分别对输入文本 和 进行编码，并且在编码过程中不会 有交互。文本编码模型的典型模型是暹罗神经网络（Koch et al.， 2015）。文本交互模型将在前馈计算中与文本表示进行交互，并且通常会获得更好的结果。典型的文本交互模型是ESIM（Chen et al.， 2017）。但是，无论使用哪种文本匹配模型，在前馈计算过程中都会获得两个输入文本表示。因此，文本匹配过程中的马尔可夫链可以表示为：

where and are the representations of and respectively and represents the final representation after the fusion of two text representations, usually the output of the representation fusion layer. Accordingly, the optimization objective of the information bottleneck in the text matching can be expressed as:
其中 和 分别是 和 的表示，分别 表示两个文本表示融合后的最终表示，通常是表示融合层的输出。因此，文本匹配中信息瓶颈的优化目标可以表示为：

Recent studies have shown that domain matching of text representation will help improve the generalization ability of text matching (Yu et al., 2022). To effectively align the distribution of text representation, (Yu et al., 2022) designs a regularizer based on distributed distance to narrow the text representations:
最近的研究表明，文本表示的领域匹配将有助于提高文本匹配的泛化能力（Yu et al.， 2022）。为了有效地对齐文本表示的分布，（Yu et al.， 2022）设计了一个基于分布距离的正则化器来缩小文本表示：

where is a metric function of distribution distance. Unlike (Yu et al., 2022), we narrow the distribution between text representations by explicitly aligning text representations with a prior distribution in text matching. Specifically, let the prior distribution be and two input text representations be and . To make the distribution between and as close as possible, we expect the distribution of and to be as close as on the premise that the model can correctly predict the target label. Therefore, the optimization objective of text matching based on text representations alignment can be expressed as:
其中 是分布距离的度量函数。与（Yu et al.， 2022）不同，我们通过将文本表示与文本匹配中的先验分布显式对齐来缩小文本表示之间的分布。具体来说，设先验分布为 ， 两个输入文本表示形式为 和 。为了使 和 之间的 尽可能接近，我们期望 和 的分布尽可能 接近，前提是模型可以正确预测目标 标签。因此，基于文本表示对齐的文本匹配的优化目标可以表示为：

We can use our method to explain DDR-Match (Yu et al., 2022). Let be the distribution after DDR-Match alignment, then in our method it can be expressed as:
我们可以用我们的方法来解释 DDR-Match（Yu 等人，2022 年）。假设 是 DDR-Match 对齐后的分布，那么在我们的方法中，它可以表示为：

By using the Lagrange multiplier method, we can remove the constraint and equivalently express the above formula as:
通过使用拉格朗日乘数方法，我们可以消除约束，并将上述公式等效地表示为：

where and are positive constants. Although some studies have shown that domain alignment of text representations will help improve the performance of text matching, the role of text representations distribution alignment is still unclear. We analyze this phenomenon and get Theorem 2.1. The proof of Theorem 2.1 can be found in appendix.
其中 和 是正常数。尽管一些研究表明，文本表示的领域对齐将有助于提高文本匹配的性能，但文本表示分布对齐的作用仍不清楚。我们分析了这种现象，得到了定理2.1。定理 2.1 的证明可以在附录中找到。

Theorem 2.1 shows that domain matching in text matching is equivalent to optimizing the information bottleneck in text matching, which indicates that domain alignment of input texts can make the learned text representation forget the input redundant information as much as possible.
定理2.1表明，文本匹配中的域匹配等同于优化文本匹配中的信息瓶颈，这表明输入文本的域对齐可以使学习的文本表示尽可能忘记输入的冗余信息。

Theorem 2.1. Given matching texts , text representations and task label , then it satisfies in asymmet-
定理 2.1.给定匹配的 文本、文本表示 和任务标签 ，那么它满足不对称
rical domains text matching:
Rical 域文本匹配：

where and are positive constants.
其中 和 是正常数。

Information bottleneck (IB) only requires that the learned representations retain as much information as possible about the task label and as little information as possible about the input and does not impose any explicit constraints on the interaction between text representations. In text matching, the performance of text interaction model is usually better than that of text encoding model, which indicates that interaction on text representations has an important impact on the result of text matching. Therefore, we extend IB and obtain representation matching information bottleneck (RMIB) in text matching. Specifically, RMIB's constraints on text representations and input are consistent with IB, but RMIB puts forward the following constraints on the interaction between text representations and the relationship between text representations and task label.
信息瓶颈 （IB） 仅要求学习的表示保留尽可能多的有关任务标签的信息，并尽可能少地保留有关输入的信息，并且不对文本表示之间的交互施加任何显式约束。在文本匹配中，文本交互模型的性能通常优于文本编码模型，这表明文本表示的交互对文本匹配结果有重要影响。因此，我们扩展了IB并得到了文本匹配中的表示匹配信息瓶颈（RMIB）。具体而言，RMIB对文本表示和输入的约束与IB一致，但RMIB对文本表示之间的交互以及文本表示与任务标签之间的关系提出了以下约束。

Sufficient: The representations and should contain as much information related to the target label as possible. This constraint is consistent with the Sufficient of IB.
充分：表示 形式应 包含与目标标签相关的尽可能多的信息。此约束与IB的充分性一致。

Interaction: The interaction between text representations should be sufficient, which means there should be enough mutual information between the two text representations.
交互：文本表示之间的交互应该是充分的，这意味着两个文本表示之间应该有足够的互信息。

Inadequacy: The final correct result cannot be obtained only by using a single text representation in text matching. In text matching, the model needs to make right decisions based on text representation of two input texts, which means that only using a single text representation can not get the right result. For example, in the text encoding model, due to the lack of interaction between the two input texts during the feedforward calculation, it is extremely absurd to use only a single text representation to accurately predict the target.
不足：在文本匹配中仅使用单个文本表示无法获得最终的正确结果。在文本匹配中，模型需要根据两个输入文本的文本表示做出正确的决策，这意味着仅使用单个文本表示无法获得正确的结果。例如，在文本编码模型中，由于前馈计算过程中两个输入文本之间缺乏交互，因此仅使用单个文本表示来准确预测目标，是极其荒谬的。

Based on the constraints proposed by RMIB for text representations, we can obtain the optimization objective of
基于RMIB提出的文本表示约束条件，可以得到
RMIB: RMIB：

A more intuitive RMIB optimization objective is shown in Figure 1. Although RMIB requires that the interaction between text representations should be sufficient, it is difficult to know the impact of the interaction of text representations on task label if we only consider the interaction of text representations without associating with task label. Further, we prove that the constraints of RMIB on text representation and task is equivalent to maximizing the mutual information between text representations on the premise that the task information is given and get the Theorem 2.2. The proof of Theorem 2.2 can be found in appendix. Theorem 2.2 shows that the optimization objective of RMIB can be expressed as:
更直观的 RMIB 优化目标如图 1 所示。虽然RMIB要求文本表示之间的交互应该是充分的，但如果我们只考虑文本表示的交互而不与任务标签关联，就很难知道文本表示的交互对任务标签的影响。进一步，我们证明了RMIB对文本表示和任务的约束等价于在给定任务信息的前提下最大化文本表示之间的互信息，并得到定理2.2。定理 2.2 的证明可以在附录中找到。定理2.2表明，RMIB的优化目标可以表示为：

Although we can see the differences between RMIB and IB from the sufficiency of text representation interaction and the inadequacy of a single text representation, we can see the differences between RMIB and IB more intuitively from Theorem 2.2. IB only requires text representation to contain as much task information as possible, but it lacks constraints on the interaction between text representations. While RMIB requires that the interaction between text representations should be as much as possible on the premise of giving task label. This shows that RMIB is more suitable for text matching than IB. Another function of Theorem 2.2 is to provide a transformation method for optimizing conditional mutual information.
虽然我们可以从文本表示交互的充分性和单个文本表示的不足性中看到 RMIB 和 IB 之间的差异，但我们可以从定理 2.2 中更直观地看到 RMIB 和 IB 之间的差异。IB只要求文本表示包含尽可能多的任务信息，但它缺乏对文本表示之间交互的约束。而 RMIB 要求文本表示之间的交互应尽可能在给出任务标签的前提下进行。这表明RMIB比IB更适合文本匹配。定理 2.2 的另一个功能是提供一种优化条件互信息的变换方法。

Theorem 2.2. Given text representations and task label , then it satisfies in asymmetrical domain text matching:
定理 2.2.给定文本 表示和任务标签 ，则它满足非对称域文本匹配：

Although RMIB gives optimization objective in text matching based on mutual information, it is difficult to directly optimize mutual information items during training process. Fortunately, as mutual information has received more attention in deep learning recently, some methods for approximate estimation of mutual information have also been
虽然RMIB给出了基于互信息的文本匹配的优化目标，但在训练过程中很难直接优化互信息项。幸运的是，随着互信息在深度学习中受到越来越多的关注，一些互信息的近似估计方法也得到了

Figure 1: The optimization objective of RMIB. and indicate that text representations and should contain as little input information as possible. and indicate that a single text representation cannot complete correct text matching. explicitly requires as much interaction as possible between and .
图 1：RMIB 的优化目标。 并指出 文本表示 形式， 并应包含尽可能少的输入信息。 并 指示单个文本表示无法完成正确的文本匹配。 显式要求 和 之间 尽可能多地交互。

proposed (Alemi et al., 2016; Belghazi et al., 2018; Hjelm et al.), which makes the optimization objective of RMIB tractable. For the optimization objective of RMIB, we can further equivalently express it as:
提议（Alemi 等人，2016 年;Belghazi 等人，2018 年;Hjelm et al.），这使得 RMIB 的优化目标变得容易处理。对于 RMIB 的优化目标，我们可以进一步等效地表示为：

where the constant C is the tightest upper bound of the theory of , and it can be easily proved that . The proof of Theorem 2.3 can be found in appendix.
其中常数 C 是 的最 紧上界，可以很容易地证明 。定理 2.3 的证明可以在附录中找到。

Theorem 2.3. Given text representations and task label , let , then it satisfies in asymmetrical domains text matching:
定理 2.3.给定文本 表示和任务标签 ，let ，则它满足非对称域中的文本匹配：

Based on the Lagrange multiplier method and Theorem 2.2, we can further equivalently express Equation (17) as:
基于拉格朗日乘数法和定理 2.2，我们可以进一步等效地将方程 （17） 表示为：

where and are positive constants. For , we can also replace it with where is the fusion of and . Theorem 2.4 shows that if the costs of optimizing and to the same value are of no different, then directly optimizing will give the optimization target of RMIB a smaller value, that is, a more optimized and . The proof of Theorem 2.4 can be found in appendix. Based on Equation (18), Theorem 2.5 gives the objective function of the RMIB that is ultimately used for optimization in the training process. The proof of Theorem 2.5 can be found in appendix. It is worth noting that the application of RMIB is not limited to text encoding model or text interaction model, because text representations and can be generated in either model.
其中 和 是正常数。对于 ，我们也可以用 和 的融合 来 代替它 。定理 2.4 表明，如果优化 的成本和 相同的值没有区别，那么直接优化 将使 RMIB 的优化目标值更小，即更优化的 和 。定理 2.4 的证明可以在附录中找到。基于方程 （18），定理 2.5 给出了 RMIB 的目标函数，该函数最终用于训练过程中的优化。定理 2.5 的证明可以在附录中找到。值得注意的是，RMIB的应用并不局限于文本编码模型或文本交互模型，因为文本表示 和可以在任一模型中生成。

Theorem 2.4. Given text representations , fusion text representation and task label , if the cost of optimizing and to the same value is the identical, then optimizing will obtain better and in RMIB.
定理 2.4.给定文本表示 、融合文本表示 和任务标签 ，如果优化 的成本和 相同的值相同，那么优化 将获得更好的 和 在RMIB中。

Theorem 2.5. The optimization objective of RMIB can be equivalently expressed as:
定理 2.5.RMIB的优化目标可以等效地表示为：

where and are positive constants and is a multilayer perceptron whose output layer activation function is sigmoid.
其中 和 是正常数， 是一个多层感知器，其输出层激活函数为 sigmoid。

SICK (Sentences Involving Compositional Knowledge) (Marelli et al., 2014) is a dataset for compositional distributional semantics, which consists of 9.8 k pairs of sentences. Each sentence pair is labelled as either contradiction, neutral or entailment. Accuracy is used as an evaluation metric for this dataset.
SICK（涉及组合知识的句子）（Marelli et al.， 2014）是一个组合分布语义数据集，由 9.8 k 对句子组成。每个句子对都被标记为矛盾、中性或蕴涵。准确性用作此数据集的评估指标。

SciTail (Science Entailment) (Khot et al., 2018) is an entailment dataset created from multiple-choice science exams and web sentences. The label could be entailment or neutral. The dataset contains a total of 27 k samples, of which 10 k samples have entailment labels, and the remaining 17 k are marked neutral. Accuracy is used as an evaluation metric for this dataset.
SciTail （Science Entailment） （Khot et al.， 2018） 是一个由多项选择科学考试和网络句子创建的蕴涵数据集。标签可以是蕴涵的，也可以是中性的。数据集共包含 27 k 个样本，其中 10 k 个样本具有蕴涵标签，其余 17 k 个样本标记为中性。准确性用作此数据集的评估指标。

WikiQA (Yang et al., 2015) is a retrieval-based question answering dataset based on Wikipedia. The binary label indicates whether the candidate sentence matches the question. The dataset contains 20.4 k training pairs, 2.7 k development pairs, and 6.2 k test pairs. Mean average precision (MAP) and mean reciprocal rank (MRR) are used as the evaluation metrics for this dataset.
WikiQA （Yang et al.， 2015） 是一个基于维基百科的基于检索的问答数据集。二进制标签指示候选句子是否与问题匹配。数据集包含 20.4 k 训练对、2.7 k 开发对和 6.2 k 测试对。平均精度 （MAP） 和平均倒数秩 （MRR） 用作该数据集的评估指标。

SNLI (Stanford Natural Language Inference) (Bowman et al., 2015) is a benchmark dataset for natural language inference which contains 570 k human annotated sentence pairs. One of the two sentences is used as "premise" and the other is used as "hypothesis". The label is one of "entailment", "neutral", "contradiction" and "-". We follow the practice in (Bowman et al., 2015) and the data labeled "-" is deleted. Accuracy is used as an evaluation metric for this dataset.
SNLI（斯坦福自然语言推理）（Bowman et al.， 2015）是自然语言推理的基准数据集，其中包含 570 k 个人类注释的句子对。两句话中的一句用作“前提”，另一句用作“假设”。标签是“蕴涵”、“中性”、“矛盾”和“-”之一。我们遵循（Bowman et al.， 2015）中的做法，并删除标有“-”的数据。准确性用作此数据集的评估指标。

Quora Question Pair is a dataset for paraphrase identification. Quora Question Pair contains 400k question pairs with binary label collected from the Quora website. This task is to determine whether two sentences are the interpretation of each other. Accuracy is used as an evaluation metric for this dataset.
Quora Question Pair 是一个用于释义识别的数据集。Quora 问题对包含从 Quora 网站收集的带有二进制标签的 400k 问题对。这个任务是确定两个句子是否是彼此的解释。准确性用作此数据集的评估指标。

The data preprocessing of SNLI, Quora Question Pair, SciTail and WikiQA is consistent with RE2 (Yang et al., 2019). We use 840B-300d GloVe (Pennington et al., 2014) to initialize the embedded layer of RE2 (Yang et al., 2019) and ESIM (Chen et al., 2017). We choose the standard Gaussian distribution as , because on the premise that the mean and variance are known, the entropy of the Gaussian distribution is the maximum, that is, the minimum prior knowledge is introduced. For the KL term in RMIB, we use the reparameter trick (Kingma & Welling, 2013) to optimize it. The hyper-parameters of ESIM and RE2 is consistent with (Yang et al., 2019). For BERT (Kenton & Toutanova, 2019) and SBERT (Reimers & Gurevych, 2019), we use the max pooling to obtain the text representations, the learning rate is 2e-5 and the epoch is 6 . We use BERT-base to initialize BERT and SBERT in our experiments. For SBERT, we minimize instead of and because and don't interact before the fusion layer. All experiments use Adam (Kingma & Ba, 2014) as the optimizer and the hyper-parameters search range for and in RMIB is . To ensure the reproducibility of the results, all experiments use the same seed.
SNLI、Quora Question Pair、SciTail 和 WikiQA 的数据预处理与 RE2 一致（Yang et al.， 2019）。我们使用 840B-300d GloVe （Pennington et al.， 2014） 来初始化 RE2 （Yang et al.， 2019） 和 ESIM （Chen et al.， 2017） 的嵌入层。我们选择标准高斯分布作为 ，因为在均值和方差已知的前提下，高斯分布的熵是最大值，即引入最小先验知识。对于RMIB中的KL项，我们使用reparameter技巧（Kingma&Welling，2013）来优化它。ESIM和RE2的超参数与（Yang等人，2019）一致。对于BERT（Kenton&Toutanova，2019）和SBERT（Reimers&Gurevych，2019），我们使用最大池化来获得文本表示，学习率为2e-5，纪元为6。我们在实验中使用 BERT 基来初始化 BERT 和 SBERT。对于 SBERT，我们最小化 而不是 和 因为 并且 在融合层之前不相互作用。所有实验都使用Adam（Kingma&Ba，2014）作为优化器，RMIB 中的 超参数搜索范围为 。为了确保结果的可重复性，所有实验都使用相同的种子。

To verify the validity of RMIB, we conduct the experiment using four common text matching models on five public datasets. BERT and SBERT are pre-training models while RE2 and ESIM are non-pre-training models. Meanwhile, BERT, RE2 and ESIM are text interaction models while SBERT is text encoding model. The experimental results are shown in Table 1. From Table 1, it can be seen that compared to the vanilla model, almost all models show significant improvements on the five datasets after using RMIB. We can find that compared with BERT, RE2 and ESIM, RMIB improves SBERT significantly in the five public datasets. For example, SBERT with RMIB can achieve gain on accuracy on SICK. Although the effect of SBERT is not as superior as that of BERT, the difference between the two decreases sharply after the addition of RMIB. For example, the F1 difference between the original SBERT and BERT on SICK is , but after the addition of RMIB, the F1 value difference is reduced to . Since the optimization goal of RMIB is the premise of known task information, the interaction between text representations should be as much as possible, while SBERT itself does not interact with text representations before the fusion layer. Therefore, RMIB can improve SBERT's model performance by explicitly increasing the interaction between text representations. This also indicates that for text matching tasks, the interaction between text representations is crucial to the final result.
为了验证RMIB的有效性，我们在五个公共数据集上使用四种常见的文本匹配模型进行了实验。BERT 和 SBERT 是预训练模型，而 RE2 和 ESIM 是非预训练模型。同时，BERT、RE2和ESIM是文本交互模型，而SBERT是文本编码模型。实验结果如表1所示。从表1可以看出，与原版模型相比，几乎所有模型在使用RMIB后在5个数据集上都表现出显著的改进。我们发现，与BERT、RE2和ESIM相比，RMIB在5个公共数据集中显著提高了SBERT。例如，配备 RMIB 的 SBERT 可以在 SICK 上实现 精度增益。虽然SBERT的效果不如BERT优越，但加入RMIB后两者之间的差异急剧减小。例如，SICK 上原始 SBERT 和 BERT 之间的 F1 差值为 ，但在添加 RMIB 后，F1 值差减小到 。由于 RMIB 的优化目标是已知任务信息的前提，因此文本表示之间的交互应尽可能多，而 SBERT 本身在融合层之前不与文本表示交互。因此，RMIB 可以通过显式增加文本表示之间的交互来提高 SBERT 的模型性能。这也表明，对于文本匹配任务，文本表示之间的交互对最终结果至关重要。

We find that RMIB can bring greater improvement in the case of data hunger. The improvement of the four text matching models on SICK, SciTail and WikiQA is significantly greater than that on SNLI and Quora. We think this is because SNLI and Quora have significantly more data than SICK, SciTail, WikiQA. For example, SNLI has about 57 times the data than SICK. (Koltchinskii, 2001) indicates that when the number of training data sets increases, the generalization performance of the model will also increase. When there are enough training sets, the model itself has good generalization performance, so RMIB can bring less improvement. We also find that BERT's accuracy on SNLI
我们发现，RMIB可以在数据饥渴的情况下带来更大的改善。SICK、SciTail 和 WikiQA 上 4 种文本匹配模型的改进明显大于 SNLI 和 Quora。我们认为这是因为 SNLI 和 Quora 的数据明显多于 SICK、SciTail 和 WikiQA。例如，SNLI 的数据量是 SICK 的 57 倍。（Koltchinskii，2001）指出，当训练数据集的数量增加时，模型的泛化性能也会增加。当训练集足够多时，模型本身具有良好的泛化性能，因此RMIB可以带来的改进较少。我们还发现 BERT 在 SNLI 上的准确性

Table 1: Performance of four text match model with RMIB on five public dataset. Right superscript * represents the results reproduced in our experimental environment. The other data in the table is obtained from (Yu et al., 2022). Due to the lack of results reported on Quora in (Yu et al., 2022), we use - for representation.
表 1：4 个文本匹配模型与 RMIB 在 5 个公共数据集上的性能。右上标 * 表示在我们的实验环境中重现的结果。表中的其他数据来自（Yu et al.， 2022）。由于 （Yu et al.， 2022） 中缺乏关于 Quora 的报告结果，我们使用 - 表示。