MultiplexNet: Towards Fully Satisfied Logical Constraints in Neural Networks
多层神经网络:朝着在神经网络中完全满足逻辑约束的目标前进

We consider a data set of $N$ i.i.d. samples from a hybrid (some mixture of discrete and/or continuous variables) probability density (Belle, Passerini, and Van den Broeck 2015). Moreover, we assume that: (1) the data set was generated by some random process $p^{*}(x)$; and (2) there exists domain or expert knowledge, in the form of a logical formula $\Phi$, about the random process $p^{*}(x)$ that can express the domain where $p^{*}(x)$ is feasible (non-zero). Both of these assumptions are summarised in Eq. 1. In Eq. 1, the notation $x\models\Phi$, denotes that the sample $x$ satisfies the formula $\Phi$ (Barrett et al. 2009). For example, if $\Phi:=x>3.5\land y>0$, and given some sample $(x,y)=(5,2)$, we denote: $(x,y)\models\Phi$.
我们考虑一个数据集,包含来自混合概率密度(离散和/或连续变量的某种混合)的 $N$ 个独立同分布的样本(Belle、Passerini 和 Van den Broeck 2015 年)。此外,我们假设:(1)数据集是由某种随机过程 $p^{*}(x)$ 生成的;(2)存在以逻辑公式 $\Phi$ 形式表达的领域或专家知识,描述了随机过程 $p^{*}(x)$ 中 $p^{*}(x)$ 可行(非零)的域。这两个假设总结在等式 1 中。在等式 1 中,符号 $x\models\Phi$ 表示样本 $x$ 满足公式 $\Phi$ (Barrett 等人,2009 年)。例如,如果 $\Phi:=x>3.5\land y>0$ ,且给定某个样本 $(x,y)=(5,2)$ ,我们表示: $(x,y)\models\Phi$ 。

Our aim is to approximate $p^{*}(x)$ with some parametric model $p_{\theta}(x)$ and to incorporate the domain knowledge $\Phi$ into the maximum likelihood estimation of $\theta$, on the available data set.
我们的目标是使用一些参数模型 $p^{*}(x)$ 来近似 $p_{\theta}(x)$ ,并将领域知识 $\Phi$ 纳入 $\theta$ 最大似然估计中,基于可用的数据集。

Given knowledge of the constraints $\Phi$, we are interested in ways of integrating these constraints into the training of a network that approximates $p^{*}(x)$. We desire an algorithm that does not require novel engineering to solve a reparameterisation of the network and moreover, especially salient for safety-critical domains, any sample $x$ from the model, $x\sim p_{\theta}(x)$, should imply that the constraints are satisfied. This is an especially important aspect to consider when comparing this method to alternative approaches, namely Fischer et al. (2019) and Xu et al. (2018), that do not give this same guarantee.
基于对约束条件 $\Phi$ 的了解,我们对将这些约束条件整合到近似 $p^{*}(x)$ 的网络训练过程中感兴趣。我们希望找到一种算法,不需要对网络进行重新参数化就可以解决这个问题,而且,对于安全至关重要的领域,从模型中生成的任何样本 $x$ , $x\sim p_{\theta}(x)$ ,都应该满足这些约束条件。这一点在与其他方法(如 Fischer et al. (2019) 和 Xu et al. (2018) 提出的方法)进行比较时尤为重要,因为这些方法并不能提供同样的保证。

Let $\tilde{x}$ denote the unconstrained output of a network. Let $g$ be a network activation that is element-wise non-negative (for example an exponential function, or a ReLU (Nair and Hinton 2010) or Softplus (Dugas et al. 2001) layer). If the property to be encoded is a simple inequality $\Phi:\forall x\;cx\geq b$, it is sufficient to constrain $\tilde{x}$ to be non-negative by applying $g$ and thereafter applying a linear transformation $f$ such that: $\forall\tilde{x}:cf(g(\tilde{x}))\geq b$. In this case, ${f}$ can implement the transformation $f(z)=sgn(c)z+\frac{b}{c}$ where $sgn$ is the operator that returns the sign of $c$. By construction we have:
Let $\tilde{x}$ denote the unconstrained output of a network. Let $g$ be a network activation that is element-wise non-negative (for example an exponential function, or a ReLU (Nair and Hinton 2010) or Softplus (Dugas et al. 2001) layer). If the property to be encoded is a simple inequality $\Phi:\forall x\;cx\geq b$ , it is sufficient to constrain $\tilde{x}$ to be non-negative by applying $g$ and thereafter applying a linear transformation $f$ such that: $\forall\tilde{x}:cf(g(\tilde{x}))\geq b$ . In this case, ${f}$ can implement the transformation $f(z)=sgn(c)z+\frac{b}{c}$ where $sgn$ is the operator that returns the sign of $c$ . By construction we have: 人工神经网络的输出可以表示为 $\tilde{x}$。网络激活函数 $g$ 可以是非负的元素级函数,如指数函数、ReLU (Nair 和 Hinton 2010)或 Softplus (Dugas 等人 2001)层。如果要编码的属性是简单不等式 $\Phi:\forall x\;cx\geq b$ ,只需将 $\tilde{x}$ 约束为非负,即应用 $g$,然后应用线性变换 $f$,使得 $\forall\tilde{x}:cf(g(\tilde{x}))\geq b$ 。在这种情况下, ${f}$ 可以实现变换 $f(z)=sgn(c)z+\frac{b}{c}$,其中 $sgn$ 是返回 $c$ 符号的运算符。通过构建,我们有:

It follows that more complex conjunctions of constraints can be encoded by composing transformations of the form presented in Eq. 2. We present below a few examples to demonstrate how this can be achieved for a number of common constraints (where $\tilde{x}$ always refers to the unconstrained output of the network):
因此,更复杂的约束组合可以通过组合 Eq. 2 中给出的形式的变换来编码。我们在下面给出几个例子,演示如何实现一些常见约束(其中 $\tilde{x}$ 始终指网络的无约束输出):

In Eq 3, we introduce the function $k(a,b)$. This is merely a function to compute the correct offset for a given activation $g$. In the case of the Softplus function, which is the function used in all of our experiments, $k(a,b)=log(exp(b-a)-1)$.
在等式 3 中,我们引入了函数 $k(a,b)$ 。这只是一个计算给定激活 $g$ 的正确偏移量的函数。对于我们所有实验中使用的 Softplus 函数而言, $k(a,b)=log(exp(b-a)-1)$ 。

In Section: Experiments, we implement three varied experiments that demonstrate how complex constraints can be constructed from this basic primitive in Eq. 2. Conceptually, appending additional conjunctions to $\Phi$ serves to restrict the space that the output can represent. However, in many situations domain knowledge will consist of complicated formulae that exist well beyond mere conjunctions of inequalities.
我们在实验一节中实施了三个不同的实验,以演示如何从等式 2 中的基本原语构建复杂的约束。从概念上讲,附加到 $\Phi$ 的其他连词有助于限制输出可以表示的空间。然而,在许多情况下,领域知识将由超出简单不等式的复杂公式组成。

While conjunctions serve to restrict the space permitted by the network’s output, disjunctions serve to increase the permissible space. For two terms $\phi_{1}$ and $\phi_{2}$ in $\phi_{1}\lor\phi_{2}$ there exist three possibilities: namely, that $x\models\phi_{1}$ or $x\models\phi_{2}$ or $(x\models\phi_{1})\land(x\models\phi_{2})$. Given the fact that any unconstrained network output can be transformed to satisfy some term $\phi_{k}$, we propose to introduce multiple transformations of a network’s unconstrained output, each to model the different terms $\phi_{k}$. In this sense, the network’s output layer can be viewed as a multiplexor in a logical circuit that permits for a branching of logic. If $h_{1}(\tilde{x})$ represents the transformation of $\tilde{x}$ that satisfies $\phi_{1}$ and $h_{2}(\tilde{x})\models\phi_{2}$ then we know the output must also satisfy $\phi_{1}\lor\phi_{2}$ by choosing either $h_{1}$ or $h_{2}$. It is this branching technique for dealing with disjunctions that gives rise to the name of the approach: MultiplexNet.
虽然连词有助于限制网络输出的空间，析取词则有助于增加可接受的空间。对于 $\phi_{1}\lor\phi_{2}$

We finally turn to the desideratum of allowing any Boolean formula over linear inequalities as the input for the domain constraints. The suggested approach can represent conjunctions of constraints and disjunctions between these conjunctive terms, which is exactly a DNF representation. Thus, the approach can be used with any transformed version of $\Phi$ that is in DNF (Darwiche and Marquis 2002). We propose to use an off-the-shelf solver, e.g., Z3 (De Moura and Bjørner 2008), to provide the logical input to the algorithm that is in DNF. We thus assume the domain knowledge $\Phi$ is expressed as:
我们终于转向允许任何布尔公式作为域约束输入的需求。建议的方法可以表示约束的并和约束项的析取,这正是一种析取范式(DNF)表示。因此,该方法可以用于 $\Phi$ 的任何转换版本,只要它处于 DNF 形式(Darwiche 和 Marquis 2002)。我们建议使用现成的求解器(如 Z3)(De Moura 和 Bjørner 2008)来为算法提供 DNF 形式的逻辑输入。因此,我们假定域知识 $\Phi$ 表示为:

If $h_{k}$ is the branch of MultiplexNet that ensures the output of the network $x\models\phi_{k}$ then it follows by construction that $h_{k}(\tilde{x})\models\Phi$ for all $k\in[1,\dots,K]$. For example, consider a network with a single real-valued output $\tilde{x}\in\mathbb{R}$. If the knowledge $\Phi:=(x\geq 2)\lor(x\leq-2)$, we would then have the two terms $h_{1}(\tilde{x})=g(\tilde{x})+2$ and $h_{2}(\tilde{x})=-g(-\tilde{x})-2$. Here, $g$ is the network activation that is element-wise non-negative that was referred to in Section: Satisfiability as Reparameterisation. It is clear that both $x_{1}=h_{1}(\tilde{x})$ and $x_{2}=h_{2}(\tilde{x})$ satisfy the formula $\Phi$.
如果 $h_{k}$ 是确保网络 $x\models\phi_{k}$ 输出的 MultiplexNet 分支,那么根据构造,对于所有 $k\in[1,\dots,K]$ , $h_{k}(\tilde{x})\models\Phi$ 成立。例如,考虑一个只有单个实值输出 $\tilde{x}\in\mathbb{R}$ 的网络。如果知识 $\Phi:=(x\geq 2)\lor(x\leq-2)$ ,我们就会有两个术语 $h_{1}(\tilde{x})=g(\tilde{x})+2$ 和 $h_{2}(\tilde{x})=-g(-\tilde{x})-2$ 。这里, $g$ 是在"可满足性作为重新参数化"一节中提到的元素级非负网络激活。很明显, $x_{1}=h_{1}(\tilde{x})$ 和 $x_{2}=h_{2}(\tilde{x})$ 都满足公式 $\Phi$ 。

MultiplexNet introduces a latent Categorical variable $k$ that selects among the different terms $\phi_{k},k\in[1,\dots,K]$. The model then incorporates a constraint transformation term $h_{k}$ conditional on the value of the Categorical variable.
多维 Network 介绍了一个潜在的分类变量 $k$ ，用于选择不同的术语 $\phi_{k},k\in[1,\dots,K]$ 。该模型随后根据分类变量的值纳入了一个约束转换项 $h_{k}$ 。

A lower bound on the likelihood of the data can be obtained by introducing a variational approximation to the latent Categorical variable $k$. This standard form of the variational lower bound (ELBO) is presented in Eq. 8.
通过引入对潜在分类变量 $k$ 的变分近似,可以获得数据可能性的下限。ELBO(变分下界)的标准形式如等式 8 所示。

Gradient based methods require calculating the derivative of Eq. 8. However, as $q(k)$ is a Categorical distribution, the standard reparameterisation trick cannot be applied (Kingma and Welling 2014). One possibility for dealing with this expectation is to use the score function estimator, as in REINFORCE (Williams 1992); however, while the resulting estimator is unbiased, it has a high variance (Mnih and Gregor 2014). It is also possible to replace the Categorical variable with a continuous approximation as is done by Maddison, Mnih, and Teh (2017) and Jang, Gu, and Poole (2016); or, if the dimensionality of the Categorical variable is small, it can be marginalised out as in (Kingma et al. 2014). In the experiments in Section: Experiments, we follow Kingma et al. (2014) and marginalise this variable,²²2Although we note that the alternatives should also be explored. leading to the following learning objective:
基于梯度的方法需要计算公式 8 的导数。然而,由于 $q(k)$ 是一个分类分布,标准重参数化技巧无法应用(Kingma 和 Welling,2014)。处理这个期望的一种可能性是使用得分函数估计量,如 REINFORCE(Williams,1992);然而,尽管所得到的估计量是无偏的,但它具有较高的方差(Mnih 和 Gregor,2014)。也可以将分类变量替换为连续逼近,如 Maddison、Mnih 和 Teh(2017)和 Jang、Gu 和 Poole(2016)所做的;或者,如果分类变量的维度很小,可以将其边缘化,如(Kingma 等人,2014)所做的。在第:实验章节的实验中,我们遵循 Kingma 等人(2014)的做法,将此变量边缘化, ² 导致如下学习目标:

We show in Section: Experiments that this approach can be applied equally successfully for a generative modeling task (where the goal is density estimation) as for a discriminative task (where the goal is structured classification). This helps to demonstrate the universal applicability of incorporating domain knowledge into the training of networks.
我们在实验部分演示,这种方法同样适用于生成性建模任务(目标是密度估计)和判别性任务(目标是结构化分类)。这有助于证明将领域知识纳入网络训练的通用适用性。

In this illustrative experiment, we consider a target data set that consists of the six rectangular modes that are shown in Figure 1. The samples from the true target density are shown, along with $8$ rectangular boxes in red. The rectangular boxes represent the domain constraints for this experiment. Here, we show that an expert might know where the data can exist but that the domain knowledge does not capture all of the details of the target density. Thus, the network is still tasked with learning the intricacies of the data that the domain constraints fail to address (e.g., not all of the area within the constraints contains data). However, we desire that the knowledge leads the network towards a better solution, and also to achieve this on fewer data samples from the true distribution.
在本说明性实验中,我们考虑一个由六个矩形模式组成的目标数据集,如图 1 所示。显示了真实目标密度的样本,以及红色的 $8$ 矩形框。矩形框代表了本实验的域约束。这里,我们展示了专家可能知道数据可能存在的位置,但域知识并未捕捉到目标密度的所有细节。因此,网络仍需要学习域约束无法解决的数据细节(例如,约束区域内并非全部包含数据)。然而,我们希望知识能够引导网络走向更好的解决方案,并且能够在从真实分布中采样较少的数据中实现这一目标。

This experiment represents a density estimation task and thus we use a likelihood-based generative model to represent the unknown target density, using both data samples and domain knowledge. We use a variational autoencoder (VAE) which optimizes a lower bound to the marginal log-likelihood of the data. However, a different generative model, for example a normalizing flow (Papamakarios et al. 2019) or a GAN (Goodfellow et al. 2014), could as easily be used in this framework. We optimize Eq. 9 where, for this experiment, the likelihood term $\log p_{\theta}(\cdot\mid k)$ is replaced by the standard VAE loss. Additional experimental details, as well as the full loss function, can be found in Appendix A.
这个实验表示一个密度估计任务,因此我们使用基于似然的生成模型来表示未知的目标密度,同时使用数据样本和领域知识。我们使用变分自编码器(VAE)来优化数据边际对数似然的下界。然而,在这个框架中也可以轻松使用其他生成模型,例如标准化流(Papamakarios et al. 2019)或 GAN(Goodfellow et al. 2014)。我们优化了公式 9,对于这个实验,似然项 $\log p_{\theta}(\cdot\mid k)$ 被标准 VAE 损失所取代。附录 A 中提供了额外的实验细节以及完整的损失函数。

We vary the size of the training data set with $N\in\{100,250,500,1000\}$ as the four experimental conditions. We compare the lower bound to the marginal log-likelihood under three conditions: the MultiplexNet approach, as well as two baselines. The first baseline (Unaware VAE) is a vanilla VAE that is unaware of the domain constraints. This scenario represents the standard setting where domain knowledge is simply ignored in the training of a deep generative network. The second baseline (DL2-VAE) represents a method that appends a loss term to the standard VAE loss. It is important to note that this approach, from DL2 (Fischer et al. 2019), does not guarantee that the constraints are satisfied (clearly seen in Figure 2(b)).
我们通过训练数据集大小的变化来设置 $N\in\{100,250,500,1000\}$ 作为四种实验条件。我们将边际对数似然的下界与三种情况进行比较:MultiplexNet 方法以及两种基线。第一个基线(Unaware VAE)是一个未意识到域约束的标准 VAE。这种情况代表了忽略域知识在深度生成网络训练中的标准设置。第二个基线(DL2-VAE)表示一种在标准 VAE 损失中附加一个损失项的方法。需要注意的是,这种方法(来自 DL2(Fischer 等人,2019))无法保证满足约束条件(在图 2(b)中可以清楚地看到)。

Figure 2 presents the results where we run the experiment on the specified range of training data set sizes. The top plot shows the variational loss as a function of the number of epochs. For all sizes of training data, the MultiplexNet loss on a test set can be seen to outperform the baselines. By including domain knowledge, we can reach a better result, and on fewer samples of data, than by not including the constraints. More important than the likelihood on held-out data is that the samples from the models’ posterior should conform with the constraints. Figure 2(b) shows that the baselines struggle to learn the structure of the constraints. While the MultiplexNet solution is unsurprising, the constraints are followed by construction, the comparison to the baselines is stark. We also present samples from both the prior and the posterior for all of these models in Appendix A. In all of these, MultiplexNet learns to approximate the unknown density within the predefined boundaries of the provided constraints.
图 2 展示了在指定范围的训练数据集大小上运行实验的结果。顶部图显示了变分损失随纪元数的函数。对于所有训练数据大小,MultiplexNet 在测试集上的损失都优于基线。通过包含领域知识,我们可以在使用较少的数据样本的情况下取得更好的结果,而不是不包含约束条件。比保留数据的可能性更重要的是,模型后验的样本应该符合约束条件。图 2(b)显示基线难以学习约束结构。虽然 MultiplexNet 解决方案并不出人意料,但通过构建,约束条件得到遵循,与基线相比是很明显的差异。我们还在附录 A 中展示了这些模型先验和后验的样本。在所有这些中,MultiplexNet 都学会在提供的约束边界内逼近未知密度。

We demonstrate how a structured data set, in combination with the relevant domain knowledge, can be used to make novel inferences in that domain. Here, we use a similar experiment to that from Kingma et al. (2014) where we model the MNIST digit data set in an unsupervised manner. Moreover, we take inspiration from Manhaeve et al. (2018) for constructing a structured data set were the images represent the terms in a summation (e.g., $image(2)+image(3)=5$). However, we add to the complexity of the task by (1) using no labels for any of the images;³³3In the MNIST experiment from Manhaeve et al. (2018), the authors use the result of the summation as labels for the algorithm. We have no such analogy in this experiment and thus cannot use their DeepProbLog implementation as a baseline. and, (2) considering a generative task.
我们演示如何利用结构化数据集和相关领域知识进行新的推论。在这里,我们使用类似于 Kingma 等人(2014)的实验,以无监督的方式对 MNIST 数字数据集进行建模。此外,我们从 Manhaeve 等人(2018)中获得灵感,构建一个结构化数据集,其中图像表示求和中的项(例如, $image(2)+image(3)=5$ )。但是,我们增加了任务的复杂性:(1)不使用任何图像的标签; ³ 和(2)考虑生成任务。

Kingma et al. (2014) propose a generative model that reasons about the cluster assignment of a data point (a single image). In particular, in their popular “Model 2,” they describe a generative model for an image $x$ such that the probability of the image pixel values are conditioned on a latent variable ($z$) and a class label ($y$): $p_{\theta}(x\mid z,y)p(z\mid y)p(y)$. We can interpret this model using the MultiplexNet framework where the cluster assignment label $y=k$ implies that the image $x$ was generated from cluster $k$. Given a reconstruction loss for image $x$, conditioned on class label $y$ ($\mathcal{L}(x,y)$), the domain knowledge in this setting is: $\Phi:=\bigvee_{k=1}^{10}\mathcal{L}(x,y)\land(y=k)$. We can successfully model the clustering of the data using this setup but there is no means for determining which label corresponds to which cluster assignment.
金玛等人(2014 年)提出了一种生成性模型,该模型推理数据点(单个图像)的簇分配。具体而言,在他们广受欢迎的"模型 2"中,他们描述了一个图像的生成模型 $x$ ,其中图像像素值的概率以潜在变量( $z$ )和类标签( $y$ )为条件: $p_{\theta}(x\mid z,y)p(z\mid y)p(y)$ 。我们可以使用 MultiplexNet 框架解释这个模型,其中簇分配标签 $y=k$ 意味着图像 $x$ 是从簇 $k$ 生成的。给定图像 $x$ 的重建损失,以类标签 $y$ ( $\mathcal{L}(x,y)$ )为条件,该设置中的领域知识为: $\Phi:=\bigvee_{k=1}^{10}\mathcal{L}(x,y)\land(y=k)$ 。我们可以成功地使用这种设置对数据进行聚类建模,但没有任何办法确定哪个标签对应于哪个簇分配。

We therefore propose to augment the data set such that each input is a quintuple of four images $(x_{1},x_{2},x_{3},x_{4})$ in the form $label(x_{1})+label(x_{2})=(label(x_{3}),label(x_{4}))$. Here, the inputs $label(x_{1})$ and $label(x_{2})$ can be any integer from $0$ to $9$ and the result $(label(x_{3}),label(x_{4}))$ is a two digit number from $(00)$ to $(18)$. While we do not know explicitly any of the cluster labels, we do know that the data conform to this standard. Thus for all $i,j,k$ where $k=i+j$, the domain knowledge is of the form:
因此,我们建议扩充数据集,使每个输入都是四个图像的五元组 $(x_{1},x_{2},x_{3},x_{4})$ 的形式 $label(x_{1})+label(x_{2})=(label(x_{3}),label(x_{4}))$ 。其中,输入 $label(x_{1})$ 和 $label(x_{2})$ 可以是从 $0$ 到 $9$ 的任意整数,结果 $(label(x_{3}),label(x_{4}))$ 是从 $(00)$ 到 $(18)$ 的两位数。虽然我们不知道任何聚类标签,但我们知道数据符合这一标准。因此,对于所有 $i,j,k$ 其中 $k=i+j$ ,域知识的形式是:

In this setting, the categorical variable in the MultiplexNet chooses among the $100$ combinations that satisfy $label(x_{1})+label(x_{2})=(label(x_{3}),label(x_{4}))$. This experiment has similarities to DeepProbLog (Manhaeve et al. 2018) as the primitive $\mathcal{L}(x,y)$ is repeated for each digit. In this sense, it is similar to the “neural predicate” used by Manhaeve et al. (2018), and the MultiplexNet output layer implements what would be the logical program from DeepProbLog. However, it is not clear how to implement this label-free, generative task within the DeepProbLog framework.
在这种设置下,MultiplexNet 中的分类变量选择满足 $label(x_{1})+label(x_{2})=(label(x_{3}),label(x_{4}))$ 的 $100$ 组合。这个实验与 DeepProbLog(Manhaeve et al. 2018)有相似之处,因为原语 $\mathcal{L}(x,y)$ 对每个数字都会重复。从这个意义上说,它类似于 Manhaeve et al.(2018)使用的"神经谓词",而 MultiplexNet 输出层实现了 DeepProbLog 中的逻辑程序。但是,在 DeepProbLog 框架内实现这种无标签的生成性任务并不明确。

In Figure 3, we present samples from the prior, conditioned on the different class labels. The model is able to learn a class-conditional representation for the data, given no labels for the images. This is in contrast to a vanilla model (from Kingma et al. (2014)) which does not use the structure of the data set to make inferences about the class labels. We present these baseline samples as well as the experimental details and additional notes in Appendix A. Empirically, the results from this experiment were sensitive to the network’s initialisation and thus we report the accuracy of the top $5$ runs. We selected the runs based on the loss (the ELBO) on a validation set (i.e., the labels were still not used in selecting the run). The accuracy of the inferred labels on held out data is $97.5\pm 0.3$.
在图 3 中，我们呈现了基于不同类别标签的先验样本。该模型能够学习数据的类条件表示，而无需使用图像的标签。这与 Kingma et al. (2014)提出的普通模型形成对比,该模型未利用数据集的结构来推断类别标签。我们同时呈现了基准样本以及实验细节和附加说明,详见附录 A。经验表明,该实验的结果对网络的初始化敏感,因此我们报告了前 $5$ 个运行的准确性。我们基于验证集的损失(ELBO)选择了这些运行(即标签未用于选择运行)。在保留数据上推断标签的准确性为 $97.5\pm 0.3$ 。

The final experiment demonstrates how to encode hierarchical domain knowledge into the output layer of a network. The CIFAR100 (Krizhevsky, Hinton et al. 2009) data set consists of $100$ classes of images where the $100$ classes are in turn broken into $20$ super-classes (SC). We wish to encode the belief that images from the same SC are semantically related. Following the encoding in Fischer et al. (2019), we consider constraints which specify that groups of classes should together be very likely or very unlikely. For example, suppose that the SC label is trees and the class label is maple. Our domain knowledge should state that the trees group must be very likely even if there is uncertainty in the specific label maple. Intuitively, it is egregious to misclassify this example as a tractor but it would be acceptable to make the mistake of oak. This can be implemented by training a network to predict first the SC for an unknown image and thereafter the class label, conditioned on the value for the SC.
最终实验演示如何将分层域知识编码到网络的输出层中。CIFAR100（Krizhevsky，Hinton 等 2009 年）数据集由 $100$ 个图像类组成,其中 $100$ 个类进一步划分为 $20$ 个超类(SC)。我们希望编码这样的信念:来自同一 SC 的图像在语义上是相关的。遵循 Fischer et al.（2019 年）的编码方式,我们考虑指定类群应该非常可能或非常不可能的约束。例如,假设 SC 标签是树木,类标签是枫树。我们的域知识应该声明,即使对具体标签枫树存在不确定性,树木组也应该很可能。直观地说,将此示例错误分类为拖拉机是严重的,但将其错误分类为橡树是可以接受的。可以通过训练网络首先预测未知图像的 SC,然后根据 SC 的值预测类标签来实现这一点。

We chose rather to implement this same knowledge using the MultiplexNet framework. Let $x_{k}\in SC_{i}$ denote the output of a network that predicts the $k^{th}$ class label within the $i^{th}$ SC. Let $\alpha\in[0,1]$ denote the minimum requirement for a SC prediction (e.g., if $\alpha=0.95$, we require that a SC be predicted with probability $0.95$ or more). The domain knowledge is:
我们选择使用 MultiplexNet 框架来实现这个知识。令 $x_{k}\in SC_{i}$ 表示预测 $k^{th}$ 类别标签的网络的输出,其中 $i^{th}$ 表示 SC。令 $\alpha\in[0,1]$ 表示 SC 预测的最小要求(例如,如果 $\alpha=0.95$ ,我们要求 SC 的预测概率为 $0.95$ 或更高)。

Eq. 11 states that for all labels within a SC group, the unnormalised logits of the network should be greater than the normalised sum of the other labels belonging to the other SCs with a margin of $\log(\frac{\alpha}{1-\alpha})$. We explain Eq. 11 further and present other experimental details in Appendix A. This constraint places a semantic grouping on the data as the network is forced into a low entropy prediction at the super class level.
等式 11 表明,对于一个 SC 组内的所有标签,网络的未归一化逻辑输出应当大于其他 SC 的其他标签归一化和加一个 $\log(\frac{\alpha}{1-\alpha})$

We compare the performance of MultiplexNet to three baselines and report the prediction accuracy on the fine class label as well that on the super class label. We use a Wide ResNet 28-10 (Zagoruyko and Komodakis 2016) model in all of the experimental conditions. The first two baselines (Vanilla) only use the Wide ResNet model and are trained to predict the fine class and the super class labels respectively. The second baseline (Hierarchical) is trained to predict the super class label and thereafter the fine class label, conditioned on the value for the super class label. This represents the bespoke engineering solution to this hierarchical problem. The final baseline (DL2) implements the same logical specification that is used for MultiplexNet but uses the DL2 framework to append to the standard cross-entropy loss function.
我们比较 MultiplexNet 与三种基线模型的性能,并报告它们在细粒度类标签和超类标签上的预测准确率。我们在所有实验条件下使用 Wide ResNet 28-10(Zagoruyko 和 Komodakis 2016)模型。前两个基线模型(Vanilla)仅使用 Wide ResNet 模型,分别被训练用于预测细粒度类标签和超类标签。第二个基线模型(Hierarchical)被训练用于首先预测超类标签,然后基于超类标签的值来预测细粒度类标签,这代表了针对这个层次性问题的定制工程解决方案。最后一个基线模型(DL2)实现了与 MultiplexNet 相同的逻辑规范,但使用 DL2 框架来附加到标准的交叉熵损失函数上。

Table 1 presents the results for this experiment. Firstly, it is important to note the difficulty of this task. The Vanilla ResNet that predicts only the super-class labels for the images under performs the baseline that is tasked with predicting the true class label. Moreover, while the hierarchical baseline does outperform the vanilla models on the task of super-class prediction, this comes at a cost to the true class accuracy. As the hierarchical baseline represents the bespoke engineering solution to the problem, it also achieves $100\%$ constraint satisfaction, but this comes at the cost of domain specific and custom implementation. The MultiplexNet approach provides a slight improvement at the SC classification accuracy and importantly, the domain constraints are always met. As the domain knowledge prioritizes the accuracy at the SC level, we note that the MultiplexNet approach does not outperform the Vanilla ResNet at the class accuracy. Surprisingly, the DL2 baseline improves upon the class accuracy but it has a limited impact on the super class accuracy and on the constraint satisfaction.
表 1 展示了这个试验的结果。首先,需要注意到这个任务的难度。只预测图像超类标签的 Vanilla ResNet 的表现不如预测真实类别标签的基线。此外,虽然分层基线在超类预测任务上优于普通模型,但这是以真实类别准确性为代价的。作为针对该问题的定制工程解决方案,分层基线也满足了 $100\%$ 约束条件,但需要特定领域和定制实现。MultiplexNet 方法在超类分类准确性上略有提高,并且始终满足领域约束。由于领域知识重点关注超类级别的准确性,我们注意到 MultiplexNet 方法在类别准确性上并未优于 Vanilla ResNet。令人惊讶的是,DL2 基线在类别准确性上有所改善,但对超类准确性和约束满足度的影响有限。