Spatio-Temporal Self-Supervised Learning for Traffic Flow Prediction
用于交通流预测的时空自监督学习

We firstly propose a spatio-temporal (ST) encoder to jointly preserve the ST contextual information over the traffic flow graph, so as to jointly model the sequential patterns of traffic data across different time steps and the geographical correlations among spatial regions. Towards this end, we integrate the temporal convolutional component with the graph convolutional propagation network as the backbone for spatial-temporal relational representation.
我们首先提出了一种时空（ST）编码器来联合保存交通流图上的 ST 上下文信息，从而联合建模不同时间步长的交通数据的顺序模式以及空间区域之间的地理相关性。为此，我们将时间卷积组件与图卷积传播网络集成，作为时空关系表示的骨干。

Our ST encoder is built with a “sandwich” block structure, in which TC $\to$ SC $\to$ TC is each individual block. By stacking multiple blocks, we can obtain a sequence of embedding matrix $\left(\bm{H}_{t-T^{\prime}},\dots,\bm{H}_{t}\right)$ with the temporal dimension of $T^{\prime}$ after several convolutions. After ST encoder-based embedding propagation and aggregation, the temporal dimension $T^{\prime}$ reduces to zero and we generate the final embedding matrix $\bm{H}\in\mathbb{R}^{N\times D}$ for our ST encoder, in which each row $\bm{h}_{n}\in\mathbb{R}^{D}$ denotes the final embedding of region $r_{n}$.
我们的 ST 编码器采用“三明治”块结构构建，其中 TC $\to$ SC $\to$ TC 是每个单独的块。通过堆叠多个块，经过多次卷积后，我们可以得到时间维度为 $T^{\prime}$ 的嵌入矩阵序列 $\left(\bm{H}_{t-T^{\prime}},\dots,\bm{H}_{t}\right)$ 。在基于 ST 编码器的嵌入传播和聚合之后，时间维度 $T^{\prime}$ 减少到零，我们为 ST 编码器生成最终的嵌入矩阵 $\bm{H}\in\mathbb{R}^{N\times D}$ ，其中每行 $\bm{h}_{n}\in\mathbb{R}^{D}$ 的最终嵌入。

In the next subsection, we will perform the adaptive augmentation over the $\left(\bm{B}_{t-T},\dots,\bm{B}_{t}\right)$ output from the first TC encoder layer (Sec 3.2), and self-supervised learning with the spatial-temporal heterogeneity modeling based on the final region embedding matrix $\bm{H}$ (Sec 3.3-Sec 3.4).
在下一小节中，我们将对第一个 TC 编码器层（第 3.2 节）的 $\left(\bm{B}_{t-T},\dots,\bm{B}_{t}\right)$ 输出执行自适应增强，并使用基于最终区域嵌入的时空异质性建模进行自监督学习矩阵 $\bm{H}$ （第 3.3 节-第 3.4 节）。

We devise two phases of graph augmentation schemes on TFG $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\bm{A},\mathcal{X}_{t-T:t}\right)$ with traffic-level data augmentation and graph topology-level structure augmentation, which is adaptive to the learned heterogeneity-aware region dependencies in terms of their traffic regularities.
我们在 TFG $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\bm{A},\mathcal{X}_{t-T:t}\right)$ 上设计了两个阶段的图增强方案，其中包括流量级数据增强和图拓扑级结构增强，该方案根据流量规律适应学习到的异质性感知区域依赖性。

Given the heterogeneity-aware augmented TFG, we aim to enable the region embeddings to effectively preserve the spatial heterogeneity with auxiliary self-supervised signals.
考虑到异质性感知增强 TFG，我们的目标是使区域嵌入能够通过辅助自监督信号有效地保留空间异质性。

To achieve this goal, we design a soft clustering-based self-supervised learning (SSL) task over regions, to map them into multiple latent representation spaces corresponding to diverse urban region functionalities (e.g., residential zone, shopping mall, transportation hub). Specifically, we generate $K$ cluster embeddings $\{\bm{c}_{1},\dots,\bm{c}_{K}\}$ (indexed by $k$) as latent factors for region clustering. Formally, the clustering process is performed with $\tilde{z}_{n,k}=\bm{c}_{k}^{\top}\tilde{\bm{h}}_{n}$. Here, $\tilde{\bm{h}}_{n}\in\mathbb{R}^{D}$ is the region embedding of region $r_{n}$ encoded from the augmented TFG $\tilde{\mathcal{G}}$. $\tilde{z}_{n,k}$ represents the estimated relevance score between region $r_{n}$’s embedding and the embedding $\bm{c}_{k}$ of the $k$-th cluster. Afterwards, the cluster assignment of region $r_{n}$ is generated with $\tilde{\bm{z}}_{n}=(\tilde{z}_{n,1},\dots,\tilde{z}_{n,K})^{\top}$.
为了实现这一目标，我们设计了一个基于软聚类的区域自监督学习（SSL）任务，将它们映射到与不同城市区域功能（例如住宅区、购物中心、交通枢纽）相对应的多个潜在表示空间。具体来说，我们生成 $K$ 聚类嵌入 $\{\bm{c}_{1},\dots,\bm{c}_{K}\}$ （由 $k$ 索引）作为区域聚类的潜在因素。形式上，聚类过程是通过 $\tilde{z}_{n,k}=\bm{c}_{k}^{\top}\tilde{\bm{h}}_{n}$ 执行的。这里， $\tilde{\bm{h}}_{n}\in\mathbb{R}^{D}$ 是从增强的 TFG $\tilde{\mathcal{G}}$ 编码的区域 $r_{n}$ 的区域嵌入。 $\tilde{z}_{n,k}$ 表示区域 $r_{n}$ 的嵌入与第 $k$ 集群的嵌入 $\bm{c}_{k}$ 之间的估计相关性得分。然后，使用 $\tilde{\bm{z}}_{n}=(\tilde{z}_{n,1},\dots,\tilde{z}_{n,K})^{\top}$ 生成区域 $r_{n}$ 的簇分配。

By incorporating the supervision on ${\bm{h}}_{n}$ with the heterogeneity-aware region cluster assignment $\tilde{\bm{z}}_{n}$, we make the region embedding ${\bm{h}}_{n}$ to be reflective of spatial heterogeneity within the global urban space.
通过将对 ${\bm{h}}_{n}$ 的监督与异质性感知区域集群分配 $\tilde{\bm{z}}_{n}$ 相结合，我们使区域嵌入 ${\bm{h}}_{n}$ 反映全球城市内的空间异质性空间。

For any assignment $\tilde{\bm{Z}}\in\tilde{\mathcal{Z}}$, we can use it to map the embedding matrix $\tilde{\bm{H}}=(\tilde{\bm{h}}_{1},\dots,\tilde{\bm{h}}_{N})^{\top}\in\mathbb{R}^{N\times D}$ into the cluster matrix $\bm{C}=(\bm{c}_{1},\dots,\bm{c}_{K})^{\top}\in\mathbb{R}^{K\times D}$. Thus, we search for the optimal solution by maximizing the similarity between the embeddings and the clusters, i.e.,
对于任何赋值 $\tilde{\bm{Z}}\in\tilde{\mathcal{Z}}$ ，我们可以使用它将嵌入矩阵 $\tilde{\bm{H}}=(\tilde{\bm{h}}_{1},\dots,\tilde{\bm{h}}_{N})^{\top}\in\mathbb{R}^{N\times D}$ 映射到聚类矩阵 $\bm{C}=(\bm{c}_{1},\dots,\bm{c}_{K})^{\top}\in\mathbb{R}^{K\times D}$ 中。因此，我们通过最大化嵌入和聚类之间的相似度来搜索最佳解决方案，即

where $\mathrm{tr}(\cdot)$ is the trace operator that sums elements on the main diagonal of a square matrix, $H(\tilde{\bm{Z}})$ is the entropy function defined as $-\sum_{n,k}\tilde{z}_{n,k}\log\tilde{z}_{n,k}$, and $\epsilon$ is a parameter that controls the smoothness of the assignment. Finally, the original assignment in Eq. (6) is replaced with the optimal solution. Refer to the Appendix for the solution procedure.
其中 $\mathrm{tr}(\cdot)$ 是对方阵主对角线上的元素求和的迹运算符， $H(\tilde{\bm{Z}})$ 是定义为 $-\sum_{n,k}\tilde{z}_{n,k}\log\tilde{z}_{n,k}$ 的熵函数，而 $\epsilon$

In this component, we further design a self-supervised learning (SSL) task to inject the temporal heterogeneity into time-aware region embeddings, by enforcing the divergence among time step-specific traffic pattern representations.
在此组件中，我们进一步设计了一个自监督学习（SSL）任务，通过强制特定于时间步的流量模式表示之间的差异，将时间异质性注入时间感知区域嵌入中。

Finally, we obtain the overall loss by incorporating the self-supervised spatial and temporal heterogeneity modeling losses in Eq. (6) and (11) into the joint learning objective:
最后，我们通过将自我监督的空间和时间异质性建模损失纳入方程（1）中来获得总体损失。 (6)和(11)纳入联合学习目标：

Our model can be trained via the back-propagation algorithm. The entire training procedure can be summarized into four stages: $i$) given a TFG $\mathcal{G}$, we generate a region embedding matrix $\bm{H}$ by the ST encoder. $ii$) Meanwhile, we perform adaptive augmentation to refine $\mathcal{G}$ as $\tilde{\mathcal{G}}$, which is fed into the shared ST encoder to output $\tilde{\bm{H}}$. $iii$) By using $\bm{H}$ and $\tilde{\bm{H}}$, we calculate the losses $\mathcal{L}_{s}$, $\mathcal{L}_{t}$, and $\mathcal{L}_{p}$ that are used to produce the joint loss $\mathcal{L}_{joint}$. $iv$). We employ the back-propagation algorithm to train ST-SSL until $\mathcal{L}_{joint}$ converges.
我们的模型可以通过反向传播算法进行训练。整个训练过程可以概括为四个阶段： $i$ ）给定一个TFG $\mathcal{G}$ ，我们通过ST编码器生成一个区域嵌入矩阵 $\bm{H}$ 。 $ii$ ）同时，我们执行自适应增强将 $\mathcal{G}$ 细化为 $\tilde{\mathcal{G}}$ ，将其输入共享 ST 编码器以输出 $\tilde{\bm{H}}$ 。 $iii$ ）通过使用 $\bm{H}$ 和 $\tilde{\bm{H}}$ ，我们计算损失 $\mathcal{L}_{s}$ 、 $\mathcal{L}_{t}$ 和 $\mathcal{L}_{p}$ 用于产生联合损失 $\mathcal{L}_{joint}$ 。 $iv$ ）。我们采用反向传播算法来训练 ST-SSL，直到 $\mathcal{L}_{joint}$ 收敛。