2.1. Source prompt training
2.1.來源提示訓練

SPE trains multiple source prompts for each source task(S = {S₁, S₂, …, S_N}) individually through prompt tuning.
SPE 透過提示調整，針對每個來源任務(S=_{S1,_S2, ...,_SN}) 個別訓練多個來源提示。

Given a task T with training data (X,Y)={xi,yi}Ni=1 and a pre-trained language model(PLM) with parameters Θ, prompt tuning randomly initializes a learnable prompt matrix(i.e., soft prompt) P ∈ ℝ^l×d with length l and prepends it to the input embedding sequence T ∈ ℝ^n×d of the PLM. Then, [P; X] is fed into the pre-trained model to make predictions. In the training process, only prompt embeddings are updated by maximizing the likelihood of output Y, as follows:

View Source\begin{equation*}\mathop {\max }\limits_P P{r_\Theta }(Y\mid [P;X])\tag{1}\end{equation*}

給定一個具有訓練資料 (X,Y)={xi,yi}Ni=1 的任務T和一個具有參數 Θ 的預先訓練語言模型 (PLM)，提示調諧隨機初始化一個長度為l的可學習提示矩陣 (即軟提示)P∈ℝl^×d，並將其加入 PLM 的輸入嵌入序列T∈ ^ℝn×d。然後將[P;X] 輸入預先訓練好的模型來進行預測。在訓練過程中，只會透過最大化輸出Y 的可能性來更新提示嵌入，如下所示：

maxPPrΘ(Y∣[P;X])(1)

View Source\begin{equation*}\mathop {\max }\limits_P P{r_\Theta }(Y\mid [P;X])\tag{1}\end{equation*}

2.2. Target prompt training
2.2.目標提示訓練

After acquiring an album of multiple source snapshot prompts, our method aims to apply a cross-task attention module to adaptively ensemble the multiple soft prompts and obtain a fused and instance-dependent prompt for the target task.
在取得多個來源快照提示的相簿之後，我們的方法旨在應用跨任務注意模組來自適應性地組合多個軟提示，並針對目標任務取得融合且依據實例的提示。

Adaptive Snapshot Prompts Ensembling
自適應快照提示組合

SPE uses a cross-task attention module to adaptively generate weights to ensemble the source prompts based on their competence on the target instance. Because of the different length between input instance X and source prompt, the module first performs max-pool over the token embedding sequence X = [x₁, x₂, …, x_n] ∈ ℝ^n×d and prompt embedding sequence P_i,j = [p₁, p₂, …, p_n] ∈ ℝ^l×d, to obtain X^∈Rd and P^i,j∈Rd. Then apply a feed-forward network to project x to prompt representational space, as follows:

H=WT2⋅γ(WT1⋅X^)(2)

View Source\begin{equation*}H = W_2^T \cdot \gamma \left( {W_1^T \cdot \hat X} \right)\tag{2}\end{equation*} where W₁ ∈ ℝ^d×r and W₂ ∈ ℝ^r×d are trainable weights, γ(•) is a non-linear activation function. We apply Layer Norm[13] on H to get the projected representation H^∈Rd of the target input instance. Subsequently, compute the attention score by calculating the dot product between P^i,j and H^ as follows:

ai,j=eP^i,jH^∑Ni=1∑Kj=1eP^i,jH^Pinstance=∑i=1N∑j=1Kai,jPi,j(3)(4)

View Source\begin{align*} & {a_{i,j}} = \frac{{{e^{{{\hat P}_{i,j}}\hat H}}}}{{\sum\nolimits_{i = 1}^N {\sum\nolimits_{j = 1}^K {{e^{{{\hat P}_{i,j}}\hat H}}} } }}\tag{3} \\ & {P_{instance}} = \sum\limits_{i = 1}^N {\sum\limits_{j = 1}^K {{a_{i,j}}} } {P_{i,j}}\tag{4}\end{align*}
SPE 使用跨任務注意模組來自適應性地產生權重，以根據來源提示在目標實例上的能力來集合來源提示。由於輸入實例X與來源提示之間的長度不同，該模組首先對代號嵌入序列X=[_x1,_x2,...,_xn] ∈^ℝn×d和提示嵌入序列Pi_,j=[_p1, p2,...,_pn] ∈^ℝl×d 執行最大池，得到 X^∈Rd 和 P^i,j∈Rd 。然後應用前饋網路將x投射到提示表徵空間，如下所示：

H=WT2⋅γ(WT1⋅X^)(2)

View Source\begin{equation*}H = W_2^T \cdot \gamma \left( {W_1^T \cdot \hat X} \right)\tag{2}\end{equation*} 其中_W1∈^ℝd×r和_W2∈^ℝr×d是可訓練的權值，γ(-) 是非線性激活函數。我們對H應用 Layer Norm [13] ，得到目標輸入實例的投影表示 H^∈Rd 。之後，透過計算 P^i,j 與 H^ 之間的點乘來計算注意力分數，如下所示：

ai,j=eP^i,jH^∑Ni=1∑Kj=1eP^i,jH^Pinstance=∑i=1N∑j=1Kai,jPi,j(3)(4)

View Source\begin{align*} & {a_{i,j}} = \frac{{{e^{{{\hat P}_{i,j}}\hat H}}}}{{\sum\nolimits_{i = 1}^N {\sum\nolimits_{j = 1}^K {{e^{{{\hat P}_{i,j}}\hat H}}} } }}\tag{3} \\ & {P_{instance}} = \sum\limits_{i = 1}^N {\sum\limits_{j = 1}^K {{a_{i,j}}} } {P_{i,j}}\tag{4}\end{align*}

Table 1. Results on GLUE
表 1. GLUE 的結果

The obtained prompt is a fused prompt that incorporates information across different tasks and different training phases. Meanwhile, P_instance is instance-dependent. Compared to other SPT methods[4], the cross-task attention module can effectively learn more suitable prompt from source prompts for target data instance.
所得到的提示是融合了不同任務和不同訓練階段資訊的提示。同時，_Pinstance取決於實例。相較於其他 SPT 方法 [4] ，跨任務注意模組能有效地從來源提示中學習到更適合目標資料實例的提示。

Considering that instance-dependent prompts may ignore the importance of task-specific information[14], SPE initializes a task-wise prompt P_task for the target task, which is trained and shared by all the instances of the target task. The final target prompt is the combination of instance-wise prompt and task-wise prompt as follows:

Ptarget=Pinstance+Ptask(5)

View Source\begin{equation*}{P_{target}} = {P_{instance}} + {P_{task}}\tag{5}\end{equation*}
考慮到依據實例的提示可能會忽略特定任務資訊的重要性 [14] ，SPE 會初始化目標任務的任務性提示_Ptask，此Ptask經過訓練，並由目標任務的所有實例共享。最終的目標提示是實例提示和任務提示的組合，如下所示：

Ptarget=Pinstance+Ptask(5)

View Source\begin{equation*}{P_{target}} = {P_{instance}} + {P_{task}}\tag{5}\end{equation*}

Finally, we concentrate the target prompt to the input and train the model by maximizing the maximizing the likelihood:

maxPtask,GPrθ(Y∣[Ptarget,X])(6)

View Source\begin{equation*}\mathop {\max }\limits_{{P_{task}},G} P{r_\theta }\left( {Y\mid \left[ {{P_{target}},X} \right]} \right)\tag{6}\end{equation*}
最後，我們將目標提示集中到輸入，並透過最大化最大似然率來訓練模型：

maxPtask,GPrθ(Y∣[Ptarget,X])(6)

View Source\begin{equation*}\mathop {\max }\limits_{{P_{task}},G} P{r_\theta }\left( {Y\mid \left[ {{P_{target}},X} \right]} \right)\tag{6}\end{equation*}

During target prompt training, only prompt P_task and the cross-task attention module G are updated via P_target, while source prompts and the pre-trained LM θ are frozen.
在目標提示訓練期間，只有提示Ptask 和跨任務注意模組G會透過_Ptarget 更新，而來源提示和預先訓練的 LMθ則會凍結。

2.3. Parameter Efficiency of SPE
2.3.SPE 的參數效率

Since extracting multiple source prompts is done in one training, the trainable parameters in this process are only N source prompt (i.e., N × l × d). There are no extra parameters in multiple prompt extraction. SPE also trains a cross-task attention module which includes two projection layers, one layer norms, and a task prompt, which requires 2×d×r+2×d+l×d parameters. In total, for SPE, the number of trainable parameters is (1+ N)× l × d +2 × d × r(~0.7M in our experiment), which is less than 0.4% of full fine-tuning a T5-base model.
由於抽取多個來源提示是在一次訓練中完成，因此在這個過程中可訓練的參數只有N個來源提示 (即N×l×d)。在多重提示擷取中沒有額外的參數。SPE 還會訓練一個跨任務注意模組，這個模組包括兩個投影層、一個層規範和一個任務提示，需要2×d×r+2×d+l×d的參數。總的來說，對 SPE 而言，可訓練的參數為 (1+N)×l×d+2 ×d×r（在我們的實驗中約為 0.7M），少於 T5 基礎模型完全微調的 0.4%。

3.1. Experiments settings
3.1.實驗設定

Following the same setting of ATTEMPT[5], we use 6 high-resource datasets (MNLI, QNLI, QQP, SST-2, SQuAD, and ReCoRD) as source tasks. We use 8 tasks from GLUE[15] as target tasks with pre-trained T5-base model. Besides full fine-tuning, we compare our model with parameter-efficient methods, such as Adapter[16], Bitfit[17], PT[1], and soft prompt transfer methods, like SPoT[4] and ATTEMPT[5].
依照 ATTEMPT [5] 的相同設定，我們使用 6 個高資源資料集（MNLI、QNLI、QQP、SST-2、SQuAD 和 ReCoRD）作為源任務。我們使用 GLUE [15] 中的 8 個任務作為目標任務，並使用預先訓練的 T5 基礎模型。除了完全微調之外，我們還將我們的模型與參數效 率高的方法進行比較，例如 Adapter [16] , Bitfit [17] , PT [1] , 以及軟提示轉換方法，例如 SPoT [4] 和 ATTEMPT [5] 。

We report Pearson Correlation for STS-B, and accuracy for the other tasks as metrics and use a default setting training for a batch size of 32, a learning rate of 3e-4. We train our model on a single NVIDIA A100 with 80G of memory. We use 100 prompt vectors as prompt length for all benchmarks. During source prompt tuning, SPE takes snapshots when training to 60%, 80%, and 100% respectively.
我們報告 STS-B 的 Pearson Correlation 以及其他任務的準確度作為指標，並使用預設設定訓練，批次大小為 32，學習率為 3e-4。我們在具有 80G 記憶體的單一 NVIDIA A100 上訓練模型。我們使用 100 個提示向量作為所有基準的提示長度。在來源提示調整期間，SPE 分別在訓練到 60%、80% 和 100% 時進行快照。

3.2. Main Results 3.2.主要結果

Table 1 shows the performance of our method and other baselines on the GLUE benchmark. SPE outperforms baselines and achieves competitive results on several NLU tasks.
Table 1 顯示我們的方法和其他基線在 GLUE 基準上的表現。SPE 的表現優於基準，並在多項 NLU 任務上取得具有競爭力的結果。

Specifically, SPE achieves comparable performance with full parameter fine-tuning despite only tuning 0.4% of its parameters. Compared to vanilla’s prompt tuning, the average scores of our method are substantially improved from 72.2% to 86.0%, an improvement of 13.8%, highlighting the benefits of transferring knowledge from multiple source tasks. Compared with PET methods, the performance is also improved to varying degrees. Compared with the soft prompt transfer method SPoT and ATTEMPT, SPE improves by 3.7% and 2.6%, respectively. We posit that during source prompt iterative training, the prompts obtain task-specific knowledge gradually. SPE takes snapshots at the different training phases. Thus the snapshot prompts are equipped with different richness of task-specific knowledge. A probable reason for the good performance of SPE is that SPE can ensemble prompts with different levels of task-specific knowledge from different tasks.
具體來說，SPE 儘管只調整了 0.4% 的參數，卻達到了完全參數微調的可比性能。與 vanilla 的提示調整相比，我們方法的平均分數大幅提升，從 72.2% 提升至 86.0%，提升幅度達 13.8%，突顯出從多種來源任務轉移知識的優點。與 PET 方法相比，性能也有不同程度的提升。與軟提示轉移方法 SPoT 和 ATTEMPT 相比，SPE 分別提高了 3.7% 和 2.6%。我們認為，在來源提示反覆訓練的過程中，提示會逐漸獲得特定任務的知識。SPE 會在不同的訓練階段進行快照。因此，快照提示擁有不同豐富度的特定任務知識。SPE 擁有優異表現的一個可能原因是，SPE 能夠從不同的任務中集合具有不同程度特定任務知識的提示。

Table 2. Few-shot Results(k = 4, 16, 32): FT, AD, and ATP denote Fine-tuning, Adapter, ATTEMPT.
表 2. Few-shot Results(k = 4, 16, 32)：FT、AD、ATP 表示微調、適配器、ATTEMPT。

3.3. Few-shot Adaptation 3.3.少量調整

Following prior work[5], we conduct few-shot adaptation experiments on BoolQ[18], CB[19], and SciTail[20] tasks to prove the generalization ability. Table 2 shows that our method outperforms other methods. The result demonstrates that the proposed SPE shows good generalizability as it performs well in both full-dataset and few-shot settings on different tasks.
根據先前的工作 [5] ，我們在 BoolQ [18] 、CB [19] 和 SciTail [20] 任務上進行了少量適應實驗，以證明泛化能力。 Table 2 顯示我們的方法優於其他方法。結果顯示，建議的 SPE 在不同的任務上，無論是全資料集或少點設定，都有良好的表現，顯示出良好的泛化能力。

3.4. Ablation Studies 3.4.消融研究

We conduct experiments to explore the effectiveness of each component of SPE. The results on STS-B and RTE are shown in Figure 2. no snapshot means that obtain a single prompt on each task. no attention means that ensemble source prompts with an average weight. no task prompt means that do not add the task prompt to combine with the instance prompt.
我們進行實驗來探索 SPE 各個元件的有效性。在 STS-B 和 RTE 上的結果如 Figure 2 所示。無快照表示在每個任務上取得單一提示。無注意力表示以平均權重集合來源提示。無任務提示表示不加入任務提示與實例提示結合。

Fig. 2.  圖 2.

Ablation Studies on STS-B and RTE.
STS-B 和 RTE 的消融研究。

Show All

It can be found that the complete SPE method achieves the best performance. When any part of the model is removed, the performance of the model decreases, indicating that each part is essential to our method. From the results, we derive the following insights. The cross-attention module enables SPE to ensemble the source prompts effectively, and the task-wise prompt provides information at different levels of granularity for the target task. These two components ensure the effectiveness of SPE. Moreover, snapshots at source prompt tuning allow SPE to obtain more prompts with diverse and rich knowledge, which further enhances the performance and stability of SPE.
可以發現完整的 SPE 方法達到了最好的性能。當移除模型的任何部分時，模型的效能都會降低，這說明每個部分對我們的方法都是不可或缺的。從結果中，我們得到以下啟示。交叉注意模組能讓 SPE 有效地集合來源提示，而任務性提示則為目標任務提供不同粒度層級的資訊。這兩個元件可確保 SPE 的有效性。此外，來源提示調整時的快照可讓 SPE 獲得更多具有多元豐富知識的提示，進一步提升 SPE 的效能與穩定性。

Table 3. Results on Chinese tasks
表 3. 中文任務的結果

3.5. Analysis on Universality
3.5.普遍性分析

We extend our method to Chinese NLU tasks and different backbone LM sizes. We choose AFQMC, IFLYTEK, OCNLI, and Tnews from CLUE benchmark[21] as source tasks, using Randeng-T5-77M-MultiTask-Chinese, Randeng-T5-784M-MultiTask-Chinese[22] as pre-trained model parameters.
我們將此方法擴展至中文 NLU 任務和不同的主幹 LM 大小。我們選擇 CLUE 基準 [21] 中的 AFQMC、IFLYTEK、OCNLI 和 Tnews 作為源任務，使用 Randeng-T5-77M-MultiTask-Chinese 和 Randeng-T5-784M-MultiTask-Chinese [22] 作為預訓模型參數。

Table 3 summarizes the performance of baselines and our methods with different LM sizes on four CLUE tasks. Our method largely benefits from backbone LM size increase, outperforming most methods on larger backbone model. We hypothesize that Chinese semantic patterns are more complicated and SPE needs a larger number of parameters to capture enough semantic information. SPE can achieve the best performance on most tasks, which demonstrates the effectiveness and universality of our method on English and Chinese NLU tasks. A possible reason is that our method is language-independent.
Table 3 總結了基線和我們的方法在四個 CLUE 任務上不同 LM 大小的表現。我們的方法主要受益於主幹 LM 大小的增加，在較大的主幹模型上表現優於大多數方法。我們假設中文語意模式較為複雜，SPE 需要較多參數才能捕捉足夠的語意資訊。SPE 在大部分任務上都能達到最佳效能，這證明我們的方法在英文和中文 NLU 任務上的有效性和普遍性。可能的原因是我們的方法與語言無關。