2.1. Language posterior bias
2.1. 語言後驗偏差

The language posterior bias approach [17] has been developed on the hybrid CTC/attention ASR model, which comprises an encoder module, a decoder module, and a CTC module [19], [20]. These encoder and decoder modules consist of conformer encoder layers and transformer decoder layers [2], [9], respectively.
語言後驗偏差方法 [17] 是在混合 CTC/注意力 ASR 模型上開發的，該模型包括編碼器模塊、解碼器模塊和 CTC 模塊 [19] , [20] 。這些編碼器和解碼器模塊分別由 conformer 編碼器層和 transformer 解碼器層 [2] , [9] 組成。

Consider a speech signal with its acoustic features X = (x_t ∈ ℝ^F|t = 1,…,T) and token sequence W = (w_n ∈ V|n = 1,…,N), where V is a vocabulary of size V, T and N are the lengths of acoustic features and token sequence, respectively. The encoder generates output H = (h_t ∈ ℝ^D|t = 1,…,T₁) from X, which are subsequently fed into the decoder and CTC modules. Tokens are first embedded into W = (w_n ∈ ℝ^D|n = 1,…,N) before being fed into the decoder module along with H. The ASR model is optimized with a language diarization (LD) decoder jointly, where the LD decoder computes a V ^ld-dimensional token-level language posterior bias p(l_n−1|w_1:n−1,X). Here, V ^ld is the language vocabulary size and l_n−1 is the language index for the n-th token. The token embedding w_n−1is then biased by its language posterior. The ASR decoder output is subsequently computed via

View Source\begin{align*} & {\mathbf{H}} = {\text{ Encoder }}({\mathbf{X}}),\tag{1} \\ & {\mathbf{w}}_{n - 1}^\prime = \operatorname{Concat} \left( {{{\mathbf{w}}_{n - 1}},{\mathbf{p}}\left( {{l_{n - 1}}\mid {w_{1:n - 1}},{\mathbf{X}}} \right)} \right),\tag{2} \\ & p\left( {{w_n}\mid {w_{1:n - 1}},{\mathbf{X}}} \right) = \operatorname{Decoder} \left( {{\mathbf{w}}_{1:n - 1}^\prime ,{\mathbf{H}}} \right),\tag{3}\end{align*}

Ljoint =αLctc+(1−α)Latt+βLld(4)

View Source\begin{equation*}{\mathcal{L}_{{\text{joint }}}} = \alpha {\mathcal{L}_{{\text{ctc}}}} + (1 - \alpha ){\mathcal{L}_{{\text{att}}}} + \beta {\mathcal{L}_{{\text{ld}}}}\tag{4}\end{equation*}

考慮一個具有其聲學特徵 X = (x _t ∈ ℝ ^F |t = 1,…,T) 和標記序列 W = (w _n ∈ V |n = 1,…,N) 的語音信號，其中 V 是大小為 V 的詞彙表，T 和 N 分別是聲學特徵和標記序列的長度。編碼器從 X 生成輸出 H = (h _t ∈ ℝ ^D |t = 1,…,T ₁ )，然後將其餵入解碼器和 CTC 模塊。標記首先被嵌入到 W = (w _n ∈ ℝ ^D |n = 1,…,N) 中，然後與 H 一起餵入解碼器模塊。語音識別模型與語言分割（LD）解碼器聯合優化，其中 LD 解碼器計算 V ^ld 維度的標記級語言後偏差 p(l _n−1 |w _1:n−1 ,X)。這裡，V ^ld 是語言詞彙表大小，l _n−1 是第 n 個標記的語言索引。然後，標記嵌入 w _n−1 受其語言後偏差影響。語音識別解碼器的輸出隨後通過

H= Encoder (X),w′n−1=Concat(wn−1,p(ln−1∣w1:n−1,X)),p(wn∣w1:n−1,X)=Decoder(w′1:n−1,H),(1)(2)(3)

View Source\begin{align*} & {\mathbf{H}} = {\text{ Encoder }}({\mathbf{X}}),\tag{1} \\ & {\mathbf{w}}_{n - 1}^\prime = \operatorname{Concat} \left( {{{\mathbf{w}}_{n - 1}},{\mathbf{p}}\left( {{l_{n - 1}}\mid {w_{1:n - 1}},{\mathbf{X}}} \right)} \right),\tag{2} \\ & p\left( {{w_n}\mid {w_{1:n - 1}},{\mathbf{X}}} \right) = \operatorname{Decoder} \left( {{\mathbf{w}}_{1:n - 1}^\prime ,{\mathbf{H}}} \right),\tag{3}\end{align*} 計算，其中 Concat(•) 表示連接操作。矩陣 W′=(w′n∈RD+Vld∣n=1,…,N) 包含語音識別解碼器的輸入標記嵌入，隨後通過線性層投影回 D 維度。 該模型經過

Ljoint =αLctc+(1−α)Latt+βLld(4)

View Source\begin{equation*}{\mathcal{L}_{{\text{joint }}}} = \alpha {\mathcal{L}_{{\text{ctc}}}} + (1 - \alpha ){\mathcal{L}_{{\text{att}}}} + \beta {\mathcal{L}_{{\text{ld}}}}\tag{4}\end{equation*} 進行優化，解碼過程類似於 (3) ，但 ASR 解碼器的輸入標記嵌入 W 被 W ^′ 取代。整合模型通過多任務目標函數進行優化。在這裡，β 是多任務學習參數，ℒ _ld 是 LD 解碼器預測和地面真實語言標籤之間的標籤平滑交叉熵損失。

2.2. Interactive language biases
2.2. 互動式語言偏見

We propose to extend the language posterior bias method to frame-level language information. We note that frame-level language identification (LID) is undesirable since the LID performance generally degrades with shorter speech [21], [22], [23]. However, since acoustic frames are tightly associated with tokens in ASR, frame-level language identification over H may benefit from token-level language diarization in (2). The frame-level language posteriors, therefore, enhance the hidden output H to achieve high language discrimination before being fed into the ASR and LD decoders. Consequently, frame- and token-level language posteriors interact and jointly improve the model performance in CS-ASR.
我們提議將語言後驗偏見方法擴展到框級語言信息。我們注意到框級語言識別（LID）是不可取的，因為 LID 性能通常隨著較短的語音 [21] ， [22] ， [23] 而下降。然而，由於聲學框緊密地與 ASR 中的標記相關，因此在 H 上的框級語言識別可能受益於 (2) 中的標記級語言話語。因此，框級語言後驗增強了隱藏輸出 H，以實現高語言區分度，然後再餵入 ASR 和 LD 解碼器。因此，框級和標記級語言後驗相互作用，共同提高了 CS-ASR 模型的性能。

Fig. 1.  圖 1.

The hybrid CTC/attention model with interactive language biases.
具有互動語言偏差的混合 CTC/注意力模型。

Show All

With reference to Fig. 1, the frame-level language bias is achieved through a LID layer before being concatenated with the hidden output. In particular, the biased hidden output H^′ is computed via

View Source\begin{equation*}{\mathbf{h}}_t^\prime = \operatorname{Concat} \left( {{{\mathbf{h}}_t},{\mathbf{p}}\left( {{l_t}\mid {{\mathbf{h}}_{\mathbf{t}}}} \right)} \right){\text{, }}\tag{5}\end{equation*}

p(wn∣w1:n−1,X)=Decoder(w′1:n−1,H′).(6)

View Source\begin{equation*}p\left( {{w_n}\mid {w_{1:n - 1}},{\mathbf{X}}} \right) = \operatorname{Decoder} \left( {{\mathbf{w}}_{1:n - 1}^\prime ,{{\mathbf{H}}^\prime }} \right).\tag{6}\end{equation*}

參考 Fig. 1 ，在與隱藏輸出串聯之前，通過 LID 層實現幀級語言偏差。特別地，通過

h′t=Concat(ht,p(lt∣ht)), (5)

View Source\begin{equation*}{\mathbf{h}}_t^\prime = \operatorname{Concat} \left( {{{\mathbf{h}}_t},{\mathbf{p}}\left( {{l_t}\mid {{\mathbf{h}}_{\mathbf{t}}}} \right)} \right){\text{, }}\tag{5}\end{equation*} 計算偏置隱藏輸出 H ^′ ，隨後替換 (3) 中的 H 以促進幀和標記之間的語言信息交互作用。此外，H ^′ 也用於開發語言感知 CTC 模塊。ASR 解碼器輸出隨後通過

p(wn∣w1:n−1,X)=Decoder(w′1:n−1,H′).(6)

View Source\begin{equation*}p\left( {{w_n}\mid {w_{1:n - 1}},{\mathbf{X}}} \right) = \operatorname{Decoder} \left( {{\mathbf{w}}_{1:n - 1}^\prime ,{{\mathbf{H}}^\prime }} \right).\tag{6}\end{equation*} 實現。

During training, the frame-level LID is optimized in an unsupervised manner similar to [10] (i.e., frame-level language annotations are not provided during training). However, frames in H^′ are trained to be aligned with their corresponding token-level language labels within the language diarization decoder. An assumption made here is that an accurate frame-to-token alignment enriches the unsupervised LID process with supervised information through backpropagation. Optimization of the model is achieved similarly to that of (4).
在訓練期間，幀級 LID 以無監督方式進行優化，類似於 [10] （即，在訓練期間不提供幀級語言標註）。然而，在語言分割解碼器中，H ^′ 中的幀被訓練以與其對應的標記級語言標籤對齊。這裡假設的一個前提是，準確的幀到標記的對齊通過反向傳播豐富了無監督 LID 過程的監督信息。模型的優化類似於 (4) 。

During inference, the frame- and token-level language posterior are computed before biasing the hidden output and the ASR decoding, respectively. The decoding process is similar to that presented in [19], which is defined to maximize the linear combination of the logarithmic CTC and attention objectives, i.e.,

View Source\begin{equation*}\hat W = \mathop {\operatorname{argmax} }\limits_W \left\{ {\alpha \log {p_{{\text{ctc }}}}(W\mid {\mathbf{X}}) + (1 - \alpha )\log {p_{{\text{att }}}}(W\mid {\mathbf{X}})} \right\}.\tag{7}\end{equation*}

在推論過程中，分別計算了框架級和標記級的語言後驗，然後分別對隱藏輸出和 ASR 解碼進行偏置。解碼過程類似於 [19] 中提出的過程，該過程被定義為最大化對數 CTC 和注意力目標的線性組合，即

W^=argmaxW{αlogpctc (W∣X)+(1−α)logpatt (W∣X)}.(7)

View Source\begin{equation*}\hat W = \mathop {\operatorname{argmax} }\limits_W \left\{ {\alpha \log {p_{{\text{ctc }}}}(W\mid {\mathbf{X}}) + (1 - \alpha )\log {p_{{\text{att }}}}(W\mid {\mathbf{X}})} \right\}.\tag{7}\end{equation*} 。

3.1. Dataset and experiment setup
3.1. 數據集和實驗設置

All experiments are conducted on datasets from the ASRU 2019 Mandarin-English code-switching speech recognition challenge [24]. This challenge comprises four datasets, including a 500-hour Mandarin-only training set, a 200-hour intra-sentence English-Mandarin code-switching training set, a 40-hour intra-sentence English-Mandarin code-switching development set, and a 20-hour intra-sentence English-Mandarin code-switching test set. We employed ESPnet¹ to train all models on the 200-hour CS training set, which are validated on the development set and evaluated on the test set [25].
所有實驗均在 ASRU 2019 年普通話-英語代碼切換語音識別挑戰的數據集上進行。該挑戰包括四個數據集，包括一個 500 小時的僅普通話訓練集，一個 200 小時的句內英語-普通話代碼切換訓練集，一個 40 小時的句內英語-普通話代碼切換開發集，以及一個 20 小時的句內英語-普通話代碼切換測試集。我們使用 ESPnet 在 200 小時的代碼切換訓練集上訓練所有模型，這些模型在開發集上進行驗證，並在測試集上進行評估。

SpecAugment is applied to augment the training data [26]. Words are transformed into a total of V = 6,923 tokens that include 3,000 English byte-pair encoding (BPE) tokens, 3,920 Mandarin characters, and three special tokens for unk, blank, and sos/eos. All tokens are transformed to language labels building V^ld, which comprises e for English BPEs, m for Mandarin characters, and sos/eos. Language labels in V^ld are used as LD outputs. We extracted F = 83 dimensional features comprising 80-dimensional log-fbanks and 3dimensional pitch for each speech sample before applying global mean and variance normalization.
SpecAugment 被應用於擴增訓練數據 [26] 。單詞被轉換為總共 V = 6,923 個標記，其中包括 3,000 個英文字節對編碼（BPE）標記，3,920 個中文字符，以及三個特殊標記用於未知、空白和 sos/eos。所有標記被轉換為語言標籤構建 V ^ld ，其中包括 e 代表英文 BPE，m 代表中文字符，以及 sos/eos。V ^ld 中的語言標籤被用作 LD 輸出。在應用全局均值和方差歸一化之前，我們提取了 F = 83 維特徵，每個語音樣本包括 80 維的 log-fbanks 和 3 維的音高。

We chose a hybrid CTC/Attention ASR model comprising twelve conformer encoder layers and six transformer decoder layers as the baseline model [2], [9], [27]. In addition, we adopted the multi-task learning model and the language posterior bias approach as our benchmark [17]. All self-attention encoder and decoder layers have four attention heads with input and output dimensions being D = 256, and the inner layer of the position-wise feed-forward network is of 2048 dimensions. During training, we set parameters α = 0.3 and β = 0.8 in (4), while a label smoothing factor of 0.1 is used for all cross-entropy losses. The ten best models during validation are averaged for inference. All models are trained on two GeForce RTX 3090 GPUs, where the baseline was trained for seventy epochs, while other models were trained for eighty epochs due to their higher number of parameters.
我們選擇了一個混合的 CTC/Attention ASR 模型，包括十二個 conformer 編碼器層和六個 transformer 解碼器層作為基準模型。此外，我們採用了多任務學習模型和語言後驗偏差方法作為我們的基準。所有自注意力編碼器和解碼器層都有四個注意力頭，輸入和輸出維度為 D = 256，位置智能前向網絡的內部層為 2048 維。在訓練期間，我們設置了參數 α = 0.3 和 β = 0.8，在所有交叉熵損失中使用了 0.1 的標籤平滑因子。在推理過程中，我們對驗證期間的十個最佳模型進行了平均。所有模型都在兩個 GeForce RTX 3090 GPU 上進行訓練，其中基準模型訓練了七十個時期，而其他模型由於參數數量較多，因此訓練了八十個時期。

During inference, we set parameters α = 0.4 in (7). Tenbest beam search is used before selecting the best hypothesis. The language model (LM) used in this paper is a sixteen-layer transformer model with each attention layer comprising eight heads. The proposed systems are evaluated by employing mix error rate (MER) comprising word error rate (WER) for English and character error rate (CER) for Mandarin.
在推論過程中，我們在 (7) 中設置參數 α = 0.4。在選擇最佳假設之前使用 Tenbest 波束搜索。本文中使用的語言模型（LM）是一個十六層的 transformer 模型，每個注意力層包含八個 heads。提出的系統通過使用混合錯誤率（MER），包括英語的詞錯誤率（WER）和普通話的字錯誤率（CER）進行評估。

Table 1. Performance comparison of models utilizing various-level language information without using external language model by employing MER (%)
表 1。利用各級語言信息的模型性能比較，不使用外部語言模型，採用 MER（%）進行評估

3.2. Baseline and single language-biased models
3.2。基準和單一語言偏向模型

The results of the benchmark models are shown in Table 1 as systems 1.0, 1.1, and 1.2. Compared to the vanilla hybrid CTC-attention CS-ASR model, incorporating an auxiliary language diarization task and employing token-level LPB proposed in [17] lead to higher performance. These indicate that incorporating language information benefits the CS-ASR process, which is consistent with the observation presented in [17]. However, the token-level LPB approach shows no performance improvement over the multi-task optimization since Mandarin is the primary language in this dataset and languages do not switch frequently.
基準模型的結果顯示在 Table 1 中，系統 1.0、1.1 和 1.2。與純粹的混合 CTC-注意力 CS-ASR 模型相比，將輔助語言辨識任務和採用 [17] 中提出的基於標記級的 LPB 的方法納入，可以提高性能。這表明納入語言信息有助於 CS-ASR 過程，這與 [17] 中提出的觀察一致。然而，基於標記級的 LPB 方法在多任務優化方面並未表現出性能改善，因為普通話是該數據集中的主要語言，語言不會頻繁切換。

The above data characteristics also result in performance degradation for model configuration 1.3 when frame-level LID is not sufficiently accurate. To prevent the CTC outputs from interacting with the unsupervised frame-level LID, the input of CTC is set to H while the input of the ASR decoder is set to H^′ in model configuration 1.3. As mentioned in Section 2.2, frame-level LID is generally less accurate than token-level LID. Those incorrect language posteriors may increase language confusion when being transmitted into the ASR decoder module.
上述數據特徵也導致模型配置 1.3 在幀級 LID 不夠準確時性能下降。為了防止 CTC 輸出與無監督的幀級 LID 產生交互作用，模型配置 1.3 中 CTC 的輸入設置為 H，而 ASR 解碼器的輸入設置為 H ^′ 。正如 Section 2.2 中所提到的，幀級 LID 通常比標記級 LID 不夠準確。這些不正確的語言後驗概率可能會在傳輸到 ASR 解碼器模塊時增加語言混淆。

3.3. Results of models with interactive language biases
具有互動語言偏差模型的結果

We next investigate how the interaction between frame- and token-level language information improves the model performance using systems 1.4, 1.5, and 1.6. In model configuration 1.4, as opposed to model configuration 1.3, the input of CTC is set to H^′ so as to bias the CTC module with language information. The CTC performs frame-level classification before computing the optimal alignment, where the language biases are infused with acoustic features and combined intrinsically when generating tokens. Therefore, the performance improvement shown in Table 1 when comparing model configuration 1.4 with 1.3 indicates that language-biased frames can also perform better than vanilla frames. This underpins the efficacy of the frame-level language bias when being used for CTC.
我們接下來研究幀級和標記級語言信息之間的互動如何提高系統 1.4、1.5 和 1.6 的模型性能。在模型配置 1.4 中，與模型配置 1.3 相反，CTC 的輸入設置為 H ^′ ，以便用語言信息偏置 CTC 模塊。在計算最佳對齊之前，CTC 執行幀級分類，其中語言偏差與聲學特徵融合，並在生成標記時內在結合。因此，比較模型配置 1.4 和 1.3 時顯示的性能改善 Table 1 表明，具有語言偏差的幀也可以比普通幀表現更好。這支持了在用於 CTC 時使用幀級語言偏差的功效。

Table 2. Performance comparison of models using external language model during inference by employing MER (%), where "Reduction" denotes the absolute MER reduction compared to their no-LM counterparts
表 2. 使用外部語言模型在推斷期間的模型性能比較，通過使用 MER（%）來衡量，“減少”表示與其無 LM 對應物相比的絕對 MER 減少

Model configuration 1.5 employs frame- and token-level language biases jointly but excludes the CTC module from being biased. Model configuration 1.5 shows significantly higher performance than single-language-biased models 1.2 and 1.3. This implies that the use of token-level language bias compensates for the inaccurate frame-level LID especially when model configuration 1.3 degrades the performance of model 1.1, which demonstrates that the interactive language biases are effective for CS-ASR.
模型配置 1.5 同時使用框架級和標記級語言偏差，但排除了 CTC 模塊的偏差。模型配置 1.5 的性能顯著高於單語言偏差模型 1.2 和 1.3。這意味著在模型配置 1.3 降低模型 1.1 的性能時，標記級語言偏差的使用補償了不準確的框架級 LID，這表明交互式語言偏差對 CS-ASR 有效。

Model configuration 1.6 further combines two language biases with the CTC module and achieves the highest performance among all model considerations, with a 7.8% relative improvement compared to the baseline model. It is not surprising that this configuration achieves higher performance than model configurations 1.4 and 1.5 since biasing CTC with language information improves the performance over the encoder LPB approach. In addition, the above implies that enriching all modules within a CS-ASR model with language information obtains higher gain compared to a single module, which is consistent with our proposition in Section 1.
模型配置 1.6 進一步結合了兩種語言偏差與 CTC 模塊，並在所有模型考慮中實現了最高性能，相對於基準模型提高了 7.8%。這個配置實現比模型配置 1.4 和 1.5 更高的性能並不奇怪，因為用語言信息偏置 CTC 會比使用編碼器 LPB 方法提高性能。此外，上述意味著在 CS-ASR 模型中豐富所有模塊的語言信息相比單個模塊獲得更高的增益，這與我們在 Section 1 中的主張一致。

3.4. Results of external language modeling
外部語言建模的結果

Since the end-to-end ASR approaches internally perform language modeling, we explore whether the internal LM is stronger than the external LM when being trained on the same corpora.
由於端到端的 ASR 方法內部執行語言建模，我們探討當在相同語料庫上訓練時，內部 LM 是否比外部 LM 更強大。

We present the results with respect to external language models in Table 2. The vanilla hybrid CTC/attention model shows higher performance after being integrated with external LM during inference. However, the results show that all language-aware CS-ASR models suffer from performance degradation compared to the baseline model. This implies that the CS-ASR model biased by language information could develop a more robust internal language model compared to an external model trained on the same text data. Since training an external language model can be time-consuming [28], robust internal language modeling can thus be concluded as an advantage of the proposed interactive language biases approach.
我們在 Table 2 中呈現了與外部語言模型相關的結果。在推斷過程中，普通的混合 CTC/注意力模型在與外部 LM 整合後表現更好。然而，結果顯示，所有具有語言感知能力的 CS-ASR 模型與基準模型相比都表現出性能下降。這意味著受語言信息影響的 CS-ASR 模型可能會發展出比在相同文本數據上訓練的外部模型更強大的內部語言模型。由於訓練外部語言模型可能耗時 [28] ，因此可以得出結論，穩健的內部語言建模是所提出的互動式語言偏見方法的優勢。

Fig. 2.  圖 2.

Comparison between attention matrices with respect to the frame-to-token alignment within language diarization decoder after employing token-level LPB (above) and interactive language biases (below).
在採用令牌級 LPB（上方）和互動式語言偏見（下方）後，在語言分離解碼器內關於幀到令牌對齊的注意力矩陣之間的比較。

Show All