A Brief Analysis of the Exposure Bias Phenomenon in Seq2Seq, and Some Countermeasures
A few days ago I wrote Having Used CRF, Why Not Also Learn About the Faster MEMM?, which discussed the pros and cons of MEMM's local normalization versus CRF's global normalization. This got me thinking about Seq2Seq models, because the typical training scheme for Seq2Seq — Teacher Forcing — is itself a locally-normalized model, so it too suffers from the ailments that come with local normalization, i.e., what we often call "Exposure Bias." With this idea in mind, I thought it through a bit further, and I'm recording the results of that thinking in this post.
Diagram of a classic Seq2Seq model
This post is somewhat more advanced, and is aimed at readers who already have some understanding of Seq2Seq models and want to deepen their understanding or improve model performance further. For an introduction to Seq2Seq, see the earlier posts Playing with Keras Seq2Seq for Automatic Title Generation and From Language Models to Seq2Seq: Transformer as Theater, All Thanks to Masking.
The content of this post is roughly as follows:
1. An analysis of the causes of Exposure Bias, with examples;
2. A simple and practical strategy for mitigating Exposure Bias.
Softmax
First, let's review some facts about softmax. As we all know, for a vector $(x_1,x_2,\dots,x_n)$, its softmax is
\begin{equation}(p_1,p_2,\dots,p_n)=\frac{1}{\sum\limits_{i=1}^n e^{x_i}}\left(e^{x_1},e^{x_2},\dots,e^{x_n}\right)\end{equation}
Since $e^t$ is a strictly monotonically increasing function of $t$, if $x_k$ is the largest among $x_1,x_2,\dots,x_n$, then $p_k$ is also the largest among $p_1,p_2,\dots,p_n$.
For classification problems, the loss we generally use is cross-entropy, namely
\begin{equation}-\log p_t = \log\left(\sum\limits_{i=1}^n e^{x_i}\right) - x_t\end{equation}
where $t$ is the target class. As discussed in In Search of a Smooth Maximum Function, the first term above is actually a smooth approximation of $\max\left(x_1,x_2,\dots,x_n\right)$, so to get an intuitive feel for cross-entropy, we can write
\begin{equation}-\log p_t \approx \max\left(x_1,x_2,\dots,x_n\right) - x_t\end{equation}
In other words, cross-entropy is actually shrinking the gap between the target class's score $x_t$ and the global maximum. This gap can obviously shrink to at most 0, at which point the target class's score is itself the maximum. So the effect of softmax plus cross-entropy is to "make the target class's score become the maximum."
Teacher Forcing
Now let's turn to Seq2Seq. It models the joint probability distribution via conditional factorization:
\begin{equation}\begin{aligned}p(\boldsymbol{y}|\boldsymbol{x})=&\,p(y_1,y_2,\dots,y_n|\boldsymbol{x})\\ =&\,p(y_1|\boldsymbol{x})p(y_2|\boldsymbol{x},y_1)\dots p(y_n|\boldsymbol{x},y_1,\dots,y_{n-1}) \end{aligned}\end{equation}
Each term is naturally also modeled with softmax, i.e.,
\begin{equation}\begin{aligned}&p(y_1|\boldsymbol{x})=\frac{e^{f(y_1;\boldsymbol{x})}}{\sum\limits_{y_1}e^{f(y_1;\boldsymbol{x})}},\\ &p(y_2|\boldsymbol{x},y_1)=\frac{e^{f(y_1,y_2;\boldsymbol{x})}}{\sum\limits_{y_2}e^{f(y_1,y_2;\boldsymbol{x})}},\\ &\dots,\\ &p(y_n|\boldsymbol{x},y_1,\dots,y_{n-1})=\frac{e^{f(y_1,y_2,\dots,y_n;\boldsymbol{x})}}{\sum\limits_{y_n}e^{f(y_1,y_2,\dots,y_n;\boldsymbol{x})}} \end{aligned}\end{equation}
Multiplying them together gives
\begin{equation}p(\boldsymbol{y}|\boldsymbol{x})=\frac{e^{f(y_1;\boldsymbol{x})+f(y_1,y_2;\boldsymbol{x})+\dots+f(y_1,y_2,\dots,y_n;\boldsymbol{x})}}{\left(\sum\limits_{y_1}e^{f(y_1;\boldsymbol{x})}\right)\left(\sum\limits_{y_2}e^{f(y_1,y_2;\boldsymbol{x})}\right)\dots\left(\sum\limits_{y_n}e^{f(y_1,y_2,\dots,y_n;\boldsymbol{x})}\right)}\label{eq:join-target}\end{equation}
and the training objective is
\begin{equation}-\log p(\boldsymbol{y}|\boldsymbol{x})=-\log p(y_1|\boldsymbol{x})-\log p(y_2|\boldsymbol{x},y_1)-\dots -\log p(y_n|\boldsymbol{x},y_1,\dots,y_{n-1})\end{equation}
This direct training objective is called Teacher Forcing, because to compute $-\log p(y_2|\boldsymbol{x},y_1)$ we need to know the true $y_1$, to compute $-\log p(y_3|\boldsymbol{x},y_1,y_2)$ we need to know the true $y_1,y_2$, and so on. It's as if an experienced teacher has already laid out most of the path for us in advance, leaving us only to figure out the next step. This method is simple to train, and when combined with models like CNNs or Transformers it enables parallel training, but it can bring about the Exposure Bias problem.
Exposure Bias
Actually, the very name "Teacher Forcing" already hints that it inherently harbors an Exposure Bias problem. Recall the process by which a teacher teaches a student to solve a problem — the typical steps are:
1. How should we think about the first step;
2. Once we've figured out the first step, what options do we have for the second step;
3. Having settled on the second step, what can we do for the third step;
...
n. With these n-1 steps in hand, the final step isn't hard to see.
This process is actually the same as the assumption underlying Seq2Seq's Teacher Forcing scheme. Anyone with teaching experience knows that students typically nod along and feel like they understand everything — and then when asked to solve problems on their own after class, most of them are utterly lost. Why is that? One reason is Exposure Bias. Put plainly, the problem is that the teacher always assumes the student can come up with the preceding several steps, and then teaches the next step based on that assumption. But what if one of the earlier steps was gotten wrong, or the student simply can't think of it? Then the whole process breaks down — you can't reach the correct answer. This is the Exposure Bias problem.
Beam Search
In fact, when we actually solve problems ourselves, things don't always go this way. If we get stuck and can't decide on some step, we try out several options and keep going, then use the results further down the line to help us decide on that uncertain earlier step. Mapped onto Seq2Seq, this is essentially the decoding process based on Beam Search.
For Beam Search, we should be able to observe that a larger beam size isn't always better — in some cases, a beam size of 1 is actually best, which seems a bit unreasonable, since a larger beam size should in theory find a sequence closer to the optimal one, and thus should be more likely to be correct. In fact, this too is one manifestation of the Exposure Bias phenomenon.
From equation $\eqref{eq:join-target}$ we can see that the scoring function Seq2Seq assigns to a target sequence $y_1,y_2,\dots,y_n$ is:
\begin{equation}f(y_1;\boldsymbol{x})+f(y_1,y_2;\boldsymbol{x})+\dots+f(y_1,y_2,\dots,y_n;\boldsymbol{x})\end{equation}
Normally, we would want the target sequence to have the highest score among all candidate sequences. Following the softmax approach introduced at the start of this post, the probability distribution we should build would be
\begin{equation}p(\boldsymbol{y}|\boldsymbol{x})=\frac{e^{f(y_1;\boldsymbol{x})+f(y_1,y_2;\boldsymbol{x})+\dots+f(y_1,y_2,\dots,y_n;\boldsymbol{x})}}{\sum\limits_{y_1,y_2,\dots,y_n}e^{f(y_1;\boldsymbol{x})+f(y_1,y_2;\boldsymbol{x})+\dots+f(y_1,y_2,\dots,y_n;\boldsymbol{x})}}\label{eq:ideal-target}\end{equation}
But the denominator above requires summing over all paths, which is infeasible, so equation $\eqref{eq:join-target}$ has been widely adopted as a compromise. However, equation $\eqref{eq:join-target}$ is not equivalent to equation $\eqref{eq:ideal-target}$, so even if the model has been successfully optimized, we may still see the phenomenon where "the optimal sequence is not the target sequence."
A Simple Example
Let's work through a simple example. Suppose the sequence length is just 2, the candidate sequences are $(a,b)$ and $(c,d)$, and the target sequence is $(a,b)$. After training is complete, the model's probability distribution is as follows:
$$\begin{array}{c|c} \hline p(a) & p(c)\\ \hline 0.6 & 0.4 \\ \hline \end{array}\qquad \begin{array}{c|c|c|c} \hline p(b|a) & p(d|a) & p(b|c) & p(d|c)\\ \hline 0.55 & 0.45 & 0.1 & 0.9\\ \hline \end{array}$$
If beam size is 1, then since $p(a) > p(c)$, the first step can only output $a$; then since $p(b|a) > p(d|a)$, the second step can only output $b$, successfully producing the correct sequence $(a,b)$. But if beam size is 2, then the first step outputs $(a,0.6),(c,0.4)$, and for the second step we enumerate all combinations, giving us
$$\begin{array}{c|c|c|c} \hline (a, b) & (a, d) & (c, b) & (c, d)\\ \hline 0.33 & 0.27 & 0.04 & 0.36\\ \hline \end{array}$$
so the wrong sequence $(c,d)$ is output.
Is this because the model wasn't trained well enough? Not at all. As mentioned earlier, the point of softmax plus cross-entropy is to make the target's score the maximum. For the first step we have $p(a) > p(c)$, so the training objective for the first step has already been achieved; and for the second step, given that $a$ is known in advance, we have $p(b|a) > p(d|a)$, which shows that the training objective for the second step has also been achieved. So the model can already be considered well-trained — it's just that, perhaps due to limitations in model capacity or other factors, the scores aren't particularly high. But the goal of "maximizing the target's score" has indeed been accomplished.
Thinking About Countermeasures
From the example above, readers may already see where the problem lies: mainly, $p(d|c)$ is too high, while $p(d|c)$ has never been trained on, and there's no explicit mechanism to suppress $p(d|c)$ from growing large. Hence the phenomenon of "the optimal sequence is not the target sequence."
At this point, readers may already think of a straightforward countermeasure: add an extra optimization objective to suppress the scores of those non-target sequences found by Beam Search — wouldn't that work? Indeed, this is an effective solution, and related results were published in the 2016 paper Sequence-to-Sequence Learning as Beam-Search Optimization. But this approach essentially requires running Beam Search on every sample before every training step, which is computationally very expensive. There are also more recent results, such as the ACL 2019 Best Long Paper Bridging the Gap between Training and Inference for Neural Machine Translation, which focuses specifically on solving the Exposure Bias problem. In addition, directly optimizing metrics like BLEU via reinforcement learning can also mitigate Exposure Bias to some extent.
However, as far as I understand, most of the methods devoted to solving Exposure Bias make sweeping changes to the training process, and may even sacrifice the parallelism of the original model's training (they need to recursively sample negative examples — fine if the model itself is an RNN, but very costly if it's a CNN or Transformer). The increase in cost is far greater than the increase in performance.
Constructing Negative Samples
Looking across most of the papers addressing Exposure Bias, and combining that with our own example and intuition above, it's not hard to see that the core idea is to construct representative negative samples and then, during training, lower the probability assigned to these negative samples. So the question becomes how to construct "representative" negative samples. Here I'll present a simple strategy I came up with; experiments show it can mitigate Exposure Bias to some extent and improve text generation performance. Importantly, this strategy is quite simple — essentially plug-and-play — and incurs almost no loss in training efficiency.
The method is very simple: randomly replace some of the Decoder's input words (the Decoder's input words have a special name, "oracle words"), as shown in the figure below:
A simple strategy for mitigating Exposure Bias: directly and randomly replace part of the Decoder's input words with other words.
Here the purple [R] represents a word that has been randomly replaced. Quite a few Exposure Bias papers actually use this same idea, just with different schemes for randomly selecting words. The scheme I propose is very simple:
1. With 50% probability, make no change;
2. With 50% probability, replace 30% of the words in the input sequence, where the replacement candidates are drawn from any word in the original target sequence.
In other words, the probability of random replacement happening is 50%, the proportion of words replaced is 30%, and the sampling pool consists of the words in the target sequence. The inspiration behind this strategy is: although Seq2Seq may not always generate the target sequence exactly, it can usually generate most of the words in the target sequence (though possibly in the wrong order, or with some words repeated). So an input sequence modified in this way can typically serve as a representative negative sample. By the way, I should mention that the 50% and 30% figures were chosen fairly arbitrarily, without careful tuning, since tuning a generative model even once is exhausting enough.
How well does it work? I ran two title (abstract) generation experiments (the first two tasks from CLGE), where the baseline is task_seq2seq_autotitle_csl.py. The code is open-sourced at:
GitHub: https://github.com/bojone/exposure_bias
The results are shown in the table below:
$$\begin{array}{c} \text{CSL title generation results}\\ {\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{baseline} & 1 & 63.81 & 65.45 & 54.91 & 45.52 \\ \text{random replace} & 1 & \textbf{64.44} & \textbf{66.09} & \textbf{55.56} & \textbf{46.1} \\ \hline \text{baseline} & 2 & 64.44 & 66.09 & 55.75 & 46.39 \\ \text{random replace} & 2 & \textbf{65.04} & \textbf{66.75} & \textbf{56.51} & \textbf{47.19} \\ \hline \text{baseline} & 3 & 64.75 & 66.34 & 56.06 & 46.7 \\ \text{random replace} & 3 & \textbf{65.15} & \textbf{66.96} & \textbf{56.74} & \textbf{47.42} \\ \hline \end{array}$$}\\
\\
\text{LCSTS Summary Generation Experiment Results}\\
{$$\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{baseline} & 1 & 27.99 & 29.57 & \textbf{18.04} & \textbf{11.72} \\ \text{random replace} & 1 & \textbf{28.61} & \textbf{29.92} & 17.72 & 11.23 \\ \hline \text{baseline} & 2 & \textbf{29.2} & 30.7 & \textbf{19.17} & \textbf{12.64} \\ \text{random replace} & 2 & 29.15 & \textbf{30.79} & 18.56 & 11.75 \\ \hline \text{baseline} & 3 & \textbf{29.45} & \textbf{30.95} & \textbf{19.5} & \textbf{12.93} \\ \text{random replace} & 3 & 29.14 & 30.88 & 18.76 & 11.91 \\ \hline \end{array}$$}
\end{array}
We can see that on the CSL task, the random-replacement-based strategy consistently improves all text-generation metrics, while on the LCSTS task the metrics are more mixed — some better, some worse. Considering that LCSTS is inherently a harder task and its metrics are already low to begin with, the CSL results should be considered more convincing. This suggests that the strategy proposed above is indeed worth trying. (Note: all experiments were repeated twice and averaged, so the results should be reasonably reliable.)
Adversarial Training
Having gotten this far, let's let our imagination run a bit further: since one approach to solving Exposure Bias is to construct representative negative-sample inputs — which really amounts to making the model robust to predicting correctly under perturbation — and since we just recently discussed a method for generating perturbed samples, why not use that? Indeed, that method is adversarial training. If we directly add adversarial training to the baseline model, can it improve model performance? For simplicity, I ran an experiment adding gradient penalty (which is also a form of adversarial training) to the baseline model, with the comparison results as follows:
$$\begin{array}{c} \text{CSL title generation results}\\ {\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{baseline} & 1 & 63.81 & 65.45 & 54.91 & 45.52 \\ \text{random replace} & 1 & 64.44 & 66.09 & 55.56 & 46.1 \\ \text{gradient penalty} & 1 & \textbf{65.41} & \textbf{67.29} & \textbf{56.64} & \textbf{47.37} \\ \hline \text{baseline} & 2 & 64.44 & 66.09 & 55.75 & 46.39 \\ \text{random replace} & 2 & 65.04 & 66.75 & 56.51 & 47.19 \\ \text{gradient penalty} & 2 & \textbf{65.94} & \textbf{67.84} & \textbf{57.38} & \textbf{48.16} \\ \hline \text{baseline} & 3 & 64.75 & 66.34 & 56.06 & 46.7 \\ \text{random replace} & 3 & 65.15 & 66.96 & 56.74 & 47.42 \\ \text{gradient penalty} & 3 & \textbf{66.1} & \textbf{68.08} & \textbf{57.7} & \textbf{48.56} \\ \hline \end{array}$$}\\
\\
\text{LCSTS Summary Generation Experiment Results}\\
{$$\begin{array}{c|c|cccc} \hline & \text{beam size} & \text{Rouge-L} & \text{Rouge-1} & \text{Rouge-2} & \text{BLEU} \\ \hline \text{baseline} & 1 & 27.99 & 29.57 & 18.04 & 11.72 \\ \text{random replace} & 1 & 28.61 & 29.92 & 17.72 & 11.23 \\ \text{gradient penalty} & 1 & \textbf{30.75} & \textbf{31.83} & \textbf{19.38} & \textbf{11.78} \\ \hline \text{baseline} & 2 & 29.2 & 30.7 & 19.17 & \textbf{12.64} \\ \text{random replace} & 2 & 29.15 & 30.79 & 18.56 & 11.75 \\ \text{gradient penalty} & 2 & \textbf{30.88} & \textbf{32.19} & \textbf{19.96} & 12.32 \\ \hline \text{baseline} & 3 & 29.45 & 30.95 & 19.5 & \textbf{12.93} \\ \text{random replace} & 3 & 29.14 & 30.88 & 18.76 & 11.91 \\ \text{gradient penalty} & 3 & \textbf{30.39} & \textbf{31.76} & \textbf{19.74} & 12.14 \\ \hline \end{array}$$}
\end{array}
We can see that adversarial training (gradient penalty) further improves all metrics for CSL generation, while on LCSTS it mainly improves the Rouge metrics, with BLEU dropping somewhat. So adversarial training can also be added to the list of "tricks with potential for improving text generation models."
Summary
This post discussed the Exposure Bias phenomenon in Seq2Seq, attempted to analyze its causes both intuitively and theoretically, and presented simple and practical countermeasures for mitigating Exposure Bias — including a random-replacement strategy I devised, as well as a strategy based on adversarial training. The advantage of both strategies is that they are essentially plug-and-play, and experiments show that they can improve various text-generation metrics to some degree.
Translated automatically with claude-sonnet-5; all equations are reproduced verbatim from the source. Copyright remains with the original author.