What Grade Level Has BERT Reached? Using Seq2Seq to Tackle Elementary School Math Word Problems

BERT可以上几年级了？Seq2Seq“硬刚”小学数学应用题 苏剑林 · 2020-10-19 · Machine Learning language model text generation

Those years spent on "chicken-rabbit cage" problems

"Surplus and deficit problems," "age problems," "tree-planting problems," "cows grazing grass problems," "profit problems"... Back in elementary school, were you ever tormented by all sorts of fancy math word problems? No worries — nowadays machine learning models can help us solve word problems too. Let's see what grade level it can reach!

This post presents a baseline for solving elementary school Math Word Problems, trained on the ape210k dataset. It directly uses a Seq2Seq model to generate executable math expressions, and the final Large model achieves 75% accuracy, notably higher than the results reported in the ape210k paper. By "tackle head-on," I mean that we did not apply any special transformation to the expressions, nor did we resort to template-based processing — we simply generate readable expressions similar to how humans would solve the problems. more

Data Processing

Let's first take a look at what the ape210k dataset looks like:

{
    "id": "254761",
    "segmented_text": "小 王 要 将 150 千 克 含 药 量 20% 的 农 药 稀 释 成 含 药 量 5% 的 药 水 ． 需 要 加 水 多 少 千 克 ？",
    "original_text": "小王要将150千克含药量20%的农药稀释成含药量5%的药水．需要加水多少千克？",
    "ans": "450",
    "equation": "x=150*20%/5%-150"
}

{
    "id": "325488",
    "segmented_text": "一 个 圆 形 花 坛 的 半 径 是 4 米 ， 现 在 要 扩 建 花 坛 ， 将 半 径 增 加 1 米 ， 这 时 花 坛 的 占 地 面 积 增 加 了 多 少 米 * * 2 ．",
    "original_text": "一个圆形花坛的半径是4米，现在要扩建花坛，将半径增加1米，这时花坛的占地面积增加了多少米**2．",
    "ans": "28.26",
    "equation": "x=(3.14*(4+1)**2)-(3.14*4**2)"
}

As you can see, the fields we mainly care about are original_text, equation, and ans, where original_text is the problem statement, equation is the arithmetic process (generally starting with x=), and ans is the final answer. We want to train a model that generates equation from original_text, from which the answer can then be obtained directly via Python's eval function.

However, some preprocessing is needed, because not all of the equations given in ape210k can be directly evaluated. For instance, in the example above, 150*20%/5%-150 is an invalid expression as far as Python is concerned. Here's the processing I applied:

1. Percentages of the form a% are uniformly replaced with (a/100);
2. Mixed numbers of the form a(b/c) are uniformly replaced with (a+b/c);
3. Proper fractions of the form (a/b) have their parentheses removed in the problem text, becoming a/b;
4. Ratio colons : are uniformly replaced with /.

After this processing, most equations can be evaluated directly, and we can compare the result against ans, keeping only those problems where the two match. There's still one more refinement to make, though: the resulting expressions may contain redundant parentheses (i.e., removing them yields an equivalent expression), so we add a step to strip such parentheses — iterating over every pair of parentheses and removing it if doing so leaves the result unchanged. This gives expressions with a shorter average length, and shorter expressions are easier to generate.

In the end, we obtained the following usable dataset:

$$\begin{array}{c|ccc} \hline & \text{training set} & \text{val set} & \text{test set} \\ \hline \text{original count} & 200488 & 5000 & 5000\\ \hline \text{retained count} & 200390 & 4999 & 4998\\ \hline \end{array}$$

What's left over is basically flawed or garbled problems, which we ignore for now.

Model Overview

There isn't really much to say about the model itself: it simply takes original_text as input and equation as output, using "BERT + UniLM" as the base architecture to train a Seq2Seq model. If you have any doubts about the model, please read From Language Models to Seq2Seq: Transformer as Theater, All Thanks to Masking.

Project link: http://github.com/bojone/ape210k_baseline

I trained the model on a single 22G TITAN RTX card, using the Adam optimizer with a learning rate of 2e-5. The Base version used batch_size=32 and required roughly 25 epochs of training, with each epoch taking about 50 minutes (including validation set evaluation time); the Large version used batch_size=16 and required roughly 15 epochs, with each epoch taking about 2 hours (including validation set evaluation time).

By the way, speaking of Large — since UniLM borrows part of the weights from the MLM component, we can't use HIT's open-sourced RoBERTa-wwm-ext-large, because the MLM weights in that release were randomly initialized (though its Base version is fine and can be used). For the Large version, I recommend the weights open-sourced by Tencent UER. These were originally in PyTorch format; I converted them to TF format, and they can be downloaded here (extraction code l0k6).

The results are as follows:

$$\begin{array}{c|ccc} \hline & \text{beam_size} & \text{val set} & 测试集 \\ \hline \text{Base} & 1 & 71.67\% & 71.65\%\\ \text{Base} & 2 & 71.81\% & 72.27\%\\ \text{Base} & 3 & \textbf{71.85}\% & \textbf{72.35}\%\\ \hline \text{Large} & 1 & 74.51\% & 74.43\%\\ \text{Large} & 2 & 74.97\% & 74.99\%\\ \text{Large} & 3 & \textbf{75.04}\% & \textbf{75.01}\%\\ \hline \end{array}$$

The Large model's result is already noticeably higher than the 70.20% reported in the ape210k paper, Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problems, which shows that the model here is a reasonably decent baseline. I suspect that applying some Seq2Seq tricks to alleviate the Exposure Bias problem (see A Brief Analysis of Exposure Bias in Seq2Seq and Countermeasures) could further improve the model; introducing a copy mechanism might also help improve consistency between the numbers in the output and those in the input; and there might be ways to further shorten the sequence length (e.g., replacing the four-character 3.14 with the two-letter pi). I'll leave these for readers to try out.

Standardized Output

From a purely modeling standpoint, our task is actually already complete — the model just needs to output the expression, and evaluation only requires checking whether the result of evaluating the expression matches the reference answer. But from a practical usability standpoint, we still need to further standardize the output, i.e., decide, based on the specific problem, whether the output should be a decimal, an integer, a fraction, or a percentage, and so on. This requires us to: 1) decide when to output which format; 2) convert the result according to the specified format.

The first step is fairly simple — generally, some keywords in the problem statement or the equation are enough to make this determination. For example, if the expression contains a decimal, the output is generally a decimal as well; if the question asks "how many vehicles," "how many items," "how many people," etc., then the output should be an integer; if the question directly asks "what fraction" or "what percentage," then the answer should correspondingly be a fraction or a percentage. The trickiest case is probably rounding-type problems — for instance, "Each cake box costs 7.90 yuan; with 50 yuan, at most how many boxes of cake can you buy?" requires us to round 50/7.90 down, but sometimes rounding should go up instead. What surprised me, though, is that ape210k doesn't actually contain any rounding-type problems, so this issue doesn't arise here. If you encounter a dataset that does include rounding, and rule-based judgment turns out to be difficult, the most direct approach would be to include the rounding symbol in the equation itself and let the model predict it.

The second step looks a bit more complicated, mainly due to the fraction scenario. Readers may not be aware of how to keep an expression's result as a fraction — if you directly do eval('(1+2)/4'), you get 0.75 (in Python 3), but sometimes we want the fractional result 3/4 instead. In fact, preserving fractions as fractions falls under the realm of CAS (Computer Algebra System) — in other words, it's symbolic computation rather than numerical computation. And Python conveniently has just such a tool: SymPy. We can use SymPy to achieve what we want. See the example below for details:

from sympy import Integer
import re

r = (Integer(1) + Integer(2)) / Integer(4)
print(r)  # 输出是 3/4 而不是 0.75

equation = '(1+2)/4'
print(eval(equation))  # 输出是 0.75

new_equation = re.sub('(\d+)', 'Integer(\\1)', equation)
print(new_equation)  # 输出是 (Integer(1)+Integer(2))/Integer(4)
print(eval(new_equation))  # 输出是 3/4

Summary

This post introduced a baseline for solving math word problems with a Seq2Seq model, the main idea being to directly convert the problem into an evaluable expression via "BERT + UniLM," followed by some practical experience on standardizing the results. Using UniLM built on the BERT Large model, we achieved 75% accuracy, surpassing the results released with the original paper.

So, what grade level do you think it's reached?