Blowing up over a "miracle paper" with data pulled out of thin air

开局一段扯，数据全靠编？真被一篇“神论文”气到了 苏剑林 · 2021-12-04 · Life-Feeling sentiment model

This post is about my experience getting angry at a "miracle paper" that came out yesterday.

The "miracle paper" in question is How not to Lie with a Benchmark: Rearranging NLP Leaderboards. The gist of the paper is that many current leaderboards use the arithmetic mean when averaging scores, and it argues that the geometric mean and harmonic mean are more reasonable choices. Most importantly, it recomputes the rankings on leaderboards like GLUE and SuperGLUE using the geometric mean and harmonic mean, and finds that the models that had previously surpassed human performance no longer do so under the new averaging schemes.

Sounds interesting, right? I thought so too, and planned to write a blog post introducing it. But while I was almost done writing the post and cross-checking the data, I discovered that the numbers in the paper's tables were a complete mess!!! The actual results don't support the paper's conclusions at all!!! So this post has turned from a "praise session" into a "criticism session"...more

Nonsense

Let's start with the first table from the "miracle paper," showing part of the scores on the GLUE leaderboard:

GLUE leaderboard results from the "miracle paper"

Setting everything else aside, the fact that this paper's tables confuse "," (comma) and "." (decimal point) is already annoying enough (the SuperGLUE table below is even worse). But if that were the only issue, I could let it slide. What's truly unbearable is that the computation rules for AM (arithmetic mean), GM (geometric mean), and HM (harmonic mean) in the table are utterly arbitrary!

After a lot of trial and error, I finally figured out the computation rule behind this table:

1. All AM values are computed using the scores of the first 10 tasks (even though the table above only shows the scores for the first 8 tasks);
2. The GM and HM in the Human row are computed using the scores of the first 10 tasks;
3. The GM and HM for the model rows are computed using the scores of all 11 tasks.

Since the score on the 11th task is lower than on the other tasks, computing things this way makes the models' GM and HM lower than Human's, and the author directly concludes that under GM and HM, humans still rank first. In fact, if everyone's scores were computed using the same set of tasks, the rankings under AM, GM, and HM would be essentially identical. Moreover, anyone with a modicum of mathematical common sense can see the problem with this result: models far outperform humans on many tasks, and only fall slightly short of humans on a few tasks—so under any reasonable averaging scheme, there's no way you'd conclude that humans vastly outperform the models. Yet the author apparently believed exactly that...

The same kind of error shows up in the SuperGLUE table as well:

SuperGLUE leaderboard results from the "miracle paper"

Its computation rule is:

1. All AM values are computed using the scores of the first 8 tasks;
2. All GM and HM values are computed using the scores of all 10 tasks.

In fact, if AM were also computed using all 10 tasks' scores, humans would rank first under AM as well. In other words, as long as everyone uses the same computation standard, the rankings under AM, GM, and HM don't differ much.

Truly at a loss

By the way, this paper was even accepted into a NeurIPS 2021 workshop. Workshop papers are usually nowhere near as rigorous as full conference papers, but I wouldn't have expected things to be this sloppy. Looking again at the paper's title, I wonder if it should be renamed "How not to Lie with this paper"?

It seems that from now on, when reading papers, we not only need to worry about whether the results are reproducible, but also need to double-check whether they've even computed sums, means, and variances correctly. Truly, anything is possible out there.