Nearly Perfectly Resolving the Conflict Between MathJax and Marked

近乎完美地解决MathJax与Marked的冲突 苏剑林 · 2024-08-26 · Questions website latex paper cool paper

In 《Making MathJax Play Nicer with Google Translate and Lazy Loading》 we mentioned that Cool Papers had added MathJax for parsing LaTeX formulas. Little did I expect this would trigger a whole host of compatibility issues. While some of these issues are admittedly just my own perfectionism talking, a solution that's as close to perfect as possible is nonetheless a pleasing thing, so I'm still willing to spend some effort on it.

In the previous post we already resolved the compatibility between MathJax and Google Translate/lazy loading; in this post we'll tackle the conflict between MathJax and Marked.

A Brief Description of the Problem

Markdown is a lightweight markup language that lets people write documents in an easy-to-read, easy-to-write plain-text format—arguably one of the most popular writing syntaxes around today. The [Kimi] feature in Cool Papers also outputs its content basically following Markdown syntax. However, Markdown isn't a language directly consumed by browsers; the language browsers understand is HTML. So before content can be shown to the user, there needs to be a Markdown-to-HTML conversion step (rendering). more

There are two ways to convert Markdown to HTML: one is to have a server on the backend convert the Markdown to HTML before sending it to the user; the other is for the user's browser to receive the Markdown and convert it to HTML on the frontend. The example in this post is mainly about the latter case, though in principle the same idea should be adaptable to the former with minor modifications. There are many libraries for converting Markdown on the frontend; Cool Papers uses Marked, which is a relatively lightweight choice.

The way Marked renders Markdown is simple: you just call marked.parse directly on the string. If we then pair this with the MathJax introduced in the previous section, we can get LaTeX code parsed as well. However, Markdown and LaTeX have some overlapping syntax, so Marked might end up converting the LaTeX code (if any) according to Markdown rules first, which means the subsequent MathJax pass can no longer get hold of the original LaTeX code, and rendering fails.

Here's a reproducible piece of code:

<div id="content"></div>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.9/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
    var div = document.getElementById('content');
    div.innerHTML = marked.parse('**cannot** render: \\(a^2 + b^2\\), **can** render: \\\\(c^2 + d^2\\\\)');
    MathJax.Hub.Typeset(div);
</script>

Existing Solutions

It's worth noting that the popular blogging framework Hexo also uses Marked by default to render Markdown, so if you search for "MathJax Marked conflict," you'll find quite a bit of material, most of it written with Hexo as the backdrop. One post that summarizes things fairly thoroughly is 《Taming Hexo [2] — The Conflict Between Hexo and MathJax and How to Fix It》, which sums up the approaches into the following four categories:

1. Manual escaping: that is, instead of writing the correct LaTeX code when composing formulas, you write LaTeX code that "becomes correct only after being rendered by Marked." For example, if the original LaTeX code is a double backslash \\, after passing through Marked it becomes a single backslash \; so you might as well write four backslashes \\\\ from the start, which becomes \\ after Marked processes it.
2. Protecting formulas: this idea is even simpler—taking advantage of the fact that Marked won't render code, you wrap the formulas in code-block markers to protect them, then after Marked has rendered everything, you extract the formulas and parse them with MathJax, as in 《Resolving the Conflict Between MathJax and Markdown》. The problem with this approach is that it's fairly easy to confuse formulas with genuine code blocks.
3. Switching engines: switch to a rendering engine that better supports mixing Markdown and LaTeX, such as Pandoc, which is commonly recommended under Hexo, as in 《Resolving the Conflict Between Hexo and MathJax》. But Pandoc is a backend rendering engine, and for frontend rendering I haven't found a better alternative.
4. Modifying the engine: this means modifying Marked's source code so that it doesn't render certain LaTeX code, thereby resolving the issue to some extent, as in 《The Coexistence Problem Between Marked.js and MathJax in Hexo》. This requires us to work out a set of rules for patterns that Marked is prone to mis-rendering and handle them one by one.

Approaches 1 and 2 require manually modifying the formula code, but the formulas in Cool Papers are generated by Kimi and can't be modified, so these are basically ruled out. Since I also haven't found a better frontend Markdown rendering engine, approach 3 is likewise ruled out. Approach 4, although it can solve the problem to some degree, is too rule-based and inelegant, and it can only ever be a matter of "treating the head when the head hurts, treating the foot when the foot hurts"—there's no way to know whether some edge case has been missed.

A Reversed Approach

In fact, there's a remarkably simple solution to this problem. Fundamentally, this is a syntax conflict caused by running Marked first and then MathJax—so why not do it the other way around: render the formulas with MathJax first, and only then render the Markdown with Marked? Since MathJax can identify mathematical formulas fairly strictly, and its rendering output almost never contains anything that looks like Markdown syntax, doing MathJax first and Marked second resolves the conflict at its root.

Here's some reference code:

<div id="content"></div>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.9/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
    var div = document.getElementById('content');
    div.innerHTML = '**can** render: \\(a^2 + b^2\\)';
    MathJax.Hub.Queue(
        ['Typeset', MathJax.Hub, div],
        function() {
            div.innerHTML = marked.parse(div.innerHTML);
        }
    );
</script>

The Perfectionist's Affliction

The final display effect of the code above is already what we wanted, but for readers with a bit of perfectionism it still falls a little short—there are two minor flaws.

The first flaw is that it briefly displays the raw Markdown text before the final rendered result appears a moment later (depending on rendering speed). Since the raw Markdown is dumped straight into the browser, it looks almost like garbled text—meaning the user first sees an almost-garbled page, and only after a short delay does the proper page appear. This hurts the reading experience. To fix this, we can instead create a separate element to do the rendering in, and only assign it to the current page once rendering is complete:

<div id="content"></div>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.9/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
    var div = document.getElementById('content');
    var div2 = document.createElement('div');
    div2.innerHTML = '**can** render: \\(a^2 + b^2\\)';
    MathJax.Hub.Queue(
        ['Typeset', MathJax.Hub, div2],
        function() {
            div.innerHTML = marked.parse(div2.innerHTML);
        }
    );
</script>

This way, what the user sees directly is the fully rendered result, with no garbled-looking transitional content in between. The second flaw is that we'll notice right-clicking on a formula no longer brings up the MathJax menu shown below:

Normally, right-clicking a formula should show the MathJax menu

Understanding this requires a bit of insight into how custom right-click menus work. In short, a custom context menu needs an event listener bound to the element, but once we overwrite the element's innerHTML, that event listener stops working. I mulled over this problem for quite a while, and eventually stumbled upon the discovery that when we run the MathJax.Hub.Typeset command again, MathJax will automatically re-render the formulas. So all we need to do, building on the code above, is first remove the existing formulas and then re-render them:

<div id="content"></div>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.9/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
    var div = document.getElementById('content');
    var div2 = document.createElement('div');
    div2.innerHTML = '**can** render: \\(a^2 + b^2\\)';
    MathJax.Hub.Queue(
        ['Typeset', MathJax.Hub, div2],
        function() {
            div.innerHTML = marked.parse(div2.innerHTML);
            div.querySelectorAll('.MathJax').forEach(e => e.remove());
            MathJax.Hub.Typeset(div);
        }
    );
</script>

This restores the context menu. But that's not quite the end of the story—digging a little further into the underlying mechanism, I found that after the first Typeset call, the original formula source is stored in a script tag, so subsequent formula removal can just call Typeset repeatedly to re-render. However, I discovered that Marked actually goes and renders the contents of script! To keep Marked from modifying the formula, we can save the original formula source before marked.parse and restore it after marked.parse:

<div id="content"></div>
<script src="https://cdn.jsdelivr.net/npm/marked/marked.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@2.7.9/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
    function parseMarkdown(text) {
        var scripts = text.match(/<script[^>]*>([\s\S]*?)<\/script>/gi);
        text = marked.parse(text);
        return text.replace(/<script[^>]*>([\s\S]*?)<\/script>/gi, m => scripts.shift());
    }
    var div = document.getElementById('content');
    var div2 = document.createElement('div');
    div2.innerHTML = '**can** render: \\(J\'_\\theta = J_\\theta\\)';
    MathJax.Hub.Queue(
        ['Typeset', MathJax.Hub, div2],
        function() {
            div.innerHTML = parseMarkdown(div2.innerHTML);
            div.querySelectorAll('.MathJax').forEach(e => e.remove());
            MathJax.Hub.Typeset(div);
        }
    );
</script>