Non-English characters overlaps and Misplaced \hline

Question

Non-English characters overlaps and Misplaced \hline

fivecakes opened this issue 4 years ago · 13 comments

MathJax version 2	MathJax version 3	bug description
		Version 2 is normal, and the text in version 3 overlaps.This situation occurs in non-English characters.Chinese characters are used in the example
		Version 2 can be rendered normally, and version 3 give an error of `Misplaced \hline`

三生三世十里桃花 \approx 10^{10}秒

\begin{array} {|c|c|c|} \hline
\text{11} & \text{12} & \text{13} \\
\hline 
\text{21} & \text{22} & \text{23} \\
\hline 
\text{31} & \text{32} & \text{33} \\
\hline  
\end{array}

Answer 1 · 2021-01-10T16:00:56.000Z

Your images for MathJax 3 are missing. Can you add them in again?

Also, can you say (in words) what the issue is? The MathJax-node images look OK to me, so I'm not sure what you are considering to be "bugs".

Answer 2 · 2021-01-11T01:02:18.000Z

@dpvc not missing,it's indeed black, I added the description and added a sample link

Answer 3 · 2021-01-14T15:54:59.000Z

not missing,it's indeed black

It showed the "missing image" icon for me originally, but now that is not the case. Perhaps the image hadn't been fully processed when I looked. Now I see the black rectangle, as you indicate.

Questions:

Are your version 2 images from mathjax-node on a server, or from within a browser?
Which demo program are you using to create your version 3 images? (e.g., direct/tex2svg, component/tex2svg-page, etc.)
Are you using any command-line options on the program you are calling?

I understand the issue for the first expression. MathJax needs font metric information about the characters that it displays. When a character is not in its fonts (like the Chinese characters you are using), MathJax doesn't know how big they are. In the browser, it can measure the width, but on the server, that is not possible, so it has to guess. As I recall mathjax-node made special arrangements to handle the width Chinese characters better. I will look into what needs to be done for MathJax 3 to do the same.

I will check into the the black image once I know how you are creating it.

Answer 4 · 2021-01-15T02:45:17.000Z

yes, version 2 images is from mathjax-node on a server.

I use command node -r esm index.js to create version 3 images server

index.js is modified from simple/tex2svg in order to access the content of the server through http, here is index.js code:

var express = require('express');
var app = express();

const PACKAGES = 'base, autoload, require, ams, newcommand';
var argv = {
  em: 16,
  ex: 8,
  width: 1280,
  packages: 'base, autoload, require, ams, newcommand',
  fontCache: true,
  'font-cache': true,
  assistiveMml: false,
  'assistive-mml': false,
  dist: false,
  '$0': 'tex2svg' 
}

require('mathjax-full').init({
    options: {
        enableAssistiveMml: argv.assistiveMml
    },
    loader: {
        source: (argv.dist ? {} : require('mathjax-full/components/src/source.js').source),
        load: ['adaptors/liteDOM', 'tex-svg']
    },
    tex: {
        packages: argv.packages.replace('\*', PACKAGES).split(/\s*,\s*/)
    },
    svg: {
        fontCache: (argv.fontCache ? 'local' : 'none')
    },
    startup: {
        typeset: false
    }
}).then((MathJax) => {
    app.get('/', function (req, res) {
        var yourMath = req.query.latex;

        MathJax.tex2svgPromise(yourMath || '', {
            display: !argv.inline,
            em: argv.em,
            ex: argv.ex,
            containerWidth: argv.width
        }).then((node) => {
            res.setHeader('Content-Type', 'image/svg+xml');
            res.setHeader('Cache-Control', 'max-age=31536000');
            res.status(200);
            
            const adaptor = MathJax.startup.adaptor;
            res.send(adaptor.outerHTML(node.children[0]));
        });
    });
}).catch(err => console.log(err));

app.listen(6788, function () {
  console.log('app listening on port 6788!');
});

afternode -r esm index.js, request the url with a parameter of latex. In addition,passing latex via url requires url-encoding.

http://localhost:6788/?latex=%0A%5Cbegin%7Barray%7D%20%7B%7Cc%7Cc%7Cc%7C%7D%0A%5Chline%0A%5Ctext%7B11%7D%20%26%20%5Ctext%7B12%7D%20%26%20%5Ctext%7B13%7D%20%5C%5C%0A%5Chline%20%0A%5Ctext%7B21%7D%20%26%20%5Ctext%7B22%7D%20%26%20%5Ctext%7B23%7D%20%5C%5C%0A%5Chline%20%0A%5Ctext%7B31%7D%20%26%20%5Ctext%7B32%7D%20%26%20%5Ctext%7B33%7D%20%5C%5C%0A%5Chline%20%20%0A%5Cend%7Barray%7D%0A

Answer 5 · 2021-01-15T13:43:52.000Z

Thanks for the additional information. Are you using the CSS that the SVG output generates in addition to the SVG element itself? That has some important settings, including some control over the lines for arrays. I suspect that you have not included that, which would account for the black rectangle (without the CSS, the outer frame for the array would end up being filled with the default color, which is black).

Answer 6 · 2021-01-16T04:48:00.000Z

After your reminder, I located the error

Try this command

node -r esm simple/tex2svg "\begin{array} {|c|c|c|}\hline \text{11} & \text{12} & \text{13} \\ \hline \text{21} & \text{22} & \text{23} \\  \text{31} & \text{32} &\text{33} \\ \hline  \end{array}"

it's give an error Misplaced \hline

Answer 7 · 2021-01-16T18:10:49.000Z

Version 2 only return SVG element, so it can work.

Version 2 also used CSS, but version 3 uses it in more places, so you probably never ran into the situations where it was needed in v2.

Now version 3 requires css, img cannot load a data with css, do you have any ideas?

Well, you are allowed to include <style> elements within an SVG image, so you could insert the styles needed into the SVG itself. I have distilled down the declarations that you may need, and suggest that you include

const CSS = [
  'a{fill:blue;stroke:blue}',
  '[data-mml-node="merror"]>g{fill:red;stroke:red}',
  '[data-mml-node="merror"]>rect[data-background]{fill:yellow;stroke:none}',
  '[data-frame],[data-line]{stroke-width:70px;fill:none}',
  '.mjx-dashed{stroke-dasharray:140}',
  '.mjx-dotted{stroke-linecap:round;stroke-dasharray:0,140}',
  'path{stroke-width:3px}'
].join('');

after defining PACKAGES, and then replace

            res.send(adaptor.outerHTML(node.children[0]));

with

            res.send(adaptor.innerHTML(node).replace(/<defs>/, '<defs><style>' + CSS + '</style>'));

This will insert the needed CSS into your SVG image.

I directly use the img tag to load the svg generated by the server, like this:
<img src="mathjax-node.mydomain.con/?latex=x^2">

I would point out, however, that while this will insert the SVG into the page, it will not get the vertical positioning correct when the math includes anything below the baseline. So the results will not work well for in-line math, as the baseline will be out of place.

Answer 8 · 2021-01-16T18:12:42.000Z

Try this command ...

It works fine for me. Perhaps it has to do with the operating system and command shell that you are using. It may be that " ... \\ ... " is used to escape a backslash and that you are only getting ... \ ... in the string that is actually passed to tex2svg. Try doubling those backslashes.

Answer 9 · 2021-01-17T01:28:48.000Z

I will follow your advice on css. And it works when I doubling those backslashes.

Very very thank you for your meticulous advice.

Answer 10 · 2021-01-17T02:44:40.000Z

When I added css according to your suggestion, the black disappeared and it can be displayed normally now.

In the command line, I replaced " with ', it works too. The black rectangle problem is completely solved. Thank you again.

When the overlaps problem is fixed, I will upgrade my project code to version 3

Answer 11 · 2021-01-17T03:23:37.000Z

I would point out, however, that while this will insert the SVG into the page, it will not get the vertical positioning correct when the math includes anything below the baseline. So the results will not work well for in-line math, as the baseline will be out of place.

well , Thank you for your reminder, If you don’t remind me, I haven’t noticed the difference. But using the img tag to load the svg directly is more flexible for me .

<img src="mathjax.mydomain.con/?latex=x^2">

In addition , I know that https://www.zhihu.com/ also uses this method to load svg generated by MathJax-node. Zhihu has many users in China, which is equivalent to Quora abroad.

Answer 12 · 2021-01-18T13:04:11.000Z

When the overlaps problem is fixed, I will upgrade my project code to version 3

Replace the startup block of your configuration with the following in order to resolve that issue. This allows a difference width to be used for full-width characters (both the CJK width and normal width are configurable). It also sets the monospace font for unknown characters, since we are assuming a fixed width for those characters.

    startup: {
        typeset: false,
        ready() {
          //
          //  Add support for wide characters to the LiteAdaptor
          //
          const {LiteAdaptor} = MathJax._.adaptors.liteAdaptor;
          //
          //  The width for CJK and regular characters
          //
          LiteAdaptor.OPTIONS.cjkCharWidth = 1;
          LiteAdaptor.OPTIONS.unknownCharWidth = .6;
          LiteAdaptor.OPTIONS.unknownCharHeight = .8;
          //
          //  Character ranges that are CJK
          //
          LiteAdaptor.cjkChars = new RegExp(['[',
            '\u1100-\u115F', // Hangul Jamo
            '\u2329\u232A',  // LEFT-POINTING ANGLE BRACKET, RIGHT-POINTING ANGLE BRACKET
            '\u2E80-\u303E', // CJK Radicals Supplement ... CJK Symbols and Punctuation
            '\u3040-\u3247', // Hiragana ... Enclosed CJK Letters and Months
            '\u3250-\u4DBF', // Enclosed CJK Letters and Months ... CJK Unified Ideographs Extension A
            '\u4E00-\uA4C6', // CJK Unified Ideographs ... Yi Radicals
            '\uA960-\uA97C', // Hangul Jamo Extended-A
            '\uAC00-\uD7A3', // Hangul Syllables
            '\uF900-\uFAFF', // CJK Compatibility Ideographs
            '\uFE10-\uFE19', // Vertical Forms
            '\uFE30-\uFE6B', // CJK Compatibility Forms ... Small Form Variants
            '\uFF01-\uFF60\uFFE0-\uFFE6', // Halfwidth and Fullwidth Forms
            '\u1B000-\u1B001', // Kana Supplement
            '\u1F200-\u1F251', // Enclosed Ideographic Supplement
            '\u20000-\u3FFFD', // CJK Unified Ideographs Extension B ... Tertiary Ideographic Plane
          ']'].join(''), 'g');
          //
          //  Override the nodeSize() method to take wide characters into account
          //
          LiteAdaptor.prototype.nodeSize = function (node, em, local = null) {
            const text = this.textContent(node);
            const non = text.replace(LiteAdaptor.cjkChars, '').split('').length;
            const CJK = text.split('').length - non;
            return [
              CJK * this.options.cjkCharWidth + non * this.options.unknownCharWidth,
              this.options.unknownCharHeight
            ];
          };
          //
          //  Use monospaced font for unknown characters (since we are using fixed width values above);
          //
          const {TeXFont} = MathJax._.output.svg.fonts.tex_ts;
          const unknown = TeXFont.defaultCssFonts;
          Object.keys(unknown).forEach(font => {if (unknown[font][0] === 'serif') unknown[font][0] = 'monospace'});
          //
          //  Do the normal ready function
          //
          MathJax.startup.defaultReady();
        }

I will be making a pull request to include this feature in the LiteAdaptor, so this will be done automatically in the future.

Answer 13 · 2021-05-02T23:55:47.000Z

This has been included in version 3.1.4, so I am closing the issue.