jgm/djot

Canonical large-ish djot document (and related pandoc issues)

clbarnes opened this issue · 5 comments

clbarnes commented

For benchmarking and testing, it would be helpful to have a large document covering a good spread of djot's features. I tried to convert the pandoc manual into djot for this purpose, using both djot.js and djot.lua:

First, getting the manual as pandoc AST:

wget https://pandoc.org/demo/MANUAL.txt -O - | pandoc -f markdown -t json > manual.json

djot.js, as documented here https://github.com/jgm/djot.js/tree/main#installing-the-command-line-utility

$ djot -f pandoc -t djot manual.json > manual.js.dj

Error: TypeError: Cannot read properties of undefined (reading 'length')
TypeError: Cannot read properties of undefined (reading 'length')
    at fromPandocAttr (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:399:18)
    at PandocParser.fromPandocBlock (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:746:30)
    at /home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:855:25
    at Array.map (<anonymous>)
    at PandocParser.fromPandoc (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:854:34)
    at fromPandoc (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/pandoc.js:868:38)
    at Object.<anonymous> (/home/barnesc/code/djot-bench/node_modules/@djot/djot/lib/cli.js:175:43)
    at Module._compile (node:internal/modules/cjs/loader:1275:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1329:10)
    at Module.load (node:internal/modules/cjs/loader:1133:32)%

djot.lua as documented here https://github.com/jgm/djot.lua#readme

$ pandoc -f json -t djot-writer.lua > manual.lua.dj

pandoc: PandocLuaException "djot-writer.lua:8: attempt to index a nil value (local 'layout')\nstack traceback:\n\tdjot-writer.lua:8: in main chunk"

I'm not sure what the point of failure is here:

  • pandoc's markdown-formatted manual may not make clean pandoc ast
  • djot.lua may not be able to convert pandoc ast into djot
  • djot.js may not be able to convert pandoc ast into djot
  • PEBKAC (always a threat)
jgm commented

This is working fine for me:

curl  https://pandoc.org/demo/MANUAL.txt | pandoc -f markdown -t json | ~/.npm-packages/bin/djot -f pandoc -t djot

Have you installed the latest djot.js using npm install -g @djot/djot?
Are you sure you're running that version and not another one?

clbarnes commented

I have just updated djot.js with the same results, although as a local install rather than global (npm i @djot/djot). I have tried accessing the binary using npx djot and node_modules/.bin/djot.

jgm commented

Since I can't reproduce it, you may have to track down the issue yourself. I'd suggest putting some tracing code in
node_modules/@djot/djot/lib/pandoc.js:399:18
If we could see the value of pattr there it might give us a clue.
More ambitiously you could add something that prints out block.t in fromPandocBlock, so we can keep track of which block is being processed and localize the issue.

clbarnes commented

Regardless of my possibly platform-specific issue, would it be possible to publish that generated djot file (or another one of significant size and feature-breadth) for test/benchmark purposes?

bdarcus commented

... would it be possible to publish that generated djot file (or another one of significant size and feature-breadth) for test/benchmark purposes?

@clbarnes - I'm attaching it here.

pandoc-manual.txt

  • ❤️1