Images also need pandoc
Closed this issue · 38 comments
CC @jeroenooms
Pandoc is in the SystemRequirements of knitr:
https://github.com/cran/knitr/blob/e3adb6f1d7b1ed4ae9c016c015c88e5a237f1d3f/DESCRIPTION#L90-L93
We already have an entry for it in sysreqs DB:
https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json
Maybe Suggests is not used for finding system requirements?
But from the log is looks pandoc, knitr and rmarkdown are all getting installed, and it still gives the warning.
@jeroenooms Oh, indeed.
The build fails due to an xvfb problem by the way, not this warning.
From https://github.com/yihui/knitr/blob/9394e12b764c9729368200a04dae770c098ab808/R/utils-vignettes.R#L8-L24 it needs pandoc-citeproc as well, which doesn't look to be installed on R-hub. https://builder.r-hub.io/status/magick_0.3.tar.gz-12769ec3c1339f46527af5bad4b3f1ca#L174 is the only mention in your build log.
I think knitr should explicitly specify that it needs pandoc-citeproc then. Anyway, I'll put in an exception for it in a minute....
Or do you think we should always install pandoc-citeproc, whenever we install pandoc? Maybe that's the best solution....
@gaborcsardi I think you are probably correct, pandoc-citeproc should probably be in SystemRequirements: of knitr or rmarkdown (or both). These are really more like SuggestedSystemRequirements of course, but there isn't a defined field for optional system dependencies...
OK, I think we'll just install pandoc-citeproc (it is quite small) whenever pandoc is wanted.... but I need to update the sysreqs web app for that, because apparently it does not handle lists of packages....
Crossposting here: yihui/knitr#1286
@jeroenooms wait, so the missing pandoc-citeproc is not the problem for us? I thought it was.
I think in this case it is, but I am seeing this problem locally as well sometimes (and I have both...)
Did you make pandoc-citeproc part of the default image? Then I'll try again if it now works...
Not the default image, but I'll just install pandoc-citeproc, whenever
pandoc is requested.
But this is not done yet, something in sysreqs does not handle package
lists, so I need to fix that first. I'll comment here, once it is done.
On Tue, Sep 6, 2016 at 4:42 PM, Jeroen Ooms notifications@github.com
wrote:
Did you make pandoc-citeproc part of the default image? Then I'll try
again if it now works...—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAoTQLXMxZxtjdifRkLv3P8KskTEJIFUks5qnYn9gaJpZM4J1jQe
.
It should be OK now, pandoc and pandoc-citeproc go together.
Btw. these links are not very good on GitHub, because they are removed after 3 days, I mean r-hub removed the builds.... just FYI....
Did you fix this yet? I'm still getting these warnings: https://builder.r-hub.io/status/curl_1.9000.tar.gz-9a07cdd4f9607723c5e9b8f135db8376
I did, but apparently EPEL does not have a pandoc-citeproc package. Or is it called differently? Then we need to update this entry: https://github.com/r-hub/sysreqsdb/blob/master/sysreqs/pandoc.json
Hmm you're right, it's in EPEL 7 but not EPEL 6: https://apps.fedoraproject.org/packages/pandoc-citeproc
If it is available in some other repo, backported or sg., then I can add that repo. Or we need to compile it? Or download a binary from somewhere?
Meh I think you'd have to install from source. Or we could host backports somewhere on a gh-pages branch or so....
It's not clear to me why we need citeproc in the first place for vignettes without citations. @yihui
Just In case your vignettes have bibliography.
Travis simply installs Pandoc from the .deb release, which includes pandoc-citeproc https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/r.rb#L473
Yeah but centos 6 doesn't have that...
You can (pre)build both pandoc and pandoc-citeproc as standalone binaries.
I can build an rpm for it, and put it in a repo I suppose. No need to put these on GH pages, they will be fine on r-hub.
@gaborcsardi did you end up creating a pandoc-citeproc rpm for CentOS6? I would like a copy of that :D
It is not just that. pandoc in centos6 is very old, that needs to be updated, too. So I am working out a way to do this. :/
Also, wch/r-source@74c8486
I'll install it from the RStudio binaries, and via a custom install specification in sysreqs (to be added).
So no RPM, sorry.....
OK, so I am downloading and installing pandoc from the RStudio-built binaries, and still have issues. :/
Fedora:
/usr/local/bin/pandoc: error while loading shared libraries: libgmp.so.3: cannot open shared object file: No such file or directory
Error in system(paste(shQuote(pandoc_path), "--version"), intern = TRUE) :
error in running command
Ubuntu:
/usr/local/bin/pandoc +RTS -K512m -RTS file8c0380f6b8b.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output file8c0380f6b8b.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/RtmplFNlOk/rmarkdown-str8c0440fe35a.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1
pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css
HttpExceptionRequest Request {
host = "maxcdn.bootstrapcdn.com"
port = 443
secure = True
requestHeaders = []
path = "/font-awesome/4.6.3/css/font-awesome.min.css"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(ConnectionFailure getProtocolByName: does not exist (no such protocol name: tcp))
Error: pandoc document conversion failed with error 67
Execution halted
CentOS:
/usr/local/bin/pandoc +RTS -K512m -RTS filec3836351008.utf8.md --to html --from markdown+autolink_bare_uris+ascii_identifiers+tex_math_single_backslash --output filec3836351008.html --smart --email-obfuscation none --self-contained --standalone --section-divs --template /home/docker/R/rmarkdown/rmd/h/default.html --variable 'theme:bootstrap' --include-in-header /tmp/Rtmpj3riqi/rmarkdown-strc38f5af397.html --mathjax --variable 'mathjax-url:https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML' --no-highlight --variable highlightjs=/home/docker/R/rmarkdown/rmd/h/highlight --variable navigationjs=/home/docker/R/rmarkdown/rmd/h/navigation-1.1
pandoc: Could not fetch https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3
StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [("Server","nginx"),("Date","Mon, 24 Oct 2016 22:27:46 GMT"),("Content-Type","application/xml"),("Transfer-Encoding","chunked"),("Connection","keep-alive"),("Content-Encoding","gzip"),("X-Cache","MISS"),("X-Response-Body-Start","<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>57327A02ACC77868</RequestId><HostId>D59raUW8YitWFC2ovDF9FJupw95xOZcZyH4ETWQh4bN38P7XI/XTV+bICHKnQ77nlbgsyvMQ8qU=</HostId></Error>"),("X-Request-URL","GET https://maxcdn.bootstrapcdn.com:443/font-awesome/4.6.3/css/../fonts/fontawesome-webfont.eot?v=4.6.3")] (CJ {expose = []})
Error: pandoc document conversion failed with error 67
Execution halted
Debian is OK.
@gaborcsardi Can you provide the Rmd file and your rmarkdown version? AFAIK we don't use any https resources in rmarkdown, so I don't know why Pandoc was trying to download resources from bootstrapcdn.com.
@yihui It was the rchallenge package, as on CRAN. Yes, it is not you, it is an extension.
Btw. it needs libgmp and libffi on a stock Fedora 24 Docker container, it seems that the RPM versions assume old libs.
[root@32c294495d65 ~]# ldd /usr/local/bin/pandoc
linux-vdso.so.1 (0x00007ffca0fde000)
libz.so.1 => /lib64/libz.so.1 (0x00007fa37ec43000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa37ea3b000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fa37e837000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa37e633000)
libgmp.so.3 => not found
libffi.so.5 => not found
libm.so.6 => /lib64/libm.so.6 (0x00007fa37e328000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa37df65000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa37dd48000)
/lib64/ld-linux-x86-64.so.2 (0x000055d5b7335000)
Okay, I'd recommend rchallenge either to localize font-awesome, or not to use the self-contained mode. It is a waste of time and resources to download a copy of font-awesome every time a document is compiled.
I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible.
@jeroenooms pandoc RPMs for centos6: https://files.r-hub.io/pandoc/centos6/
Very minimal and much sweet.
@jeroenooms And if you want to see how they are built, or tweak: https://github.com/r-hub/pandoc-builders
I don't know how RStudio's pandoc was built, but I wonder if you have tried to build the binary by yourself. You only need to build it once, and the self-contained binary can be stored elsewhere (S3, Github releases, ...) and used later. I guess it is not that terrible.
Yes, this would not be terrible, but unfortunately it is not true. It is hard to build a static binary, and the shared binary will depend on the system libs (libgmp in particular), so it won't be portable. E.g. RStudio's latest RPM binary does not work on Fedora 24, and the CentOS6 binary I built does not, either.
So in practice, I'll need to build at least a couple of binaries. Anyway, I have an "easy" way of building them now, at least.
@jeroenooms Updated (reverted) sysreqs, and updated centos6 images to install from our own repository:
rhub-linux-builders/centos6-epel/Dockerfile
Lines 9 to 11 in d09595f
It all works fine now. And you get the latest pandoc and pandoc-citeproc, with HTTPS support.