Running Nightmare headlessly on Linux

Question

Running Nightmare headlessly on Linux

richard5mith opened this issue 9 years ago · 113 comments

I've taken the cnn.js example, and my only change is changing the require line from ../nightmare to nightmare, but when I run it I only get...

DEBUG=nightmare node --harmony cnn.js

nightmare queueing action "goto" +0ms    
nightmare queueing action "evaluate" +5ms

I don't ever seem to get the title of the page.

I've tried various other examples from Nightmare < 2 from around the web, and I can't get any of them to work either. Nothing ever errors, I just never seem to get any screenshot, pdf or elements pulled off the page in the output.

After 4 hours of bashing my head, I'm not sure what else to try.

jokeyrhyme commented 8 years ago

FYI:

Answer 1 · 2015-09-17T06:49:10.000Z

I tried this and got it returned properly.

nightmare queueing action "goto" +0ms
nightmare queueing action "evaluate" +1ms
Breaking News, U.S., World, Weather, Entertainment & Video News - CNN.com

But if I tried with a website that have a SLL problem, like below, it hang there for, forever.

    vo(run)(function(err, result) {
      if (err) throw err;
    });

    function *run() {
        var title = yield nightmare
        .goto('https://cas2.northport.com.my/myapp/wa/r/nmbctr')
        .evaluate(function() {
            return document.title;
        });
        console.log(title);
        yield nightmare.end();
    }

It just hang here. No error, and electron no exited.

    nightmare queueing action "goto" +0ms
    nightmare queueing action "evaluate" +1ms

Answer 2 · 2015-09-17T12:59:37.000Z

What's your environment for running the cnn.js script?

Entirely headless server, or in a terminal in an existing X session, or OS X etc?

Answer 3 · 2015-09-17T23:03:02.000Z

I do have the same issue on a headless server. Everything runs correctly on OSX but not on my build server.

Answer 4 · 2015-09-18T02:47:16.000Z

Yikes, major issue.

Seems like the electron linux binding is broken or something. Anyone have any ideas?

Answer 5 · 2015-09-18T21:02:18.000Z

Odd that at least you get an error message when I don't. Mine just fails silently.

My guess is that it can't get the framebuffer, which is where xvfb would normally come in. But I've tried running electron with that too and just get gtk errors.

Answer 6 · 2015-09-18T21:39:37.000Z

Okay. I'm not exactly sure how to codify this right now (or what layer to support this in), but to get it working on the official node (docker) image, you need to do the following:

# Install dependencies
apt-get update &&\
    apt-get install -y libgtk2.0-0 libgconf-2-4 \
    libasound2 libxtst6 libxss1 libnss3 xvfb
npm install segmentio/nightmare

# Start Xvfb
Xvfb -ac -screen scrn 1280x2000x24 :9.0 &
export DISPLAY=:9.0

# Test it
apt-get install vim
vim index.js
# <paste in example>
node index.js

More info: electron/electron#228
Dockerfile: https://github.com/aheuermann/docker-electron

Could use some help on getting this bug resolved.

Answer 7 · 2015-09-18T22:11:07.000Z

That doesn't solve it for me on my environment. Still no errors, but still no title from the CNN example either.

I had previously got Gtk3::Webkit Perl module running, which also uses xvfb, and had everything installed except libxss1 already. So I know my xvfb works.

Going round in circles trying to find suggestions. I see the same as this person when running electron through xvfb-run, https://gist.github.com/AspireToCodeBetter/130877925f52c8fb2557.

And the Stack Overflow question points people here, http://stackoverflow.com/questions/32612868/running-electron-atom-shell-headlessly-on-linux-server-through-nightmare-js.

Answer 8 · 2015-09-18T22:23:53.000Z

Oh I don't think this is it, but can you try running master? npm install segmentio/nightmare ?

I upgraded it to fix: electron-userland/electron-prebuilt#54 (comment)

Answer 9 · 2015-09-18T22:34:11.000Z

Hey! That did it. I can now start Xvfb on display 9 as you posted before, then I can now run the cnn example.

Looks like it was an incomplete Electron install before.

So can you modify your launch of electron process to use xvfb-run?

Answer 10 · 2015-09-19T00:15:35.000Z

@richard5mith i'm not very familiar with xvfb, by xvfb-run do you mean the instructions i provided? or is there a simpler way?

I think it's starting a server (Xvfb -ac -screen scrn 1280x2000x24 :9.0 &), which is something I'd like to avoid. or at least have a good way to clean up after we quit.

Answer 11 · 2015-09-19T07:59:21.000Z

I was thinking something like this as a variation on your steps, which is how I had run Gtk Webkit .

xvfb-run --server-args="-screen 9 1280x2000x24" ./electron

That does seem to launch correctly and the renderer process starts. So I'm not sure if you can do something similar from your JS and then do the IPC with that process.

Answer 12 · 2015-09-19T08:16:04.000Z

Actually, this works, without having Xvfb running in the background first as per your previous steps...

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

Answer 13 · 2015-09-19T23:30:37.000Z

@richard5mith thanks for that, runs for me now on my build server!

Answer 14 · 2015-09-20T01:11:35.000Z

Nice! Now just need to figure out the best way to get these dependencies on linux boxes (without using custom buildpacks or anything)

Answer 15 · 2015-09-20T12:55:51.000Z

@matthewmueller that might help:
https://github.com/shippableImages/minBase/blob/master/ubu1204/01base/containerSetup.sh
https://github.com/shippableImages/minBase/blob/master/ubu1204/01base/xvfb.sh

Using the shippable container to run nightmare and mocha.

Answer 16 · 2015-09-21T05:56:47.000Z

@matthewmueller I've narrowed it down to apt-get update && apt-get install -y xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps clang libdbus-1-dev libgtk2.0-dev libnotify-dev libgnome-keyring-dev libgconf2-dev libasound2-dev libcap-dev libcups2-dev libxtst-dev libxss1 libnss3-dev gcc-multilib g++-multilib

Hope that helps!

Answer 17 · 2015-09-21T06:01:22.000Z

Thanks man, unless I'm missing something I think this should do it though: #224 (comment). Or are you talking about without node dependencies?

I think the remaining items to sort out are:

When do these installation procedures happen? Is this a postinstall script, a node-gyp thing, or is it something that shouldn't even be handled in this library?
Verify that we can run xvfb-run --server-args="-screen 0 1024x768x24" across multiple processes (in other words, running two of these spawn commands at once on the same machine)
Ensure that we're cleaning up the xvfb server after we execute (when the node process exits)

I plan on looking at this sometime this week or weekend, but any time offered sorting this stuff out would help accelerate this fix.

Answer 18 · 2015-09-29T15:57:25.000Z

I'm unable to fix my #223 Heroku deploy. Updated to Nightmare v2.0.6 and tried custom buildpacks but couldn't sort it out. Has anyone used xvfb on Heroku?

Answer 19 · 2015-09-29T19:30:08.000Z

Would be good if we could get some sort of post install script going so it just worked on heroku. Sorry guys, I haven't had much time to look into this lately, but hopefully we can get this sorted out soon

Answer 20 · 2015-09-30T05:34:11.000Z

@matthewmueller https://github.com/Rob--W/node-xvfb might be of use for controlling the xvfb server.

Answer 21 · 2015-09-30T11:53:14.000Z

@NotSentient Jabin, Simply Wall St looks outstanding. Good find on the xvfb module.

Answer 22 · 2015-10-14T10:39:40.000Z

same issue there.
@dylanvalade did you find out how to make it work ?

Answer 23 · 2015-10-14T13:22:50.000Z

@jney Unfortunately, I had to change my plan. Using Phantom now and this explains the setup.
http://stackoverflow.com/questions/22116673/cannot-find-module-phantomjs-in-heroku/32853675#32853675

After this Linux issue gets sorted out I would much rather use Nightmare.

Answer 24 · 2015-10-20T20:47:26.000Z

One thing I just realized is that nightmare is running fine on Circle CI. Probably missing something, but anyone know how that's working? I doubt they have all this additional setup, but maybe...

Answer 25 · 2015-10-25T22:00:55.000Z

This

apt-get update && apt-get install -y xvfb x11-xkb-utils xfonts-100dpi xfonts-75dpi xfonts-scalable xfonts-cyrillic x11-apps clang libdbus-1-dev libgtk2.0-dev libnotify-dev libgnome-keyring-dev libgconf2-dev libasound2-dev libcap-dev libcups2-dev libxtst-dev libxss1 libnss3-dev gcc-multilib g++-multilib

and

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

worked for me, might be worth updating the doc for since this is going to be mostly used on headless servers.

Answer 26 · 2015-10-25T22:05:12.000Z

+1 I also have it running relatively painlessly using 'xvfb-run', after trying and failing to get it connected to a separate 'xvfb' process.

Answer 27 · 2015-11-20T10:48:14.000Z

Did not work for me :(
Have all packages installed, using master from nightmare, headless Ubuntu 14.04.3.
No error messages, just hangs. Works fine on Mac

EDIT: works with sudo, not perfect but it's something

Answer 28 · 2015-11-25T05:04:25.000Z

Not work for me either after tried install xvfb-run

➜  nightmare  DEBUG=nightmare* xvfb-run --server-args="-screen 0 1024x768x24" node --harmony test.js
  nightmare queueing action "goto" for https://www.taobao.com +0ms
  nightmare queueing action "evaluate" +6ms

Answer 29 · 2015-11-26T01:14:22.000Z

sudo yum install xorg-x11-server-Xvfb
xvfb-run --server-args="-screen 0 1024x768x24" node --harmony cnn.js

worked for me on CentOS.

Answer 30 · 2015-12-04T18:47:01.000Z

@deinwort which version of CentOS and NodeJS are you using ?

Answer 31 · 2015-12-04T18:52:24.000Z

On CentOS 7, with xvfb-run is not working :

# xvfb-run --server-args="-screen 0 1024x768x24" node --harmony cnn.js 
  nightmare queueing action "goto" for http://cnn.com +0ms
  nightmare queueing action "evaluate" +2ms

# uname -a
Linux localhost 3.10.0-229.el7.x86_64 #1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

# node -v 
4.2.3

Answer 32 · 2015-12-04T19:29:49.000Z

Xvfb need more packages under Centos otherwise it will crash

yum install xorg-x11-server-Xvfb  gtk2 libXtst GConf2 alsa-lib xorg-x11-fonts*

The best way to debug this, is by running electron directly:

xvfb-run --server-args="-screen 0 1024x768x24 -extension RANDR" ./node_modules/nightmare/node_modules/electron-prebuilt/dist/electron

Answer 33 · 2015-12-10T02:55:03.000Z

Anyone get this working in a Dockerfile? Build the image just fine and seems to be working inside the container (docker run -it nightmare /bin/bash) but once I try passing in the command or entrypoint it just hangs (not at nightmare queueing action "evaluate" +2ms, but like it's starting the xvfb server without running the script)

Here's the Dockerfile:

FROM node:5

# Updating ubuntu packages
RUN apt-get update

# Installing the packages needed to run Nightmare
RUN apt-get install -y \
  xvfb \
  x11-xkb-utils \
  xfonts-100dpi \
  xfonts-75dpi \
  xfonts-scalable \
  xfonts-cyrillic \
  x11-apps \
  clang \
  libdbus-1-dev \
  libgtk2.0-dev \
  libnotify-dev \
  libgnome-keyring-dev \
  libgconf2-dev \
  libasound2-dev \
  libcap-dev \
  libcups2-dev \
  libxtst-dev \
  libxss1 \
  libnss3-dev \
  gcc-multilib \
  g++-multilib

ENV DEBUG="nightmare"

# Add current directory to /app
ADD . /app

# Set current working directory as /app
WORKDIR /app

# Install npm packages
RUN npm install

# Default command. Assumes our file is index.js and our screen size is 1024x768
CMD DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node index.js

Trying to legitimize this a bit for us all, but major strugglez :-P

Answer 34 · 2015-12-10T15:13:56.000Z

For those not familiar with Docker, here is the step by step once you've installed it (it needs a 64 bit modern distro & kernel).

In the nightmare source directory (from this repo), create a file named Dockerfile which has the content at the bottom of this post (@matthewmueller I had to make some edits, including replacing FROM node:5 with FROM node:4.2 for version pinning since 5.x is not LTS) then run:

docker build -t nightmare-test .

This takes a little while. Then:

docker run nightmare-test

expected result is:

Thu, 10 Dec 2015 15:03:07 GMT nightmare queueing action "goto" for http://cnn.com
Thu, 10 Dec 2015 15:03:07 GMT nightmare queueing action "evaluate"
Thu, 10 Dec 2015 15:03:07 GMT nightmare running
Breaking News, Daily News and Videos - CNN.com
Thu, 10 Dec 2015 15:03:25 GMT nightmare running

My Dockerfile is:

FROM node:4.2

# Updating ubuntu packages
RUN apt-get update

# Installing the packages needed to run Nightmare
RUN apt-get install -y \
  xvfb \
  x11-xkb-utils \
  xfonts-100dpi \
  xfonts-75dpi \
  xfonts-scalable \
  xfonts-cyrillic \
  x11-apps \
  clang \
  libdbus-1-dev \
  libgtk2.0-dev \
  libnotify-dev \
  libgnome-keyring-dev \
  libgconf2-dev \
  libasound2-dev \
  libcap-dev \
  libcups2-dev \
  libxtst-dev \
  libxss1 \
  libnss3-dev \
  gcc-multilib \
  g++-multilib

ENV DEBUG="nightmare"

# Add current directory to /app
ADD . /app

# Set current working directory as /app
WORKDIR /app

# Install npm packages
RUN npm install

# Default command. Assumes our file is index.js and our screen size is 1024x768
CMD npm install vo ; xvfb-run --server-args="-screen 0 1024x768x24" node index.js

My index.js for the CNN example (in the root project folder is:

var Nightmare = require('nightmare');
var vo = require('vo');

vo(run)(function(err, result) {
  if (err) throw err;
});

function *run() {
  var nightmare = Nightmare();
  var title = yield nightmare
    .goto('http://cnn.com')
    .evaluate(function() {
      return document.title;
    });
  console.log(title);
  yield nightmare.end();
}

@matthewmueller, if you want me to do a PR with the docker file, an example directory and documentation, let me know.

Answer 35 · 2015-12-10T16:06:41.000Z

@edasque thanks for the detailed instructions! the command i'm just trying to get working is this:

docker run nightmare xvfb-run --server-args="-screen 0 1024x768x24" node index.js

or

docker run nightmare node index.js

with xvfb-run --server-args="-screen 0 1024x768x24" being an ENTRYPOINT

edit: though come to think of it, the dockerfile you provided is deployable to heroku/dokku on it's own. just looking for a couple ways to make this as easy to use as possible.

Answer 36 · 2015-12-10T16:39:24.000Z

index.js being the CNN example, right?

Yes, it's a good idea to use an ENTRYPOINT for that. Let me know what I can help with.

Note however that the ENTRYPOINT will require you to install the application dependencies through nightmare's package.json in this case. Use the exec form of ENTRYPOINT so it could be overriden, maybe?

You could also look at a node-onbuild (such as https://github.com/nodejs/docker-node/blob/master/4.2/onbuild/Dockerfile) for your base image.

Answer 37 · 2015-12-10T17:08:57.000Z

index.js being the CNN example, right?

Yep!

Let me know what I can help with.

If you have a chance, do you mind testing this command? docker run nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js

I was able to get it working by keeping everything in the Dockerfile, but it'd be nice to have flexibility in what script you run. For me when I run that command, it doesn't even run the node script, it just hangs, like it started the xvfb server but didn't do anything after. Tested a ton of iterations, but haven't been able to get a version working where you can pass the file in.

Note however that the ENTRYPOINT will require you to install the application dependencies through nightmare's package.json in this case. Use the exec form of ENTRYPOINT so it could be overriden, maybe?

Yah maybe best to stick with CMD, though you can overwrite ENTRYPOINT via docker run --entrypoint, though i'm not sure that's a great idea. I guess it depends how often you need to change the --server-args

Answer 38 · 2015-12-10T19:09:41.000Z

I have the same problem, I don't know what the cause is yet. I thought it might be in the encoding of the quotes & such but I don't think so anymore. Might be helpful to tail the logs if you know where the xvfb stuff goes (docker exec nightmare tail -f /var/log/whatever_it_is.log)

Answer 39 · 2015-12-22T16:10:31.000Z

After creating an app in in windows(Worked awesome) .. now its not running in fedora 64-bit.. I seriously think this is waste of time for everyone.. why dont you guyzz shut down nightmare, Its really nightmare for developers... All my time wasted, tried to fix for 2 days.... Unftunately its not giving any error messages... tried every steps....Its totally waste

Answer 40 · 2015-12-22T16:12:51.000Z

I have had no problem here. Have you tried the Docker approach @sriducati?

Answer 41 · 2015-12-22T16:15:17.000Z

@edasque thats the only step left bcos iam new to docker... I tried almost all steps.. trust me,each and every dependencies i fixed for electron,scanned whole internet to fix this issue... spent 2-3 days.. but still not working...

Answer 42 · 2015-12-22T16:22:08.000Z

it works using docker. we're working on a better solution for linux. keep in mind this is free software, so please be civil. if you're unhappy with nightmare, you're free to try something else.

Answer 43 · 2015-12-22T16:26:21.000Z

@matthewmueller I am sorry if my comment was offencive .I was very happy with nightmare when working on windows but suddenly all my app stopped working without any error messages... its really very hard...just don't know what to do now...

Answer 44 · 2015-12-22T16:28:26.000Z

@sriducati did you try using DEBUG=nightmare ? I think there might be a way to alter the verbosity but I can't recall.

Answer 45 · 2015-12-22T16:31:02.000Z

@edasque ya i tried debug mode below is the result...

[root@server ~]# DEBUG=nightmare node --harmony cnn.js
nightmare queueing action "goto" for http://cnn.com +0ms
nightmare queueing action "evaluate" +21ms
nightmare running +3ms

Answer 46 · 2015-12-22T16:31:47.000Z

@sriducati nightmare is base on electron which doesn't have a good support on linux speciality with headlessly window. Best way to debug this on linux is to run first electron :
./node_modules/nightmare/node_modules/electron-prebuilt/dist/electron In most case, it's a dep/library missing.
Docker solution may be a the best way for now.

Answer 47 · 2015-12-22T16:33:37.000Z

@younes200 thank you, tried your method as well...result is empty ... again the same prob when we try to run nightmare.. no errors.. no output :(

Answer 48 · 2015-12-24T10:58:13.000Z

Good new everyone.. nightmare worked well with ubuntu 12.04 64-bit

Answer 49 · 2015-12-24T11:11:38.000Z

@sriducati could you please post your results here to help others too ;)

Answer 50 · 2015-12-24T12:49:23.000Z

@matthewmueller , @edasque , @younes200 Thank you all...After changing many operating sys versions... found simple fix Ubuntu 12.04 64-bit ;)

root@server:~# xvfb-run node cnn.js
Breaking News, Daily News and Videos - CNN.com
root@server:~# node -v 
v5.3.0
root@server:~# npm -v
3.3.12
root@server:~#

Its running on OpenVz VPS , No docker used... It would be great if anyone could replay to #411 , I can write a complete help guide to install in VPS ...

Answer 51 · 2016-01-06T04:57:02.000Z

@matthewmueller
Why not put the solution/workaround in README ?

Answer 52 · 2016-01-30T18:27:26.000Z

I'm also having this issue. CentOS workaround didn't work for me.

Mine doesn't hang tho, it just quits.

$ DEBUG=nightmare node cnn.js
  nightmare queueing action "goto" for http://cnn.com +0ms
  nightmare queueing action "evaluate" +5ms
  nightmare running +4ms

I tried with xvfb but still the same issue

$ DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node cnn.js
  nightmare queueing action "goto" for http://cnn.com +0ms
  nightmare queueing action "evaluate" +5ms
  nightmare running +2ms

Answer 53 · 2016-02-05T17:16:54.000Z

It would be good to get a solution to this.
I need to get Nightmare.js running on a remote headless server.

I have a Digital Ocean virtual server running Ubuntu 14.04.3 x64
I have installed Node v5.4.1

Running my app ...
$ sudo DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" node app.js

Gives the following output in Terminal ...

   nightmare queueing action "useragent" +0ms
   nightmare queueing action "viewport" +3ms
   nightmare queueing action "goto" for http://www.bbc.co.uk +4ms
   nightmare queueing action "wait" +1ms
   nightmare queueing action "screenshot" +1ms
   nightmare running +0ms

No screenshot is created, and the final success message (that I get when testing on localhost) is never shown.

Answer 54 · 2016-02-06T16:08:25.000Z

Here, the same with a centOS docker image and node 5.5. :-/

Answer 55 · 2016-02-10T16:41:01.000Z

I worked latest nightmare on headless machine(docker node:4.2, docker-machine with virtualbox driver as host machine).
my dockerfile is

FROM node:4.2

# Updating ubuntu packages
RUN apt-get update

# Installing the packages needed to run Nightmare
RUN apt-get install -y \
  xvfb \
  x11-xkb-utils \
  xfonts-100dpi \
  xfonts-75dpi \
  xfonts-scalable \
  xfonts-cyrillic \
  x11-apps \
  clang \
  libdbus-1-dev \
  libgtk2.0-dev \
  libnotify-dev \
  libgnome-keyring-dev \
  libgconf2-dev \
  libasound2-dev \
  libcap-dev \
  libcups2-dev \
  libxtst-dev \
  libxss1 \
  libnss3-dev \
  gcc-multilib \
  g++-multilib

WORKDIR /opt/test
RUN npm install vo segmentio/nightmare

and execute

Xvfb -ac -screen scrn 1280x2000x24 :99.0 &
export DISPLAY=99.0
node index.js

and index.js content is following.

var Nightmare = require('nightmare');
var vo = require('vo');

vo(function* () {
  var nightmare = Nightmare({ show: true });
  var link = yield nightmare
    .goto('http://yahoo.com')
    .type('input[title="Search"]', 'github nightmare')
    .click('#uh-search-button')
    .wait('.ac-21th')
    .screenshot('screenshot.png')
    .evaluate(function () {
      return document.getElementsByClassName('ac-21th')[0].href;
    });
  yield nightmare.end();
  return link;
})(function (err, result) {
  if (err) return console.log(err);
  console.log(result);
});

but I don't want to write "nightmare": "segmentio/nightmare" to package.json. and my circle-ci test is failed.

Answer 56 · 2016-03-03T06:50:16.000Z

Guys, I couldn't run headless nightmare based on advices from this discussion.
So, any news?
I tryed few docker's, and it always stuck at nightmare running +1ms step..

Answer 57 · 2016-03-03T08:53:28.000Z

Just a note when trying to debug - set environment variables ELECTRON_ENABLE_LOGGING and ELECTRON_ENABLE_STACK_DUMPING true.
https://github.com/atom/electron/blob/master/docs/api/environment-variables.md

Then there should be more output.

Answer 58 · 2016-03-03T09:05:22.000Z

msokk thanks,
but there is no new here:

xvfb-run node test/index.js 
  nightmare queueing action "useragent" +0ms
  nightmare queueing action "goto" for http://google.com +5ms
  nightmare queueing action "type" +4ms
  nightmare queueing action "click" +1ms
  nightmare queueing action "wait" +1ms
  nightmare queueing action "inject" +1ms
  nightmare queueing action "evaluate" +0ms
  nightmare running +2ms

But it works good with the electrone: xvfb-run electron test/electron.js. And I get a data, that I'm waiting for. And also see tons of logs

Answer 59 · 2016-03-06T20:34:52.000Z

edasque's solution worked for me on Ubuntu 14.04 however my build server uses pm2 to run node. I can't get it to work with pm2. Any suggestions? I would hate to have to switch back to phantom. I've tried below:

pm2 start app.js --node-args="--expose-gc" -- DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24"

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" pm2 start app.js

DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24" && pm2 start app.js

pm2 start app.js --execute-command DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24"

pm2 start DEBUG=nightmare xvfb-run --server-args="-screen 0 1024x768x24” app.js

Answer 60 · 2016-03-07T10:17:51.000Z

I have the same problem as @huemans, I couldn't make PM2, xvfb and Nightmare play well together.

Any ideas?

Answer 61 · 2016-03-08T15:29:06.000Z

Still having issues with the same issue, I tried all of your solutions, everything works perfectly on OSX but when moved to the Linux server it doesn't provide any response. It's the same environment, same versions of node and nightmare etc. On linux it will just not show any output. Any ideas?

npm@3.6.0 /usr/local/lib/node_modules/npm
Air:nm sagi$ npm -l version
{ npm: '3.6.0',
ares: '1.10.1-DEV',
http_parser: '2.6.2',
icu: '56.1',
modules: '47',
node: '5.7.1',
openssl: '1.0.2g',
uv: '1.8.0',
v8: '4.6.85.31',
zlib: '1.2.8' }

DEBUG=nightmare* xvfb-run node cnn.js

nightmare queueing action "goto" for http://cnn.com +0ms
nightmare queueing action "evaluate" +9ms
nightmare running +11ms

Answer 62 · 2016-03-08T19:53:54.000Z

Might be a problem with electron running headless in Linux.

root@Ubuntu-1510-wily-64-minimal /tmp/electron-quick-start # xvfb-run /var/www/growthy/nm/node_modules/electron-prebuilt/dist/electron --type=gpu-process --channel=25956.0.799365973 --no-sandbox --supports-dual-gpus=false --gpu-driver-bug-workarounds=2,51,63 --disable-accelerated-video-decode --gpu-vendor-id=0x1234 --gpu-device-id=0x1111 --gpu-driver-vendor --gpu-driver-version --v8-natives-passed-by-fd --v8-snapshot-passed-by-fd
[27268:0308/205209:FATAL:scoped_file.cc(29)] Check failed: 0 == IGNORE_EINTR(close(fd)). : Bad file descriptor
#0 0x000001ae0a1e
#1 0x000001af6bab
#2 0x000001af716d
#3 0x000001af08c4
#4 0x000001aea530
#5 0x000001af083e
#6 0x000001af06a8
#7 0x000002b6e10d
#8 0x000002b6e015
#9 0x0000010a9fa1
#10 0x0000010a911a
#11 0x000002ea776c main
#12 0x7f93b3386a40 __libc_start_main
#13 0x000000539f09

(That's the command that node is trying to execute)

Answer 63 · 2016-03-18T11:45:40.000Z

@huemans @FdezRomero did you solve that problem? I'm trying to run my nightmare app with PM2 but I can't make it work.

Answer 64 · 2016-03-18T12:00:18.000Z

@gabrielperales No, I haven't found a solution yet. The next thing I was planning to try is the Headless module, a wrapper for Xvfb directly from Node. But I don't know if it will play well with Nightmare. Let me know if you try it!

Answer 65 · 2016-03-18T12:33:55.000Z

@FdezRomero Thank you for your answer. I have tried headless module... but it doesn't seems to work. I'll keep trying

Answer 66 · 2016-03-18T14:46:14.000Z

@FdezRomero @huemans I could do it with this module node-xvfb. I have done a test example here .

Answer 67 · 2016-03-18T14:55:25.000Z

I've got nightmare working in headless mode with code similar to the example from @gabrielperales. However, I do not think the code will work with pm2 (atleast I wasn't able to

Answer 68 · 2016-03-18T17:08:34.000Z

@SantoshSrinivas79 I have tried my code once... and I think it worked, I'll try it again later to be sure

Answer 69 · 2016-03-23T10:15:15.000Z

@gabrielperales - works perfectly, thank you!

Answer 70 · 2016-03-26T02:09:28.000Z

Anyone else running this on heroku and have it working? I'm getting crashing. Everything works fine locally. It works fine for websites on the smaller side, and using larger dynos occasionally lets it finish when it otherwise wouldnt indicating maybe some kind of memory thing?

$ node --version
v5.7.0
$ npm list | grep nightmare
+-- nightmare@2.2.0
$ DEBUG=nightmare* xvfb-run --server-args "-screen 0 1024x768x24" node cnn.js

I'm using all the deps outlined above. The last 3 lines of the log for running cnn.js.

nightmare:log did-get-response-details +3ms [{"sender":{"session":{"cookies":{},"webRequest":{}},"hostWebContents":null,"devToolsWebContents":null,"debugger":{},"_maxListeners":0,"_events":{"page-title-updated":[null,null],"crashed":[null,null],"did-finish-load":[null,null]},"_eventsCount":28,"browserWindowOptions":{"show":false,"alwaysOnTop":true,"webPreferences":{"preload":"/app/node_modules/nightmare/lib/preload.js","nodeIntegration":false}}}},false,"http://rtax.criteo.com/delivery/rta/rta.js?netId=4157&cookieName=crtg_trnr&rnd=10840509575&varName=crtg_content","http://rtax.criteo.com/delivery/rta/rta.js?netId=4157&cookieName=crtg_trnr&rnd=10840509575&varName=crtg_content",200,"GET","http://www.cnn.com/",{"cache-control":["no-cache"],"content-length":["158"],"content-type":["text/javascript; charset=utf-8"],"date":["Sat, 26 Mar 2016 01:38:45 GMT"],"expires":["Mon, 26 Jul 1997 05:00:00 GMT"],"p3p":["CP=\"NON DSP COR CURa PSA PSD OUR BUS NAV STA\""],"pragma":["no-cache"],"server":["Microsoft-IIS/8.5"],"x-powered-by":["ASP.NET"]}]
nightmare:log did-stop-loading +118ms [{"sender":{"session":{"cookies":{},"webRequest":{}},"hostWebContents":null,"devToolsWebContents":null,"debugger":{},"_maxListeners":0,"_events":{"page-title-updated":[null,null],"crashed":[null,null],"did-finish-load":[null,null]},"_eventsCount":28,"browserWindowOptions":{"show":false,"alwaysOnTop":true,"webPreferences":{"preload":"/app/node_modules/nightmare/lib/preload.js","nodeIntegration":false}}}}]
nightmare:log crashed +0ms [{"sender":{"session":{"cookies":{},"webRequest":{}},"hostWebContents":null,"devToolsWebContents":null,"debugger":{},"_maxListeners":0,"_events":{"page-title-updated":[null,null],"crashed":[null,null],"did-finish-load":[null,null]},"_eventsCount":28,"browserWindowOptions":{"show":false,"alwaysOnTop":true,"webPreferences":{"preload":"/app/node_modules/nightmare/lib/preload.js","nodeIntegration":false}}}}]

Looks like it's crashing in the renderer process. Planning on trying to upgrade electron to 0.37 and see what happens. Ideally i'd like to keep this on heroku, but maybe not in the cards. I can make a new issue if more useful.

Answer 71 · 2016-03-28T07:25:39.000Z

@benogle Unfortunately I don't think anyone have made it work on heroku. I tried a lot, but never did make it work. Now I follow this issue to see if anyone does i the future. Maybe when Chromium gets better at running headless.

Answer 72 · 2016-04-15T23:56:34.000Z

I made a buildpack for running Electron on Heroku.

🍰 Works for me. YMMV. 🎉

Answer 73 · 2016-04-19T20:51:51.000Z

Hi, I have updated my example with a Dockerfile. That is all you will need to run your nightmarejs code in headless mode. I hope you can enjoy it.

Answer 74 · 2016-05-16T17:34:34.000Z

hey @benschwarz , how did you make it work with heroku? which dyno size were you using? mine is crashing after a certain (very small) amount of actions queued? I think I'm running into the whole heroku /dev/shm being too small (5 mb in heroku I believe, which is useless).
Also, I'm getting electron:stderr Xlib: extension "RANDR" missing on display ":99"., not sure if relevant...

Answer 75 · 2016-05-16T18:27:45.000Z

@rickmed I found it to be pretty unstable too (admittedly, on the 512m dyno), I decided to use phantomjs instead. I'd love to have used electron and nightmare, but instead I shipped phantomjs and had no issues since. I'm sure there is a simple reason that this is happening on heroku environments, but I just didn't have the time to spend on it.

Answer 76 · 2016-05-18T03:09:26.000Z

Think I found the "best" answer:

Official Electron documentation on testing headlessly

It shows what to put in .travis.yml and recommends the super-handy xvfb-maybe utility.

Answer 77 · 2016-06-06T01:29:39.000Z

Simple gist with commands for run nightmare headless.

Answer 78 · 2016-06-06T12:45:08.000Z

Is this issue still a bug in nightmare, or it was fixed in any new version..?? I am not able to run nightmare on google appengine for node.js using any of provided solution. Asked regarding this on stackoverflow. Can someone answer my question over there.

Answer 79 · 2016-06-06T17:00:44.000Z

This is not a nightmare issue but an electron (chromium really) issue. @rosshinkley maybe this should be closed?

@hhsadiq I'll try to help you via stackoverflow.

EDITED: for anyone interested in the answer: http://stackoverflow.com/questions/37625327/how-to-run-nightmare-js-on-google-appengine-for-node-js/37663861#37663861

Answer 80 · 2016-06-07T23:57:54.000Z

@rickmed I was planning on closing it with the resolution of #502 (where Nightmare would take care of the framebuffer for you, at least until Chromium's headless features trickle into Electron), but the details of that have proven much more difficult than I had anticipated. It's not even as simple as "just use Xvfb" as sometimes this causes Chromium to hang.

Answer 81 · 2016-06-14T13:52:19.000Z

Solution with Docker

The simplest way i found to get it working with docker...

Dockerfile

FROM node:argon

RUN apt-get update &&\
    apt-get install -y libgtk2.0-0 libgconf-2-4 \
    libasound2 libxtst6 libxss1 libnss3 xvfb

WORKDIR /app

RUN npm install nightmare
COPY cnn.js .

COPY entrypoint /
RUN chmod +x /entrypoint
ENTRYPOINT ["/entrypoint"]

CMD DEBUG=nightmare node cnn.js

entrypoint

#!/usr/bin/env bash
set -e

# Start Xvfb
Xvfb -ac -screen scrn 1280x2000x24 :9.0 &
export DISPLAY=:9.0

exec "$@"

cnn.js

var Nightmare = require('nightmare'),
  nightmare = Nightmare();

nightmare.goto('http://cnn.com')
  .evaluate(function(){
    return document.title;
  })
  .end()
  .then(function(title){
    console.log(title);
  })

Build image

docker build -t cnn-example .

Then run and see the results

docker run cnn-example
Tue, 14 Jun 2016 13:38:35 GMT nightmare queuing process start
Tue, 14 Jun 2016 13:38:35 GMT nightmare queueing action "goto" for http://cnn.com
Tue, 14 Jun 2016 13:38:35 GMT nightmare queueing action "evaluate"
Tue, 14 Jun 2016 13:38:35 GMT nightmare running
Tue, 14 Jun 2016 13:38:51 GMT nightmare electron child process exited with code 0: success!
CNN - Breaking News, U.S., World, Weather, Entertainment & Video News

I'm using docker-machine with virtualbox driver

Answer 82 · 2016-07-19T00:02:58.000Z

I got GAE deploys working in a simpler fashion described here: http://stackoverflow.com/a/38447642/562915

Answer 83 · 2016-07-26T06:29:08.000Z

I got it working on Gitlab-Ci

image:  node:latest

before_script:
  # Install and start virtual screen
  - apt-get update
  - apt-get install -y libgtk2.0-0 libgconf-2-4 libasound2 libxtst6 libxss1 libnss3 xvfb
  - Xvfb -ac -screen scrn 1280x2000x24 :9.0 &
  - export DISPLAY=:9.0

  # Install npm dependencies
  - npm install --silent

build_job:
  script:
    - "npm test"

Answer 84 · 2016-08-12T00:49:57.000Z

@huemans, @FdezRomero, @gabrielperales, @SantoshSrinivas79 . To get nightmare working headlessly with pm2, you just need to wrap your mind around what Xvfb does, what xvfb-run does, how they interact with your node process, and how this interaction is facilitated.

Going to give you an idea of the basics and then tell you how to get this all going in pm2. Forgive me if I am oversimplifying things:

Xvfb is a frame buffer which allows us to mock a desktop interface where, in this case, the electron browser can run.
Xvfb sets a display number (the initial default is "99"). The application using the desktop must have an idea of this display number to use the frame buffer.
xvfb-run (which is a shell script) is perfect for single process node applications as what it actually does (as far as I understand) is setup the Xvfb process, set the "DISPLAY" environment variable to ":99", and then spawn the process you specify with access to the environment variable so it can access the display.
For PM2 to manage processes happily it must spawn all the processes that it controls. I think this is where xvfb-run falls apart with pm2. It creates processes that need to be controlled and then exits and pm2 doesn't like this.
Instead what we can do is spawn two processes simultaneously (do what xvfb-run does essentially): one being Xvfb and the other being a process for your application, setting the display number and "DISPLAY" environment variable explicitly.
The best way to spawn the two processes is to create a PM2 Process file json as explained here in pm2 docs, and then use the pm2 start command.

This is what your process file would look like, substituting for your specific case:

{
  "apps" : [{
    "name"        : "<your_app_name>",
    "script"      : "<your_main.js>",
    "env": {
      "DISPLAY": ":99"
    }
  },
    {
      "name"        : "Xvfb",
      "interpreter" : "none",
      "script"      : "Xvfb",
      "args"        : ":99"
    }]
}

If you place this process file in the same directory as <your_main.js>, and feed the process file to pm2, you're all set. Assuming we call the file process.json then you would execute this pm2 command to get the two processes started:

pm2 start process.json

Tested and working on Debian 8.

Answer 85 · 2016-09-01T14:40:11.000Z

Everybody is going astray. the bug is simple, just get my F* bot to work... and yet everybody is failing to give a simple answer.This is the oneliner that will save your day ( obviously if you have xfvb installed... modify to your needs.. this one is for slackware 14.1).

*Note, on some systems "xvfb" is called "Xvfb" (capital X) and in other "xvfb-run"

DEBUG=nightmare* Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 & node --harmony app.js

Answer 86 · 2016-09-06T19:57:53.000Z

~~I'm not much of a terminal user, and while I tried Electron's travis-ci recommendations, my tests are still timing out; even with 10+ second limits.~~

The only thing that worked for me was to use xvfb-maybe and to use a forced viewport size (via browser.viewport()).

Answer 87 · 2016-09-18T14:50:34.000Z

I got an image working based on this discussion, but when I create a screenshot, I don't see any text. Selecting any text via nightmare works though. Anyone ever seen that issue?

SOLUTION : I based my docker image on 'node:slim', after changing to 'node:6.6.0-wheezy' it works!

Answer 88 · 2016-10-14T09:55:29.000Z

I tried to install
this yum install xorg-x11-server-Xvfb gtk2 libXtst GConf2 alsa-lib xorg-x11-fonts*
but it doesn't work

Node version

v6.4.0

Centos Version

v7.2

Do centos work nightmarejs?

Answer 89 · 2016-10-18T14:19:31.000Z

From a performance POV, does anyone know how Nightmare running with Xvfb compares to PhantomJS as headless?

Answer 90 · 2016-10-18T17:49:11.000Z

@pelhage well, Electron sure doesn't take a few seconds to open like PhantomJS does.

Answer 91 · 2016-10-18T18:16:42.000Z

@pelhage I haven't used phantomJS since Nightmare decided to bundle electron instead of webkit. I think there are many reason you should choose Electron instead of phantom's webkit such as but not limited to :
-Browser technology up to date with standards (unlike phantomJS webkit)
-Electron is becoming a defacto standard for automation testing and it will probably take over Selenium

Answer 92 · 2016-10-21T19:53:33.000Z

I used @mashaalmemon 's solution for my Scraper cron job, and it worked for the first day, but ever since I've been getting error messages in my log and my scraper won't run. Not sure how to decipher them, can someone maybe point me in the right direction or tell me what typically causes these errors?

Gist of logs https://gist.github.com/pelhage/3f43240a3bd36a7ec06cb09d06c3be92

EDIT: I fixed this by creating a new instance of Nightmare upon each cron job. My error was related to an instance of my scraper being run multiple times, as opposed to creating a new scraper each time.

Answer 93 · 2016-11-17T18:08:55.000Z

The ultimate solution.
pm2 start --interpreter xvfb-run npm -- start

Answer 94 · 2016-11-23T23:10:32.000Z

Has anyone successfully gotten Nightmare to work on Heroku? If so, can you share your Dockerfile?

Works locally. Seeing this when running on Heroku:

cc @rosshinkley

Answer 95 · 2016-11-24T02:31:30.000Z

@stevenmiller888 Take a look at https://github.com/captain401/heroku-buildpack-xvfb. You'll also want heroku-buildpack-apt as in the instructions.

(You might also need to include libnotify4 and libxss1 in your Aptfile - let me know if I need to update the instructions.)

Answer 96 · 2016-12-07T02:52:46.000Z

My website was deployed on EC2, and I tried sudo DEBUG=nightmare xvfb-run --server-args "-screen 0 1024x768x24" node app.js, but it processes the request only once

Answer 97 · 2016-12-07T14:22:46.000Z

@otaviomedeiros : Hi ! Did u manage to launch multiple long script at the same time with this config ?
I tried and when i launch 4-5 times the same script on the same xvfb display it's hang and i got some nightmare .wait() timeout.
Same if i launch multiple xvfb display and attach one script to one display.
Seems that a docker instance can't handle many xvfb display...
No error output from nightmare. Just a Error: .wait() timed out after 180000msec at last

Answer 98 · 2016-12-07T15:51:03.000Z

@dachow : it's seems that you killed the electron child process before the other request had time to finish. show us your code !

Answer 99 · 2016-12-08T01:39:03.000Z

@GautierT Hi! I have ran it with express. The basic structure of code like this:

router.get('/', function (req, res, next) {
    nightmare
        .useragent('...')
        .goto('https://www.instagram.com/')
        .click('._aj7mu._taytv._ki5uo._o0442')
        .insert('form input[type="email"]', 'xxx@gmail.com')
        .insert('form input[type="password"]', 'xxx')
        .click('form button')
        .wait('article')
        .evaluate(function () {
            // return something
        }
        .end()
        .then(function(result){
            // some handlers
            res.render('body', {
                items: items
            });
        })
        .catch(function (error) {
            console.error('Search failed:', error);
        });
}

It processes the request only once