spec-generator, ReSpec, PhantomJS stuck with HTML 5.1 ED

Question

spec-generator, ReSpec, PhantomJS stuck with HTML 5.1 ED

tripu opened this issue 8 years ago · 4 comments

Trying to understand what is failing here (it might be an issue with spec-generator, with ReSpec, with PhantomJS, or an error of interpretation on my part):

Locally, I have set the timeout for ReSpec generation to a very long time.
I have then launched the server,
and tried to generate the WD for the latest ED of HTML 5.1 by visiting
http://localhost/?type=respec&url=https://w3c.github.io/html/?specStatus=WD;shortName=html51.
The HTTP response times out, after more than 3 minutes.
To find out what's failing, I have inspected the call to PhantomJS, copied all parameters
and invoked PhantomJS from the command line with those same parameters (adding --debug=true).
This is the result:

$ /home/tripu/t/repo/spec-generator/node_modules/phantomjs/lib/phantom/bin/phantomjs --debug=true --ssl-protocol=any /home/tripu/t/repo/spec-generator/node_modules/respec/tools/respec2html.js --exclude-script https://www.w3.org/scripts/ 'https://w3c.github.io/html/?specStatus=WD;shortName=html51;publishDate=2016-03-21;previousMaturity=WD;previousPublishDate=2016-03-10'
2016-03-21T21:27:54 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persisten cookie storage) 
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Configuration 
2016-03-21T21:27:54 [DEBUG]      0 objectName : "" 
2016-03-21T21:27:54 [DEBUG]      1 cookiesFile : "" 
2016-03-21T21:27:54 [DEBUG]      2 diskCacheEnabled : "false" 
2016-03-21T21:27:54 [DEBUG]      3 maxDiskCacheSize : "-1" 
2016-03-21T21:27:54 [DEBUG]      4 ignoreSslErrors : "false" 
2016-03-21T21:27:54 [DEBUG]      5 localToRemoteUrlAccessEnabled : "false" 
2016-03-21T21:27:54 [DEBUG]      6 outputEncoding : "UTF-8" 
2016-03-21T21:27:54 [DEBUG]      7 proxyType : "http" 
2016-03-21T21:27:54 [DEBUG]      8 proxy : ":1080" 
2016-03-21T21:27:54 [DEBUG]      9 proxyAuth : ":" 
2016-03-21T21:27:54 [DEBUG]      10 scriptEncoding : "UTF-8" 
2016-03-21T21:27:54 [DEBUG]      11 webSecurityEnabled : "true" 
2016-03-21T21:27:54 [DEBUG]      12 offlineStoragePath : "" 
2016-03-21T21:27:54 [DEBUG]      13 offlineStorageDefaultQuota : "-1" 
2016-03-21T21:27:54 [DEBUG]      14 printDebugMessages : "true" 
2016-03-21T21:27:54 [DEBUG]      15 javascriptCanOpenWindows : "true" 
2016-03-21T21:27:54 [DEBUG]      16 javascriptCanCloseWindows : "true" 
2016-03-21T21:27:54 [DEBUG]      17 sslProtocol : "any" 
2016-03-21T21:27:54 [DEBUG]      18 sslCertificatesPath : "" 
2016-03-21T21:27:54 [DEBUG]      19 webdriver : ":" 
2016-03-21T21:27:54 [DEBUG]      20 webdriverLogFile : "" 
2016-03-21T21:27:54 [DEBUG]      21 webdriverLogLevel : "INFO" 
2016-03-21T21:27:54 [DEBUG]      22 webdriverSeleniumGridHub : "" 
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Script & Arguments 
2016-03-21T21:27:54 [DEBUG]      script: "/home/tripu/t/repo/spec-generator/node_modules/respec/tools/respec2html.js" 
2016-03-21T21:27:54 [DEBUG]      0 arg: "--exclude-script" 
2016-03-21T21:27:54 [DEBUG]      1 arg: "https://www.w3.org/scripts/" 
2016-03-21T21:27:54 [DEBUG]      2 arg: "https://w3c.github.io/html/?specStatus=WD;shortName=html51;publishDate=2016-03-21;previousMaturity=WD;previousPublishDate=2016-03-10" 
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Starting normal mode 
2016-03-21T21:27:54 [DEBUG] WebPage - setupFrame "" 
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )  
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 10 
2016-03-21T21:27:54 [DEBUG] WebpageCallbacks - getGenericCallback 
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 26 
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 32 
2016-03-21T21:27:54 [DEBUG] Network - Resource request error: 301 ( "Protocol "" is unknown" ) URL: "" 
2016-03-21T21:27:54 [DEBUG] Network - Resource request error: 203 ( "Error downloading https://w3c.github.io/default.css - server replied: Not Found" ) URL: "https://w3c.github.io/default.css" 
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 37 
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 38 
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 38 
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 42 
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 46 
2016-03-21T21:27:56 [DEBUG] WebPage - setupFrame "" 
2016-03-21T21:27:56 [DEBUG] WebPage - updateLoadingProgress: 51 
2016-03-21T21:27:56 [DEBUG] WebPage - updateLoadingProgress: 100 
2016-03-21T21:27:56 [DEBUG] WebPage - setupFrame "" 
2016-03-21T21:27:56 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function() { setTimeout(function () {
      $.ajaxSetup({
        timeout: 4000
      });

      function saveToPhantom() {
        require(["core/ui", "ui/save-html"], function(ui, saver) {
          saver.show(ui, respecConfig, document, respecEvents);
          window.callPhantom({
            html: saver.toString()
          });
        });
      }
      if (document.respecDone) {
        saveToPhantom();
      } else {
        respecEvents.sub("end-all", saveToPhantom);
      }
    }, 0); })(); })()" 
2016-03-21T21:27:56 [DEBUG] WebPage - evaluateJavaScript result QVariant(, )
^C

The command gets to that last line quickly, but then never returns; I have to Ctrl-C.

Am I doing something wrong?
That particular page is small; but it's a very long spec with multiple pages (perhaps that's causing the error?).
I am not very familiar with PhantomJS. Is there an easy way to debug this, and understand where it's stuck?

Answer 1 · 2016-03-21T14:13:34.000Z

Stupid question (maybe), but are you sure ReSpec is linked in the doc? I'm on my phone, so can't check... But the HTML was not originally written using ReSpec, hence my question.

If this has changed (I thought WHATWG was using BikeShed now), then we should talk.

On 21 Mar 2016, at 11:43 PM, tripu notifications@github.com wrote:

Trying to understand what is failing here (it might be an issue with spec-generator, with ReSpec, with PhantomJS, or an error of interpretation on my part):

Locally, I have set the timeout for ReSpec generation to a very long time.
I have then launched the server,
and tried to generate the WD for the latest ED of HTML 5.1 by visiting
http://localhost/?type=respec&url=https://w3c.github.io/html/?specStatus=WD;shortName=html51.
The HTTP response times out, after more than 3 minutes.
To find out what's failing, I have inspected the call to PhantomJS, copied all parameters
and invoked PhantomJS from the command line with those same parameters (adding --debug=true).
This is the result:
$ /home/tripu/t/repo/spec-generator/node_modules/phantomjs/lib/phantom/bin/phantomjs --debug=true --ssl-protocol=any /home/tripu/t/repo/spec-generator/node_modules/respec/tools/respec2html.js --exclude-script https://www.w3.org/scripts/ 'https://w3c.github.io/html/?specStatus=WD;shortName=html51;publishDate=2016-03-21;previousMaturity=WD;previousPublishDate=2016-03-10'
2016-03-21T21:27:54 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=' to enable persisten cookie storage)
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Configuration
2016-03-21T21:27:54 [DEBUG] 0 objectName : ""
2016-03-21T21:27:54 [DEBUG] 1 cookiesFile : ""
2016-03-21T21:27:54 [DEBUG] 2 diskCacheEnabled : "false"
2016-03-21T21:27:54 [DEBUG] 3 maxDiskCacheSize : "-1"
2016-03-21T21:27:54 [DEBUG] 4 ignoreSslErrors : "false"
2016-03-21T21:27:54 [DEBUG] 5 localToRemoteUrlAccessEnabled : "false"
2016-03-21T21:27:54 [DEBUG] 6 outputEncoding : "UTF-8"
2016-03-21T21:27:54 [DEBUG] 7 proxyType : "http"
2016-03-21T21:27:54 [DEBUG] 8 proxy : ":1080"
2016-03-21T21:27:54 [DEBUG] 9 proxyAuth : ":"
2016-03-21T21:27:54 [DEBUG] 10 scriptEncoding : "UTF-8"
2016-03-21T21:27:54 [DEBUG] 11 webSecurityEnabled : "true"
2016-03-21T21:27:54 [DEBUG] 12 offlineStoragePath : ""
2016-03-21T21:27:54 [DEBUG] 13 offlineStorageDefaultQuota : "-1"
2016-03-21T21:27:54 [DEBUG] 14 printDebugMessages : "true"
2016-03-21T21:27:54 [DEBUG] 15 javascriptCanOpenWindows : "true"
2016-03-21T21:27:54 [DEBUG] 16 javascriptCanCloseWindows : "true"
2016-03-21T21:27:54 [DEBUG] 17 sslProtocol : "any"
2016-03-21T21:27:54 [DEBUG] 18 sslCertificatesPath : ""
2016-03-21T21:27:54 [DEBUG] 19 webdriver : ":"
2016-03-21T21:27:54 [DEBUG] 20 webdriverLogFile : ""
2016-03-21T21:27:54 [DEBUG] 21 webdriverLogLevel : "INFO"
2016-03-21T21:27:54 [DEBUG] 22 webdriverSeleniumGridHub : ""
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Script & Arguments
2016-03-21T21:27:54 [DEBUG] script: "/home/tripu/t/repo/spec-generator/node_modules/respec/tools/respec2html.js"
2016-03-21T21:27:54 [DEBUG] 0 arg: "--exclude-script"
2016-03-21T21:27:54 [DEBUG] 1 arg: "https://www.w3.org/scripts/"
2016-03-21T21:27:54 [DEBUG] 2 arg: "https://w3c.github.io/html/?specStatus=WD;shortName=html51;publishDate=2016-03-21;previousMaturity=WD;previousPublishDate=2016-03-10"
2016-03-21T21:27:54 [DEBUG] Phantom - execute: Starting normal mode
2016-03-21T21:27:54 [DEBUG] WebPage - setupFrame ""
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/_coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/package.json" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/coffee-script.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./lexer.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./parser.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./nodes.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././scope.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././helpers.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/././lexer.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/../coffee-script/./lib/coffee-script/./././rewriter.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r") ) )
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 10
2016-03-21T21:27:54 [DEBUG] WebpageCallbacks - getGenericCallback
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 26
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 32
2016-03-21T21:27:54 [DEBUG] Network - Resource request error: 301 ( "Protocol "" is unknown" ) URL: ""
2016-03-21T21:27:54 [DEBUG] Network - Resource request error: 203 ( "Error downloading https://w3c.github.io/default.css - server replied: Not Found" ) URL: "https://w3c.github.io/default.css"
2016-03-21T21:27:54 [DEBUG] WebPage - updateLoadingProgress: 37
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 38
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 38
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 42
2016-03-21T21:27:55 [DEBUG] WebPage - updateLoadingProgress: 46
2016-03-21T21:27:56 [DEBUG] WebPage - setupFrame ""
2016-03-21T21:27:56 [DEBUG] WebPage - updateLoadingProgress: 51
2016-03-21T21:27:56 [DEBUG] WebPage - updateLoadingProgress: 100
2016-03-21T21:27:56 [DEBUG] WebPage - setupFrame ""
2016-03-21T21:27:56 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function() { setTimeout(function () {
$.ajaxSetup({
timeout: 4000
});
  function saveToPhantom() {
    require(["core/ui", "ui/save-html"], function(ui, saver) {
      saver.show(ui, respecConfig, document, respecEvents);
      window.callPhantom({
        html: saver.toString()
      });
    });
  }
  if (document.respecDone) {
    saveToPhantom();
  } else {
    respecEvents.sub("end-all", saveToPhantom);
  }
}, 0); })(); })()" 
2016-03-21T21:27:56 [DEBUG] WebPage - evaluateJavaScript result QVariant(, )
^C
The command gets to that last line quickly, but then never returns; I have to Ctrl-C.

Am I doing something wrong?
That particular page is small; but it's a very long spec with multiple pages (perhaps that's causing the error?).
I am not very familiar with PhantomJS. Is there an easy way to debug this, and understand where it's stuck?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

Answer 2 · 2016-03-22T04:43:19.000Z

@marcoscaceres, you're right. It's a silly mistake: the HTML spec is built with Bikeshed.

Answer 3 · 2016-03-22T05:24:22.000Z

BTW, I'm working on a new respec2html.js version... it uses Nightmare.js, which uses Chromium instead of the really really really outdated WebKit. It's also view-able, and had a built-in debugger (it's just Chrome)... so it's easier to see what is going on and failing. Hope to publish soon.

Answer 4 · 2016-03-22T05:56:24.000Z

Great, @marcoscaceres!