Unbuffered code structure could not be parsed

Question

Unbuffered code structure could not be parsed

BananaAcid opened this issue 5 years ago · 10 comments

What does this error mean? Is it a bug?
Not in a Promise, the code works - but I want the callback values - that is why I am going for then-pug

  Error: C:\htdocs\abcdefghijklmno\com\views\test-mail.pug:7:1
  
  Unbuffered code structure could not be parsed; Unexpected token, expected ; (14:16) in 
  let sm = new Promise( function(ret) { 
  	sendmail({
  		from: 'no-reply@abcdefghijklmno.com',
  		to: 'test@test.test',
  		subject: 'test sendmail',
  		html: 'hello world'
  	},
  		function(e,r) {ret({e,r});}
  	)
  });
  
  
  let res = await sm();

      at makeError (C:\htdocs\abcdefghijklmno\com\node_modules\pug-error\index.js:32:13)
      at Compiler.error (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:110:15)
      at Compiler.visitCode (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:1039:16)
      at Compiler.visitNode (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:478:37)
      at Compiler.visit (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:466:26)
      at Compiler.visitBlock (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:554:28)
      at Compiler.visitNode (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:478:37)
      at Compiler.visit (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:466:26)
      at Compiler.compile (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen-module.js:263:26)
      at generateCode (C:\htdocs\abcdefghijklmno\com\node_modules\then-pug\lib\pug-code-gen.js:32:39)

Answer 1 · 2019-12-20T19:47:53.000Z

Hello,
I am not sure I understand what you want to do.
I would have to see the template to try and give you an explanation of the error.

You are probably using unbuffered code in your template (direct javascript code in the template maybe with if/else clause or brackets around pug syntax that the parser has a difficulty to understand.

The current version of then-pug works at the AST level and needs to decide where the unbuffered code starts and end. There may be a bug depending on what type of unbuffered code you have.

Answer 2 · 2019-12-21T08:23:48.000Z

sure:

test-mall.pug

div= ctx.state.type

-
	let sm = new Promise( function(ret) { 
		sendmail({
  			from: 'no-reply@abcdefghijklmno.com',
  			to: 'test@test.test',
  			subject: 'test sendmail',
  			html: 'hello world'
  		},
  			function(e,r) {ret({e,r});}
  		)
	});
	
	let res = await sm();

pre= insp(res)

div done

sendmail is a callback based fn passed in on locals (https://www.npmjs.com/package/sendmail)
ctx.state.type is just some test string from a koa2 middleware, ctx in on locals
insp is passed in on locals from require('util').inspect()
I also tried async(ret => sendmail() ... syntax with same result

The goal in this case, is to get the sendmail status.

Answer 3 · 2019-12-21T11:05:55.000Z

I tested the template. It seems that the parsing breaks because of the await keyword (compile works ok without the keyword)

then-pug was developed before async / await were introduced in node.js and they are not tested. Obviously there is some work to accept that.

then-pug works with generators so the way to async the code is to use yield.

can you try with

let res = yield sm();

other than that, make sure that Promise and sendmail are available in the templates.

you can check the tests in https://github.com/pugjs/then-pug/blob/master/packages/then-pug/test/cases-then-pug/gn-for-loop.pug and see how they are invoked in https://github.com/pugjs/then-pug/blob/master/packages/then-pug/test/run-utils.js#L52

Answer 4 · 2019-12-21T18:58:27.000Z

Thanks, is there going to be some effort in supporting Promise/async keywords? Where would that needed to be changed/extended?

Answer 5 · 2019-12-21T19:38:48.000Z

Did you try with yield and did it work ?

for your information (you can see it with compileClient) the generated code for the template is

function template(locals, pug, buf) {
  var pug_mixins = locals.pug_mixins || {},
      pug_interp,
      _ref = locals || {};

  var _ret = function (Promise, ctx, insp, sendmail) {
    function* gen() {
      buf.push("\u003Cdiv\u003E" + pug.escape(null == (pug_interp = ctx.state.type) ? "" : pug_interp) + "\u003C\u002Fdiv\u003E");
      let sm = new Promise(function (ret) {
        sendmail({
          from: 'no-reply@abcdefghijklmno.com',
          to: 'test@test.test',
          subject: 'test sendmail',
          html: 'hello world'
        }, function (e, r) {
          ret({
            e,
            r
          });
        });
      });
      let res = yield sm();
      buf.push("\u003Cpre\u003E" + pug.escape(null == (pug_interp = insp(res)) ? "" : pug_interp) + "\u003C\u002Fpre\u003E\u003Cdiv\u003Edone \u003C\u002Fdiv\u003E");
    }

    return {
      v: gen
    };
  }.call(this, "Promise" in _ref ? _ref.Promise : typeof Promise !== "undefined" ? Promise : undefined, "ctx" in _ref ? _ref.ctx : typeof ctx !== "undefined" ? ctx : undefined, "insp" in _ref ? _ref.insp : typeof insp !== "undefined" ? insp : undefined, "sendmail" in _ref ? _ref.sendmail : typeof sendmail !== "undefined" ? sendmail : undefined);

  return _ret.v;
}

for all keywords that are not considered "global", they are resolved from locals

so you can already handle Promise by adding a reference to it in your locals (see what i mean?), probably just like you do for sendmail.

I don't know yet if it would be a good think to always add it ;

regarding await, I have to dig a little to understand what that would mean inside a generator (I am not too familiar with the interaction between generators and async/await and never wondered about it)

I think that first you need to have a working version of then-pug and understand why yield is needed in your case.

then if we agree on what it would mean to add async/await to then-pug (the streamable aspect of then-pug is important) the changes would need to modify the generated code that you see above. This code is generated inside lib/pug-code-gen.js which inherits lib/pug-code-gen-module.js which is itself an AST-first port of the original pug-code-gen. You can see pugjs/pug#2708 for some history/information on this AST-first port.

Modifying this should not be too hard once the idea/template of the new generated code is agreed upon.

tell me if you manage to make your example work + tell me more about what you think you need then-pug for. The use cases are narrow enough that native pug is nearly always the best candidate.

Answer 6 · 2019-12-21T20:21:53.000Z

I added Promise to locals and got:

   TypeError: sm is not a function
      at gen (eval at compileStreaming (C:\htdocs\1234567890\node_modules\then-pug\lib\index.js:309:12), <anonymous>:36:25)
      at gen.next (<anonymous>)
      at evaluate (C:\htdocs\1234567890\node_modules\then-yield\index.js:51:27)
      at Object.spawn (C:\htdocs\1234567890\node_modules\then-yield\index.js:62:10)
      at res (C:\htdocs\1234567890\node_modules\then-pug\lib\index.js:350:21)
      at res (C:\htdocs\1234567890\node_modules\then-pug\lib\index.js:387:12)
      at C:\htdocs\1234567890\lib\cache-pug.mjs:23:30
      at new Promise (<anonymous>)
      at CachePug.render (C:\htdocs\1234567890\lib\cache-pug.mjs:23:10)
      at async C:\htdocs\1234567890\index.mjs:442:15

I found Using JavaScript Generators to yield Promises stating, a combination of yield Promise is not possible - but I don't quite get the rest

Was also trying to use AsyncFunction, added through locals - and yield the created func - which did not seem to work either

edit - frankensteined it (and it works - inspiration)
see wrapper fns here: asyncWrappers.pug

syncifyCb()
syncifyPromise()
syncifyPromiseFactory()

Answer 7 · 2019-12-22T14:56:55.000Z

it is hard to debug remotely a code for which I do not have all the keys.

in your original code, if sm is a Promise, the code should be yield sm, not yield sm(). Can you try this ?

using then-pug and yielding promises is what I do in my code. It should work without all the syncify* code you wrote.

then-pug internally uses the then-yield module to be able to do just that.

keep me posted. It should be simpler that your frankenstein version !

Answer 8 · 2019-12-22T19:43:21.000Z

Sorry wont work. Even placing a simple Promise wont work. Promises seem not be able to be yielded. With the functions/Promises at asyncWrappers.pug - I testet it.

Could you give me a dependency less code I could drop into a pug that should work? (I know, I would still need to pass Promise in)

Answer 9 · 2019-12-22T21:06:03.000Z

the simplest I can come with is the following

var pug = require('then-pug');
var tpl = `
-
  var d = Promise.resolve('test1')
  var s = yield d
div= s
`
var fn = pug.compileStreaming(tpl)

fn({}).pipe(process.stdin)

results in

<div>test1</div>

In fact you don't even need to pass Promise in locals because it is available in the global node namespace.

tell me if that helps

Answer 10 · 2019-12-24T11:17:22.000Z

... and it works. Really - no idea it didn't before, with quite the same test

-
	var s = yield new Promise( r => setTimeout( () => r('5s done'), 5000 ) );
div= s

and

-
	let smRes = yield new Promise( r => sendmail({
		from: 'no-reply@test.test',
		to: 'test@test.test',
		subject: 'test sendmail',
		html: 'Hello world.'
	}, (err,res) => r({err,res})) );

pre= JSON.stringify(smRes, null, 2)

very strange :/

But still no way to await async functions (AsyncFucntion constructor).