OvermindDL1/protocol_ex

Escript failure

olafura opened this issue · 14 comments

olafura commented

I'm using ProtocolEx for my library:
https://github.com/olafura/beam_to_ex_ast

And I'm really impressed ProtocolEx.

But I'm having problem with my escript script for it:
https://github.com/olafura/beam_to_ex

It doesn't seem for some reason that the protocols aren't getting aggregated, only
when using escript.build, not mix run or iex.

OvermindDL1 commented

Well as long as the compiler is registered last (like it is in your link just fine) then it is added to the compilers list and should be run to generate the files (into the usual beam directory). Thus whenever a build happens then the files should get compiled.

I tried this:

╰─➤  git clone https://github.com/olafura/beam_to_ex.git
Cloning into 'beam_to_ex'...
cd bea
remote: Counting objects: 28, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 28 (delta 1), reused 1 (delta 0), pack-reused 24
Unpacking objects: 100% (28/28), done.
╰─➤  cd beam_to_ex 
╰─➤  mix deps.get
warning: variable "deps" does not exist and is being expanded to "deps()", please use parentheses to remove the ambiguity or change 
the variable name
  mix.exs:12

Resolving Hex dependencies...
Dependency resolution completed:
  beam_to_ex_ast 0.3.0
  forms 0.0.1
  protocol_ex 0.3.11
* Getting beam_to_ex_ast (Hex package)
  Checking package (https://repo.hex.pm/tarballs/beam_to_ex_ast-0.3.0.tar)
  Fetched package
* Getting forms (Hex package)
  Checking package (https://repo.hex.pm/tarballs/forms-0.0.1.tar)
  Fetched package
* Getting protocol_ex (Hex package)
  Checking package (https://repo.hex.pm/tarballs/protocol_ex-0.3.11.tar)
  Fetched package
╰─➤  iex -S mix
Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:false]

warning: variable "deps" does not exist and is being expanded to "deps()", please use parentheses to remove the ambiguity or change 
the variable name
  mix.exs:12

===> Compiling forms
==> protocol_ex
Compiling 2 files (.ex)
Generated protocol_ex app
==> beam_to_ex_ast
Compiling 33 files (.ex)
...snip *ton* of warnings!!...
Generated beam_to_ex app
Interactive Elixir (1.6.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> BeamToExAst.Translate.__proto_ex_impls__()       
[BeamToExAst.Translate.BeamToExAst.Var, BeamToExAst.Translate.BeamToExAst.Tuple,
 BeamToExAst.Translate.BeamToExAst.Try,
 BeamToExAst.Translate.BeamToExAst.String,
 BeamToExAst.Translate.BeamToExAst.Remote,
 BeamToExAst.Translate.BeamToExAst.RecordField,
 BeamToExAst.Translate.BeamToExAst.Record,
 BeamToExAst.Translate.BeamToExAst.Op2, BeamToExAst.Translate.BeamToExAst.Op1,
 BeamToExAst.Translate.BeamToExAst.Nil, BeamToExAst.Translate.BeamToExAst.Match,
 BeamToExAst.Translate.BeamToExAst.MapFieldExact,
 BeamToExAst.Translate.BeamToExAst.MapFieldAssoc,
 BeamToExAst.Translate.BeamToExAst.Map, BeamToExAst.Translate.BeamToExAst.List,
 BeamToExAst.Translate.BeamToExAst.Lc,
 BeamToExAst.Translate.BeamToExAst.Integer,
 BeamToExAst.Translate.BeamToExAst.If, BeamToExAst.Translate.BeamToExAst.Fun,
 BeamToExAst.Translate.BeamToExAst.Float,
 BeamToExAst.Translate.BeamToExAst.Cons,
 BeamToExAst.Translate.BeamToExAst.Clauses,
 BeamToExAst.Translate.BeamToExAst.Clause,
 BeamToExAst.Translate.BeamToExAst.Char, BeamToExAst.Translate.BeamToExAst.Case,
 BeamToExAst.Translate.BeamToExAst.Call,
 BeamToExAst.Translate.BeamToExAst.Block,
 BeamToExAst.Translate.BeamToExAst.BinElement,
 BeamToExAst.Translate.BeamToExAst.Bin, BeamToExAst.Translate.BeamToExAst.Bc,
 BeamToExAst.Translate.BeamToExAst.BGenerate,
 BeamToExAst.Translate.BeamToExAst.Atom]

It looks like it is consolidated with iex? Do you have a complete reproduceable example or so that I can try with it, like an entire shell session or so?

OvermindDL1 commented

As well as:

╰─➤  mix escript.build
...snip *ton* more warnings!!!...

╰─➤  unzip -l beam_to_ex | grep 'Trans'
warning [beam_to_ex]:  66 extra bytes at beginning or within zipfile
  (attempting to process anyway)
      790  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.List.beam
      935  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.RecordField.beam
      952  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Op2.beam
      911  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Case.beam
      720  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Float.beam
      826  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Clauses.beam
      950  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Atom.beam
      909  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Op1.beam
      890  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Call.beam
      716  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Char.beam
     1225  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.BinElement.beam
      847  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Block.beam
      922  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Lc.beam
     1568  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Remote.beam
      748  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.MapFieldExact.beam
      954  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Cons.beam
     1192  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Map.beam
     1154  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Clause.beam
      897  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.MapFieldAssoc.beam
     6327  11-02-2017 10:20   Elixir.Logger.Translator.beam
      928  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Record.beam
     1190  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Tuple.beam
     1259  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Bin.beam
     1101  11-02-2017 10:20   Elixir.BeamToExAst.Translate.beam
      984  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Fun.beam
      985  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.BGenerate.beam
     1128  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Var.beam
      891  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.If.beam
      748  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.String.beam
     1045  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Bc.beam
      880  11-02-2017 10:20   Elixir.BeamToExAst.Translate.$ProtocolEx_description$.beam
      992  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Try.beam
      682  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Nil.beam
     1011  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Match.beam
      723  11-02-2017 10:20   Elixir.BeamToExAst.Translate.BeamToExAst.Integer.beam

It looks like they are in the escript too?

olafura commented

Yes the files are there but it doesn't seem to pick anyone of them up I'll try digging some more

OvermindDL1 commented

Yes the files are there but it doesn't seem to pick anyone of them up I'll try digging some more

Hmm, if not then that sounds like a root elixir bug I'd say, because if the beam files 'are' there, nothing I can go about how they are loaded. ^.^;

I'm very curious in a full reproduceable example, especially if it is an elixir bug so I can show that straight.

olafura commented

I can call the functions directly so they are there and available.

So there is an error in how you handle params which might help us. When I include an underscore param like _opts in BeamToExAst.Translate I get an warning:

warning: the underscored variable "_opts" is used after being set. A leading underscore indicates that the value of the variable should be ignored. If this is intended please rename the variable to remove the underscore
  lib/beam_to_ex_ast/translate.ex:4

For each protocol, but when I run ./beam_to_ex --erl erlang_file.erl instead of the workaround mix run run.exs --erl erlang_file.erl I don't get the warnings.

You might be right that it might be an elixir bug, I'll try making a pure protocol example to see if it hits that also and mess around a bit more to see what in the mix escript.build might be causing this.

OvermindDL1 commented

So there is an error in how you handle params which might help us. When I include an underscore param like _opts in BeamToExAst.Translate I get an warning:

By putting an underscore param in the protocol definition itself you are stating that Absolutely No Implementation Will Use This, so you will not want to do that in the protocol definition itself if it is ever useful in an implementation. :-)

For each protocol, but when I run ./beam_to_ex --erl erlang_file.erl

Everything is already compiled up into this escript before that, and those warnings only happen at compile-time.

You might be right that it might be an elixir bug, I'll try making a pure protocol example to see if it hits that also and mess around a bit more to see what in the mix escript.build might be causing this.

I don't see anything wrong in the generated escript here so I cannot say. >.>

Also for note, seems like it works for some constructs as-is. :-)

╰─➤  cat test_file.erl 
-module(test_file).
-export([add/2]).
add(A, B) -> A + B.

╰─➤  ./beam_to_ex --erl test_file.erl
defmodule TestFile do
  def({:add, [line: 3], {:=, [], [{:_untranslated, [], nil}, "A\n\nB"]}}) do
    _untranslated = "A + B"
  end
end
OvermindDL1 commented

I should probably not cause warnings for underscored variables though, easy to fix, hmm...

olafura commented

The output above means it only used the default catch all function, if I don't have that I just get a match error.
The output from mix run run.exs --erl test_file.erl is:

defmodule TestFile do
  def(add(A, B)) do
    A + B
  end
end

I look at it better when later today ;) Thanks for helping me debug this and opening the other issue ;)

OvermindDL1 commented

The output from mix run run.exs --erl test_file.erl is:

Ooo, that makes it sound like that the escript.build command does not call the compilers. I wonder if it would fail on other compilers too, hmm...

OvermindDL1 commented

It's a bug with elixir, specifically with escript.building. ^.^

I submitted a bug report to:
elixir-lang/elixir#7000

olafura commented

Cool @OvermindDL1 thanks for finding the bug, I had looked at the same code but I couldn't find the problem ;)

OvermindDL1 commented

So come to find out Module.create does different things in different compiling contexts and it was entirely undocumented and surprising behaviour... >.<

I'll need to make an update. Blah...

OvermindDL1 commented

New version out, fixed the beam file outputting in different compiling contexts. >.<

Just run mix deps.update protocol_ex (or better yet, update the mix version to take a minimum of 0.3.12) and this now works:

╰─➤  mix escript.build
...
╰─➤  ./beam_to_ex --erl test_file.erl
defmodule TestFile do
  def(add(A, B)) do
    A + B
  end
end

And closing. Please report any more bugs! This issue fixed a different bug in elixir and got a lot of documentation going in elixir itself. :-)

olafura commented

@OvermindDL1 Thanks for all the help ;)

  • 👍1