`mix escript.build` does not use the generated beam files

Question

`mix escript.build` does not use the generated beam files

OvermindDL1 opened this issue 8 years ago · 37 comments

Tested this in both 1.5.2 as well as 1.6.0-dev.

Minimal reproduceable project:

╰─➤  cat lib/escript_testing.ex 
defmodule EscriptTesting do
  def main(args) do
    IO.inspect(args, label: :Testing)
  end
end
╰─➤  cat mix.exs 
defmodule EscriptTesting.MixProject do
  use Mix.Project

  def project do
    [
      app: :escript_testing,
      version: "0.1.0",
      elixir: "~> 1.6-dev",
      start_permanent: Mix.env() == :prod,
      deps: deps(),
      escript: [main_module: EscriptTesting],
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"},
    ]
  end
end
─➤  mix compile
Compiling 1 file (.ex)
Generated escript_testing app
╰─➤  mix escript.build --no-compile
** (Mix) Could not generate escript, module Elixir.EscriptTesting defined as :main_module could not be loaded
╰─➤  mix escript.build                                                                                                          1 ↵
Generated escript escript_testing with MIX_ENV=dev
╰─➤  mix escript.build --no-compile
** (Mix) Could not generate escript, module Elixir.EscriptTesting defined as :main_module could not be loaded

Related code at:
https://github.com/elixir-lang/elixir/blob/7c8c26652fdf82036bfd57e7f7364148cbb3f84a/lib/mix/lib/mix/tasks/escript.build.ex#L132-134

    if Keyword.get(opts, :compile, true) do
      Mix.Task.run(:compile, args)
    end

However this also errors out if using a custom compiler, like via this part of a mix.exs file:

  def project do
    [app: :beam_to_ex,
     version: "0.0.1",
     elixir: "~> 1.3-dev",
     compilers: Mix.compilers ++ [:protocol_ex], # A custom compiler is added
     build_embedded: Mix.env == :prod,
     start_permanent: Mix.env == :prod,
     escript: [main_module: BeamToEx],
     deps: deps]
  end

The code generated via this compiler appears to be cached somehow 'inside' the compiling step instead of using the newly generated *.beam files that are actually output.

Regardless of the method, escript.build is not using the actual beam files on disk, and one of its listed options, the --no-compile fails to function even with trivial projects because of this.

I'm not certain of the fix at this moment, not looked deep enough into it yet.

Answer 1 · 2017-11-02T20:50:17.000Z

Looking like it might be two distinct bugs maybe?

The --no-compile fails because it is trying to see if a module is loaded that has not been loaded yet (unsure if because not loaded the path with those beam files or not yet).
The extra compiler's beam files are not being included because... I have no clue (technically what it does is it replaces an existing beam file, specifically the non-consolidated version with a consolidated version, and although the consolidated version ends up on the permanent storage / drive, the escript.build is somehow grabbing the original non-consolidated version, so... whaa? o.O)...

It could still be related, unsure yet...

Answer 2 · 2017-11-02T20:52:34.000Z

For note, running mix run.exs ... works properly with the consolidated file and all, it is only the escript.build is not including the consolidated beam file...

Answer 3 · 2017-11-02T20:59:30.000Z

THe --no-compile is a separate and easy fix. We need to make sure we call loadpaths if we are not compiling. The other one doesn't make sense if we look at the code. We just read the beams from disk.

Answer 4 · 2017-11-02T21:07:36.000Z

The other one doesn't make sense if we look at the code. We just read the beams from disk.

Yeah that is what I'm confused about, I'm wondering if there is a race condition between when erlang's file manager closes the beam file and the escript.build process reads back in the file, it is what I'm testing right now actually.

For note, just removing the test if the main file is loaded in memory does not fix the --no-compile issue as running the compiled escript causes things like Could not start application logger: could not find application file: logger.app to appear, so yeah bringing in elixir's normal loadpaths seems required as well...

Answer 5 · 2017-11-02T21:11:53.000Z

Yeah, even adding a delay of 2 seconds right after the Mix.Task.run(:compile, args) call is not doing it.

Running the escript is giving substantially different results than just running run:

╰─➤  cat test_file.erl                                                                                                              
-module(test_file).
-export([add/2]).
add(A, B) -> A + B.
─➤  mix run run.exs --erl test_file.erl
defmodule TestFile do
  def(add(A, B)) do
    A + B
  end
end
╰─➤  ./beam_to_ex --erl test_file.erl                                                                                               
defmodule TestFile do
  def({:add, [line: 3], {:=, [], [{:_untranslated, [], nil}, "A\n\nB"]}}) do
    _untranslated = "A + B"
  end
end

That above project is the one that brought this to my attention at (you can git clone and test it if you wish): OvermindDL1/protocol_ex#4

I'd love to know if it is my fault, but I've no clue what I'm doing wrong...

Answer 6 · 2017-11-02T21:17:37.000Z

Yeah that is what I'm confused about, I'm wondering if there is a race condition between when erlang's file manager closes the beam file and the escript.build process reads back in the file, it is what I'm testing right now actually.

That should not matter. We straight read the files from disk, we don't care about what the loader is doing. I have just built an small example that has a compiler that writes a corrupted file to ebin and the corrupted file is being picked up. Can you make sure that your files are being written to disk? Note that Elixir protocol consolidation runs after your task and it cleans up the consolidation path in certain occasions. Since protocol_ex is only about the current project (that's why I wouldn't call them protocols as they can't be implemented at any time but that's another story) I would recommend you to write directly to the ebin path.

For note, just removing the test if the main file is loaded in memory does not fix th

Yes, we need to load all paths.

Answer 7 · 2017-11-02T21:19:22.000Z

Yeah comparing the beam files between the escript/zip and the actual beam files show matches except for that one consolidated file:

╰─➤  ls -lah Elixir.BeamToExAst.Translate.beam ../_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam
-rw-rw-r--. 1 2.3K Nov  2 15:06 ../_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam
-rw-rw-r--. 1 1.1K Nov  2 15:10 Elixir.BeamToExAst.Translate.beam

The first listed is the local filesystem file (at 2.3K), the second is the non-consolidated file (at 1.1K), so it is definitely not grabbing that same file that is on disk... o.O

Note that Elixir protocol consolidation runs after your task and it cleans up the consolidation path in certain occasions.

Not elixir's consolidation but rather Mix.Task.Compiler.ProtocolEx's compiler (which just calls consolidate_all()).

Note that Elixir protocol consolidation runs after your task and it cleans up the consolidation path in certain occasions. Since protocol_ex is only about the current project (that's why I wouldn't call them protocols as the any time extensibility but that's another story) I would recommend you to write directly to the ebin path.

I know that about the elixir consolidator, and that is why I do write to the ebin (as shown at the top of this post), I do not touch the consolidated path at all because of that. ^.^

Answer 8 · 2017-11-02T21:21:24.000Z

Interestingly the beam inside the zip file has a later date than the one on disk, and yet it is the non-consolidated version, so, wha...? o.O

Either that or the timestamps are not being stored properly in the zip/escript.

Answer 9 · 2017-11-02T21:22:18.000Z

I know that about the elixir consolidator, and that is why I do write to the ebin (as shown at the top of this post), I do not touch the consolidated path at all because of that. ^.^

AFAIK your code is using consolidation path: https://github.com/OvermindDL1/protocol_ex/blob/master/lib/mix/tasks/compile.protocol_ex.ex#L10

Answer 10 · 2017-11-02T21:25:04.000Z

Uh, right I need to remove that code...

That is an unused option back from before I found out the consolidated directory dies on occasion. ^.^

If you see: https://github.com/OvermindDL1/protocol_ex/blob/master/lib/protocol_ex.ex#L253
Currently the :output option is not at all used, I need to update the mix task (EDIT: Done, removed that dead code)... ^.^;

But no, that is not the issue. :-)

Answer 11 · 2017-11-02T21:30:34.000Z

But no, that is not the issue. :-)

I see, thanks. Ok, let me know if you have more information or if you can reproduce the issue on a smaller scale. Writing dirty beam files in a simple compiler is working here.

EDIT: the --no-compile issue is already fixed in master.

Answer 12 · 2017-11-02T21:31:18.000Z

For note, here is how the consolidated module is actually created (removing the 'consolidated' directory simplified it a lot):
https://github.com/OvermindDL1/protocol_ex/blob/master/lib/protocol_ex.ex#L382

      Code.compiler_options(ignore_module_conflict: true)
      Module.create(proto_name, impl_quoted, spec.location)
      Code.compiler_options(ignore_module_conflict: false)

Answer 13 · 2017-11-02T21:32:51.000Z

Also note: This specific case of that user actually has the writing of the replaced beam file happen to a dependency's beam file, not one of the project itself, though I'm unsure if that would make a difference based on what I've been reading of this code...

Answer 14 · 2017-11-02T21:38:31.000Z

Also note: This specific case of that user actually has the writing of the replaced beam file happen to a dependency's beam file, not one of the project itself, though I'm unsure if that would make a difference based on what I've been reading of this code...

It does because Mix/Elixir/OTP provide no guarantees over which one will be loaded. If you have two beam files of the same module, you are asking for trouble.

Answer 15 · 2017-11-02T21:40:28.000Z

It does because Mix/Elixir/OTP provide no guarantees over which one will be loaded. If you have two beam files of the same module, you are asking for trouble.

Should it not be overwriting the same file though?

Answer 16 · 2017-11-02T21:41:37.000Z

Only these are shown for that Translate beam:

╰─➤  find _build -iname '*Translate*'
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.$ProtocolEx_description$.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Try.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Atom.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Record.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.String.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Remote.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Float.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.RecordField.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Match.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Lc.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Integer.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Call.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Block.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Map.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Cons.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.BGenerate.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.BinElement.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.List.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Fun.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Case.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Clauses.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Clause.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Bin.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Op1.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.If.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Op2.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Char.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Tuple.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Var.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Nil.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.MapFieldAssoc.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.MapFieldExact.beam
_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.BeamToExAst.Bc.beam

Thus it only exists in one location, _build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam specifically, which should not cause any conflicts?

Answer 17 · 2017-11-02T21:43:03.000Z

Should it not be overwriting the same file though?

It is but in an unspecified order.

Answer 18 · 2017-11-02T21:43:55.000Z

It is but in an unspecified order.

Shouldn't the order be specified by the fact that the consolidation compiler is ran 'after' the elixir compiler? Which matches the beam file on disk (the working consolidated one is larger than the escript one).

Answer 19 · 2017-11-02T21:45:09.000Z

Yes. I am talking especifically about the case where a dependnecy and a project both have the same .beam file.

Answer 20 · 2017-11-02T21:47:17.000Z

Yes. I am talking especifically about the case where a dependnecy and a project both have the same .beam file.

Oh that should definitely not happen. The dependency has the file, the parent project is only 'calling' into that module, it does not define it. The module itself has a shim that is compiled when the dependency is compiled (to allow the elixir compilation to proceed without warnings/errors) then that shim module is replaced by the consolidation compiler running. So the file is compiled once when the dependency's elixir pass is compiled, then it is compiled once more when the protocol_ex's consolidation pass is run after the elixir compilation pass is run.

Answer 21 · 2017-11-02T21:56:06.000Z

I see this as well:

─➤  strace -f -t -e trace=file mix escript.build 2>&1 | grep 'Translate.beam'                                                      
[pid 10406] 15:51:35 open("/home/<user>/tmp/beam_to_ex/_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam", O_RDONLY) = 11
[pid 10406] 15:51:35 open("/home/<user>/tmp/beam_to_ex/_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam", O_RDONLY) = 11
[pid 10406] 15:51:35 open("/home/<user>/tmp/beam_to_ex/_build/dev/lib/beam_to_ex/ebin/Elixir.BeamToExAst.Translate.beam", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 10407] 15:51:35 open("/home/<user>/tmp/beam_to_ex/_build/dev/lib/beam_to_ex_ast/ebin/Elixir.BeamToExAst.Translate.beam", O_RDONLY) = 11

Answer 22 · 2017-11-02T22:02:59.000Z

Can you please show me where you write the consolidated protocols to disk and using which path?

Answer 23 · 2017-11-02T22:05:59.000Z

Can you please show me where you write the consolidated protocols to disk and using which path?

Already did, it is the same place that I put in a prior message, ever since I took out the output path stuff I just write it via:
https://github.com/OvermindDL1/protocol_ex/blob/master/lib/protocol_ex.ex#L382

      Module.create(proto_name, impl_quoted, spec.location)

Where spec.location is the original location information of the original module file (of both file: ... and line: ...). That is the only place that I generate a module manually (I.E. not via macro's).

Answer 24 · 2017-11-02T22:06:39.000Z

Module.create does not write a beam to disk.

Answer 25 · 2017-11-02T22:08:55.000Z

Module.create does not write a beam to disk.

Hmm, it's been working thus far even in non-compiling cases or so I thought (maybe the compiler is called more often than I thought).

The documentation for it states that it is functionally equivalent to defmodule (which does write a beam to disk) and that it should be preferred when a quoted expression is handy. What else should I be using?

EDIT: Should I be wrapping it in a :defmodule call and running Code.compile_quoted instead?

Answer 26 · 2017-11-02T22:12:10.000Z

Hmm, it's been working thus far even in non-compiling cases or so I thought (maybe the compiler is called more often than I thought).

Your compiler is always called and it seems it always consolidates as it doesn't have the logic of avoiding doing the work if the work is already done. I would recommend you to add the proper tracking rules.

The documentation for it states that it is functionally equivalent to defmodule

defmodule doesn't write to disk either, try it in IEx. :) The writing to disk only happens if someone passes a file with defmodule to a compiler. You should get the result of Module.create (it returns the beam binary) and write the beam to the proper place.

Answer 27 · 2017-11-02T22:22:05.000Z

Your compiler is always called and it seems it always consolidates as it doesn't have the logic of avoiding doing the work if the work is already done. I would recommend you to add the proper tracking rules.

Yeah I've not bothered adding that yet as I've been testing things still, it's not 1.0.0+ after all. ^.^

You should get the result of Module.create (it returns the beam binary) and write the beam to the proper place.

Simple enough. At least I caught one (unrelated) bug. ^.^

Any chance of that information being added to the docs? This does not mention anything of the sort, nor what it returns, it only says it returns what defmodule does, but there is no link to defmodule, and if you do check that manually to here it says nothing about what is returned either (as well as it also does not mention that it itself is not writing out the file but rather that the elixir compiler parses out the file data to write it out instead). I think this was just a lack of documentation problem (why was it saying that it returns the same thing as something else when the something else does not document what it returns anyway? o.O). ^.^

Answer 28 · 2017-11-02T22:24:34.000Z

Any chance of that information being added to the docs?

Sure, please do send a PR!

Answer 29 · 2017-11-02T22:25:31.000Z

Actually, no worries, I am on it.

Answer 30 · 2017-11-02T22:25:40.000Z

Wait, this still does not make sense...

At here:
https://github.com/OvermindDL1/protocol_ex/blob/master/lib/protocol_ex.ex#L127

I am creating the shim file by calling consolidate with an empty implementation list, which is calling Module.create, and yet the beam file gets created and output?

Answer 31 · 2017-11-02T22:31:02.000Z

@OvermindDL1 the line you linked is called by defprotocolEx which is invoked by a file being compiled.

Answer 32 · 2017-11-02T22:31:33.000Z

You should have the consolidation function return a list of modules names and their byte codes and then you do the writing on the task.

Answer 33 · 2017-11-02T22:31:59.000Z

In addition I'm creating the $definition file via https://github.com/OvermindDL1/protocol_ex/blob/master/lib/protocol_ex.ex#L126 too:

    Module.create(desc_name, desc_body, Macro.Env.location(__CALLER__))

And that beam file gets created just fine, and that's the only time it is ever created.

@OvermindDL1 the line you linked is called by defprotocolEx which is invoked by a file being compiled.

So you are saying that Module.create does different things depending on how the file is compiled? That sounds... painfully surprising... How do I detect that? And why is it not documented? o.O

Answer 34 · 2017-11-02T22:38:16.000Z

So you are saying that Module.create does different things depending on how the file is compiled? That sounds... painfully surprising...

As said above, it is exactly the same as defmodule. It will write a beam file depending if the current file is being compiled or not.

Answer 35 · 2017-11-02T22:44:43.000Z

As said above, it is exactly the same as defmodule. It will write a beam file depending if the current file is being compiled or not.

Not documented in either though... ^.^;

Regardless, having identical code do very different things all based on when it is compiled is very surprising behaviour. For Elixir 2.0 (whenever) it should probably make it so something like defmodule always compiles a beam file and Module.create always returns the binary without compiling. Even in iex I'd still expect a beam file to be written out somewhere (perhaps something like _build/dev/iex or so). Doing different things with identical code that has identical inputs to the functions is very surprising...

It will write a beam file depending if the current file is being compiled or not.

How do I call that same path so it generates a beam file regardless. It would be far safer for me to do that then find the existing beam file and overwrite it, especially if paths start changing around or something...

Answer 36 · 2017-11-02T23:03:16.000Z

Thanks but we have no plans to change how this behaves. For example, tests use defmodule and we don't want your tests to write beam files to disk. The documentation has been pushed to master.

I believe this issue has been addressed, we can move any remaining discussion elsewhere.

Answer 37 · 2017-11-02T23:17:29.000Z

The documentation has been pushed to master.

Woot, yep I see those docs and that is perfect, it would have saved me a ton of issues tonight so hopefully it will others in the future too! :-)

term is the result of the last expression in quoted.

Huh, I assumed that was the last export, not the actual last evaluated value of the quote itself, hmm, I may be able to use that for things. Thanks for that tidbit of info too! :-)