getsentry/sentry-elixir

Compile issues

madsheep opened this issue · 15 comments

madsheep commented

Environment

Description

Every now and then sentry compilation will fail with following error:

==> sentry
Compiling 12 files (.ex)

== Compilation error on file lib/sentry/http_client.ex ==
** (MatchError) no match of right hand side value: {:error, :enoent}
    (elixir) lib/kernel/parallel_compiler.ex:117: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

could not compile dependency :sentry, "mix compile" failed. You can recompile this dependency with "mix deps.compile sentry", update it with "mix deps.update sentry" or clean it with "mix deps.clean sentry"

On our CI server (circleci) this happens more often then not recently

We see this problem outside our ci server on dev machines as well. It also happened in two different, separate applications (one phoenix, other pure otp) and across different versions of otp and elixir (listed above).

Expected Behavior

It should compile :) the builds are driving one of our devs nuts.

Actual Behavior

It does not compile sometimes. We fail to see any pattern to this.

Steps to Reproduce

The easiest way for me to get this to happen was:
0) On this environment (macbook pro 2016, 8gb, 2Ghz cpu, Sierra)

> elixir -v
Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]
Elixir 1.4.5
  1. Created fresh mix project
  2. Added following deps (copied from one of the apps that had this issue before):
[
  {:cowboy, "~> 1.0.0"},
  {:plug, "~> 1.0"},
  {:mq_tools, github: "Recruitee/mq_tools"},
  {:tesla, github: "teamon/tesla", branch: "master"},
  {:postgrex, ">= 0.0.0"},
  {:ecto, "~> 2.1"},
  {:timex, "3.1.3"},
  {:redix, "~> 0.6.0"},
  {:sentry, github: "getsentry/sentry-elixir"}
]
  1. Run this:
for ((n=0;n<100;n++)); do mix deps.clean --all && mix deps.get && mix deps.compile; done
  1. Typed in 'Compilation error' in search bar in iTerm so once the phrase pops up it gets highlighted and catches my eye
  2. Browsed 9gag for a while and watched Deadpool 2 trailer on second screen - https://www.youtube.com/watch?v=8-Cjsnq8kVU - amazing
  3. after 5-10 minutes the above error pops up highlighted in yellow on the other screen - this time on util file:
== Compilation error on file lib/sentry/util.ex ==
** (MatchError) no match of right hand side value: {:error, :enoent}
    (elixir) lib/kernel/parallel_compiler.ex:117: anonymous fn/4 in Kernel.ParallelCompiler.spawn_compilers/1

could not compile dependency :sentry, "mix compile" failed. You can recompile this dependency with "mix deps.compile sentry", update it with "mix deps.update sentry" or clean it with "mix deps.clean sentry"

I'm not 100% sure if this is caused by sentry itself or maybe one of the other deps - however it's only sentry that has those compilation issues.

Let me know what you think.

mitchellhenke commented

Thank you for opening this. Apologies for the difficulties you're having, this is indeed odd. I wasn't able to reproduce with those dependencies in a new project, and there isn't anything out of the ordinary going on in https://github.com/getsentry/sentry-elixir/blob/master/lib/sentry/util.ex with macros, etc.

When you have a chance, can you investigate:

  • Does it compile if you remove Sentry?
  • Does it compile on the latest Elixir (1.5.2) and Erlang (20.1)?

I would suspect Sentry has a bug causing this before suspecting Elixir/Erlang, but answers to those questions will help narrow it down.

madsheep commented

Hi again,
Sorry for the radio silence - I've looked again into the issue over the weekend to confirm/reconfirm it. While at it, I've accidentally solved it :) It seems the root cause was firmly located between my ears.

The only way to actually get this issue was to use elixir version compiled on different OTP then the current one you are using. In our case we've managed to achieve this feat by using asdf version manager with elixir and erlang plugin. One can very easily miss the warning they post here:
https://github.com/asdf-vm/asdf-elixir

These precompiled packages are built against every officially supported OTP version, however if you only specify the elixir version, like 1.4.5, the downloaded binaries will be those compiled against the oldest OTP release supported by that version.

If you would like to use precompiled binaries built with a more recent OTP, you can append -otp-${OTP_VERSION} to any installable version that can be given to asdf-elixir.

so do something like in your .tools-version file:

elixir 1.4.5
erlang 20.1

and you will sometimes get the compile errors.
switch that to:

elixir 1.4.5-otp-20.1
erlang 20.1

or

elixir 1.4.5-otp-19
erlang 19.3

and you are good :)

Apologies for wasting your time :) It still baffles me that only sentry had those compilation issues but I think that's just a coincidence.

mitchellhenke commented

@madsheep Thanks for following up, I am glad to hear all was solved, and I've learned to recognize this error.

Thank you!

michaelstalker commented

Sorry to comment on a months-old issue, but I was just able to replicate this error without mismatched OTP versions. Here is the error I got:

== Compilation error in file lib/sentry/plug.ex ==
** (MatchError) no match of right hand side value: {:error, :enoent}
    (elixir) lib/kernel/parallel_compiler.ex:198: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6
could not compile dependency :sentry, "mix compile" failed. You can recompile this dependency with "mix deps.compile sentry", update it with "mix deps.update sentry" or clean it with "mix deps.clean sentry"

I compiled Sentry in this Elixir 1.6.1 Docker image. The image uses Erlang/OTP 20.2.2. Since I'm using that image to compile both Elixir and Sentry, both should be compiled with OTP 20.2.2, as well.

When I tried rebuilding the app, everything compiled without errors.

Enviroment

Elixir version: 1.6.1
Erlang/OTP version: 20.2.2
Sentry version: 5.0.0
Operating system: Debian GNU/Linux 8

mitchellhenke commented

@michaelstalker thanks for the update, hopefully I can get to the bottom of it. Unfortunately, I'm not overly familiar with Docker. Is there a good way for me to reproduce this on my machine?

michaelstalker commented

I'm not sure. To be honest, I've never reproduced it locally, either. I only get the error in Jenkins, and only intermittently. I know that's not much help 😬

michaelstalker commented

I've been looking into Elixir's parallel compiler. It looks like this line is failing to find a file:

:elixir_compiler.file(file, dest)

and is returning {:error, :enoent}. I'm not quite sure why, in the context of the ParallelCompiler module, that's throwing a MatchError. It's pattern matching the result of :elixir_compiler.file(file, dest) against _, which should always match.

EDIT: Actually, I'd bet it's never getting to the point where it tries to pattern match against _. It's failing to pattern-match on line 41 of the Elixir compiler's file/2 function:

{ok, Bin} = file:read_file(File)

It looks like Erlang's read_file/1 returns {error, ernoent} if a file does not exist. {ok, Bin} = {error, ernoent} would not match, and we'd get the MatchError I'm seeing in the build output:

(MatchError) no match of right hand side value: {:error, :enoent}
mitchellhenke commented

@michaelstalker Thanks for the effort on researching this. Has this happened with any other libraries that you've seen, or just Sentry?

michaelstalker commented

Just Sentry.

mitchellhenke commented

Hmm, I will see if I can find someone with more Elixir/Erlang compiler knowledge than myself, as I'm not sure what a solution would look like.

michaelstalker commented

I posted about the issue on the Elixir Forum, just in case it was a compiler issue. José Valim suggested running mix deps.compile with strace. I'm working on trying that now, but am running into some permissions issues in Docker.

josevalim commented

The only thing I could think of that would justify this behaviour happening for sentry is if you are calling a macro or compiling a file that changes directory.

mitchellhenke commented

@josevalim Thanks for following up! I don't have any file moving/copying in Sentry as I understand it, though we do some File.read! during compilation in some cases.

Would it be helpful to strace all of my remote builds on the chance I'm able to catch it? Any suggestions on tracking it down would be super.

jeregrine commented

Thanks to @potatosalad we know the cause of the issue and can temporarily make a fix until elixir-lang/elixir#7699 is closed out

mitchellhenke commented

This should be fixed with #285 by defaulting to not including dependencies in reports, so if anyone has issues with the compiler as described in elixir-lang/elixir#7699, please comment here.

I'm going to close this issue, and have opened a new issue (#286) to track the upstream Elixir issue and will be evaluating the options once it's fixed.