tzdata crash at startup in escript

[sdanzan]

I have an application packaged as a standalone executable (generated with mix escript.build) that uses Timex to parse dates.

When the application is run from the command line, it crashes in first call to Timex.DateFormat.parse with the "classic" stack trace:

10:45:31.085 [error] Task #PID<0.83.0> started from #PID<0.81.0> terminating
** (ArgumentError) argument error
(stdlib) :ets.lookup(:tzdata_current_release, :release_version)
(tzdata) lib/tzdata/release_reader.ex:41: Tzdata.ReleaseReader.current_release_from_table/0
(tzdata) lib/tzdata/release_reader.ex:13: Tzdata.ReleaseReader.simple_lookup/1
(tzdata) lib/tzdata/release_reader.ex:7: Tzdata.ReleaseReader.zone_and_link_list/0
(tzdata) lib/tzdata.ex:61: Tzdata.zone_exists?/1
lib/timezone/timezone.ex:91: Timex.Timezone.get/2
lib/parse/datetime/parser.ex:213: Timex.Parse.DateTime.Parser.update_date/4
lib/parse/datetime/parser.ex:121: Timex.Parse.DateTime.Parser.apply_directives/3

When I call the main function of my application directly from within iex -S mix with the same arguments, it does not crash. In both case the tzdata application is loaded and started (I checked by printing inspect(Application.loaded_applications) and inspect(Application.started_applications) at the beginning of my main function.

I use Timex 0.19.5 and tzdata 0.5.3.

[triptec]

I've got the same error =(

[sdanzan]

OK, escript builds do not support well data embeded in priv (which tzdata requires). However when trying to work around that I found a very strange behaviour: tzdata is present in the list of started applications, but when I call Application.app_dir(:tzdata, ...) I get an application not started error. This easily reproduceable with an application that only Enum.each all started applications and call Application.app_dir

Anyway, I guess to properly embed Timex/Tzdata in standalone executables, I´ll have to try some Code.prepend_path magic.

[bitwalker]

@sdanzan Actually escripts do support reading data from their embedded archive (via erl_prim_loader). However the bigger problem is that tzdata is using ets:file2tab to load timezone information from an ETS dump, and the underlying implementation is using functionality which requires the file to be accessible directly, there doesn't appear to be a way to load the file as a binary and call a different function to read it.

@lau Is there a relatively straightforward fallback solution we can use with tzdata for these scenarios? One thought I had is that we could dump the ETS table to a list via tab2list, then write that to a file (as terms). We can then determine whether to load that file via erl_prim_loader or a simple File.read based on whether we can locate the tzdata priv directory. Then load the file and repopulate the table by iterating over the loaded list. It would be slower, but would work in both cases. Alternatively, we could dump both to a list and the binary (via tab2file), and choose the file we need based on the current execution environment. In either case, I think we definitely want tzdata to work for the escript use case. Thoughts?

[lau]

A feature that I am considering implementing soon is to be able to configure (using the elixir config system) an alternative dir location for the files that are currently in the priv dir. I don't know if it would be a practical solution for most people doing escripts, but I'm guessing it could solve the problem.

[Korbin73]

I'm also seeing this error but in a slightly different context. I'm not sure if this will help, but I have configured my .iex.exs file to have the following function that uses date bindings

call_service_with_cache = fn -> 
    params = {credentials, App.get_yesterdays_date_format, App.get_todays_date_format}
    garden_data = ApiCache.call_api(params, fn credentials, date_from, date_to -> 
        Api.get_garden_data(credentials, date_from, date_to)
    end)
end

I see this error when trying to format the date to a string.

[bitwalker]

@lau Yeah the same problem will still exist for escript builds, since the only way to read files from the escript is via erl_prim_loader, and storing the files externally isn't practical for an escript since the idea is to have a portable executable. If tzdata can support loading the data into ETS via an alternative means other than :ets.file2tab, then that would be the ideal fallback for when we have to read from the escript archive. The problem as I mentioned is that supporting that case requires one more storage format for the ETS table contents (or just dumping the ETS file to a term and writing that to a file and using that as the sole storage format). I looked into what :ets.file2tab is doing under the covers to see if there is a way to operate on the raw binary content, but nothing is exposed, and it's using disk_log under the covers anyway which appears to require a file on disk which is accessible via normal file APIs. So we'd either have to emulate what it's doing and parse the ETS file manually (not hard, but not ideal), or provide two ways to load the ETS data (from an ETS file, or from a list of terms)

Since datetime parsing/formatting/etc. is probably not an uncommon scenario for some escript use cases, I think it's worth the time to figure out a good solution.

[bitwalker]

@Korbin73 In your .iex.exs, add above that code snippet:

Application.load(:tzdata)
:ok = Application.ensure_started(:tzdata)

That should fix your particular issue.

[Korbin73]

Excellent! That worked. Thanks @bitwalker

[lau]

@bitwalker I see. Having two different file formats might be messy, but going down that route might be a good idea. Maybe with the alternative file format as a separate package if possible.

Using tzdata ~> 0.1.8 would probably work as all the data are just included at compile time. Could you allow that in the timex mix.exs? Then people could specify ~> 0.1.8 in their mix.exs. Are escripts compiled before distribution? Otherwise the RAM requirements for compilation of 0.1.x could be an issue if you want to run the escript on e.g. a raspberry pi.

[bitwalker]

@lau I've changed the requirements in timex to allow using tzdata 0.1.8 for now, until I can get around to my PR to Elixir's mix escript.build task, then I can tackle modifying tzdata and sending you a PR.

@sdanzan You can put the following in your mix.exs to use the above version of tzdata inside your escript instead of the current one:

def deps do
  [..., {:tzdata, "== 0.1.8", override: true}]
end

I'll update this thread when we have a longer term solution in place.

[sdanzan]

Yes, it works, thanks!

[markburns]

I had this issue, but had a little trouble finding this github issue via google, so will add the stacktrace to make this easier to find for others

Could not start application tzdata: exited in: Tzdata.App.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, {:shutdown, {:failed_to_start_child, Tzdata.EtsHolder, {%ArgumentError{message: "unknown application: :tzdata"}, [{Application, :app_dir, 1, [file: 'lib/application.ex', line: 324]}, {Application, :app_dir, 2, [file: 'lib/application.ex', line: 333]}, {Tzdata.EtsHolder, :release_dir, 0, [file: 'lib/tzdata/ets_holder.ex', line: 86]}, {Tzdata.EtsHolder, :make_sure_a_release_dir_exists, 0, [file: 'lib/tzdata/ets_holder.ex', line: 70]}, {Tzdata.EtsHolder, :make_sure_a_release_is_on_file, 0, [file: 'lib/tzdata/ets_holder.ex', line: 64]}, {Tzdata.EtsHolder, :init, 1, [file: 'lib/tzdata/ets_holder.ex', line: 10]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 328]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]}}}}
            (tzdata) lib/tzdata/tzdata_app.ex:15: Tzdata.App.start/2
            (kernel) application_master.erl:273: :application_master.start_it_old/4

[aborn]

It works tzdata 0.1.8, Thanks @bitwalker

[lau]

I would use "~> 0.1.8" instead of "== 0.1.8". Because when the next version of the timezone data comes out, you would want 0.1.9 instead of 0.1.8. And ~> 0.1.8 allows "0.1.9", "0.1.10" and so on.

[bitwalker]

Changed :)

[rauluranga]

Hi to all!
I've got the same error, downgrading tzdata to 0.1.8 fix the build problem.

[bitwalker]

This is documented in 2.0, and I'm going to look in to a PR for a solution to fix this in more recent versions as soon as I can.

[mneumark]

@bitwalker Any word on a fix for this issue? Downgrading tzdata fixes the problem but its not the best solution in the world. Any reason the issue is closed if its not really resolved?

[bitwalker]

The fundamental issue is that it's not possible to load ETS tables from files in an escript. The tzdata package relies on this functionality in all versions > 0.1.x. Basically, tzdata needs to be refactored such that it can operate under this constraint, and unfortunately I haven't had time to take a crack at a PR. It's closed because there is a workaround, even though it's not ideal, and it's really an issue that should be tracked against tzdata, not timex.