googleapis/elixir-google-api

STOP GENERATING THOUSANDS OF MODULES!!!

edescourtis opened this issue · 7 comments

edescourtis commented

I know the use of meta-programming is fun. But the BEAM doesn't handle loading thousands of modules very well. We have several Google libraries in our deployment Elixir and just loading Google API libraries takes over a minute!!! OVER A MINUTE for a few APIs!!!!

iex -S mix run --no-start
fn ->
  times =
    Application.loaded_applications()
    |> Enum.map(fn {app, _desc, _vsn} ->
      modules = Application.spec(app, :modules)
      {time, _result} = :timer.tc(fn -> Enum.each(modules, &Code.ensure_loaded!/1) end)
      {app, System.convert_time_unit(time, :microsecond, :millisecond)}
    end)
    |> Enum.sort_by(&elem(&1, 1), :asc)

  Enum.each(times, &IO.inspect/1)
  times |> Enum.map(&elem(&1, 1)) |> Enum.sum() |> IO.inspect()
end.()
# {application, time_in_ms}
...
{:google_api_you_tube, 6336}
{:google_api_dfa_reporting, 11710}
{:google_api_vision, 15091}
{:google_api_content, 29394}
lpgauth commented

Is there a plan to fix this? I'm thinking I'm just going to vendor the modules I need instead of including the full library but it feels dirty.

lpil commented

I ended up ditching this library in favour of directly working with the API I needed, it worked much better.

  • 👍3
lawik commented

I experimented with this. Kind of a thought-stream as I went. But not generating a module per Model/schema seems like good bang for the buck: https://fosstodon.org/@lawik/111306047662675335

(github branch ref for ease: https://github.com/lawik/elixir-google-api/tree/lightweight-generator)

Instead of modules and structs it will generate a bunch of functions and it will provide typespecs for the non-struct maps. So you still get t docs in iex and can know what type goes where.

For a fairly small API with a bunch of Models (Chat API V1) it went from 101 modules to 5.

  • 🎉2
petermueller commented

I've just always brought in only the sub-repos necessary (e.g. google_api_storage only results in GoogleApi.Gax and GoogleApi.Storage), or is there a cost I'm not noticing?

I can definitely see not wanting "everything" if you're pulling in the "root" repo though. That could be annoying. This whole repo at large feels abandoned though.

lawik commented

If I generate just the AiPlatform endpoints of Google which would be one of these deps. That is about a thousand modules in schemas/models. Most of which I don't use or need. Thousand modules takes a bit to compile.

petermueller commented

Ah dang, that stinks

tsloughter commented

If mix doesn't support explicitly include/excluding modules it probably should and would be a good feature to suggest in its issues.

This made me realize we should have include support in rebar3/relx :). reltool supports explicit include but in rebar3/relx we only support exclude which probably isn't good enough in this situation since you'd have to list so many.

  • 👍1