reopening 3rd party namspaces from an app

[modosc]

i've got the following scenario:

1. a foo gem using zeitwerk managing the Foo namespace:

require 'zeitwerk'
loader = Zeitwerk::Loader.for_gem
loader.setup # ready!

module Foo
end

loader.eager_load # optionally

2. a rails app using zeitwerk which wants to define Foo::Bar in lib/foo/bar.rb:

module Foo
  module Bar 
  end
end

when i pull in the foo gem in my app, everything works without require statements, but any reference in the app to Foo::Bar raises a NameError. if i move and rename Foo::Bar to Bar then everything works.

is there a workaround similar to this one but from the other side (eg, from a rails app to reopen said class/module)?

[fxn]

The lib directory of the Rails application is not managed by the loader of your gem. Does the Rails application issue a require "foo/bar"?

[modosc]

the rails application is also using zeitwerk so no explicit require is being used.

also we have this in application.rb:

    extra_load_paths = ["#{Rails.root}/lib"]
    config.autoload_paths += extra_load_paths
    config.eager_load_paths += extra_load_paths

[fxn]

OK, it is a similar situation to the one in the README.

The gem owns the namespace, and the gem does not reload (this is correct). You need to make sure the gem entry point is required early, for example by Bundler. That way, when the loader of the application finds lib/foo.rb, it knows Foo is already in memory and therefore defined by someone else, and will setup things to autoload Foo::Bar when needed. Also, on reload, it knows it has to reload Foo::Bar, but not Foo itself.

[modosc]

You need to make sure the gem entry point is required early, for example by Bundler.

we are already doing this, eg:

gem "foo",  require: "foo"

i'll try enabling Rails.autoloaders.log to see if there's any useful output.

[modosc]

also - i verified that explicitly requiring foo/bar in the rails app does resolve this but i was hoping to avoid that with some sort of config in the app (or gem if possible)

[fxn]

Is the application referring to Foo::Bar in the initializers?

[modosc]

Is the application referring to Foo::Bar in the initializers?

no.

[modosc]

with Rails.autoloaders.log! enabled i noticed that there is no reference to lib/foo/bar or Foo::Bar.

there are other similar references to similar cases, eg:

Zeitwerk@rails.main: the namespace Rack already exists, descending into lib/rack
Zeitwerk@rails.main: autoload set for Rack::Status, to be loaded from lib/rack/status.rb

[modosc]

(also sorry for all the hypotheticals, this is a private codebase so i'm having to transpose everything)

[fxn]

OK, now

bin/rails runner Rack::Status

raises?

[modosc]

bin/rails runner Rack::Status seems to work, i get the same debug output as above with the addition of:

Zeitwerk@rails.main: constant Rack::Status loaded from file lib/rack/status.rb

[fxn]

Excellent, the good news is that all you have in place seems correct. I suspect the application is accessing Rack::Status before the main autoloader is setup. Where is the NameError happening?

[modosc]

Where is the NameError happening?

right now, in any spec (or in the console) that references Foo::Bar, and also here:

bin/rails runner Foo::Bar
Please specify a valid ruby command or the path of a script to run.
Run 'rails runner -h' for help.

uninitialized constant Foo::Bar

Foo::Bar
   ^^^^^^^^^^

[fxn]

We'll find it.

Why does Foo::Bar fail and Rack::Status don't with bin/rails runner?

[modosc]

looks like we explicitly pull it in:

our config.ru:

require_relative "config/environment"
require_relative "lib/rack/status"

and also another rack example in application.rb:

    require Rails.root.join("lib/rack/force_cloudflare.rb")
    config.middleware.use(Rack::ForceCloudflare)

[modosc]

also, here's a non-rack example where we're not explicitly pulling anything in:

Zeitwerk@rails.main: the namespace Faraday already exists, descending into lib/faraday
Zeitwerk@rails.main: the namespace Faraday::Request already exists, descending into lib/faraday/request
Zeitwerk@rails.main: autoload set for Faraday::Request::FlatMultipart, to be loaded from lib/faraday/request/flat_multipart.rb

# this is lib/faraday/request/flat_multipart.rb
module Faraday
  class Request
    class FlatMultipart < Multipart
       # ...

[fxn]

Let me clarify my question above.

We had Rack::Status loading, but there was a Foo::Bar not loading. I guess you can't share what is Foo::Bar, but do you spot any difference? The key observation should be where are the constants referenced when the exception is raised.

[modosc]

the only difference i can see is that lib/rack/status is explicitly require'd in because it's used before zeitwerk is initialized (eg, in config.ru). if i require 'foo/bar' then everything works:

main:0> Foo::Bar
NameError: uninitialized constant Foo::Bar

Foo::Bar
   ^^^^^^^^^^
from (pry):1:in `<main>'
main:0> require 'foo/bar'
Zeitwerk@rails.main: autoload set for Foo::Bar, to be loaded from lib/foo/bar.rb
Zeitwerk@rails.main: constant Foo::Bar loaded from file  lib/foo/bar.rb
=> true
main:0> Foo::Bar
=> Foo::Bar

[modosc]

also i can try to setup a repo to reproduce this if you think it's worthwhile?

[fxn]

also i can try to setup a repo to reproduce this if you think it's worthwhile?

Oh, that would be the best way to debug it.

[modosc]

i started a repo and i can't reproduce this yet. at the same time i also pulled the foo gem into another project and ran into the exact same issue, so i'll keep digging on the repro 🤷🏽

[fxn]

In case it helps, if you issue

Zeitwerk::Loader.default_logger = method(:puts)

before the gems are required, you'll see traces from both the gem loader and the Rails loader. Maybe that helps you spot something.

[fxn]

Hey! I like to keep the issue tracker empty.

If you'd like to share code privately (fxn@hashref.com), or can think of any way to move forward, please count on me, I'd love to help. Otherwise, if the traces do not shed light, and you cannot reproduce in a way that can be shared, I'd finally close.

[modosc]

thanks @fxn - i got distracted by some other emergencies but i'll try the new trace suggestion on monday and update or close this.

[fxn]

Fantastic!

I suspect the data we are looking for is traces containing Foo from both loaders, with emphasis on the ones from the loader that is supposed to manage the missing Foo::Bar. (They should be tagged differently, and the tag is in the traces.)

[modosc]

i dug a bit more and couldn't come up with a definitive answer. since we've got a temporary workaround i'm going to close this out but if i can create a repro i'll ping and reopen/refile. thanks for your help.

[fxn]

@modosc If you undo the workaround, does

Object.const_source_location(:Foo)

point to the gem entry point as expected? It should, but since the situation is really weird, we cannot trust even our own shadow :).