[Question] Are classes in nested directories not auto-reloaded with `push_dir`?

[adrianthedev]

Backstory:

Avo is an engine distributed as a gem that uses configuration files in the parent app to build out an interface.

The directory structure is kind of like this:

app/
├─ avo/
│  ├─ actions/
│  │  ├─ make_admin.rb # Avo::Actions::MakeAdmin
│  ├─ resources/
│  │  ├─ user.rb # Avo::Resources::User
│  ├─ ...other Avo directories/
├─ controllers/
├─ ... other parent app Rails directories/

Following #250 I switched from manually adding each directory to the main autoloader to adding just the whole avo directory.

The issue

The issue I'm observing now is that the configuration files from the parent app are not auto-reloaded when the user updates them.

# changed files get auto-reloaded 👇
initializer "avo.autoload" do |app|
  entities = {
    'Avo::Actions': Rails.root.join('app', 'avo', 'actions'),
    'Avo::Resources': Rails.root.join('app', 'avo', 'resources'),
  }
  entities.each do |namespace, path|
    if File.directory? path.to_s
      Rails.autoloaders.main.push_dir path.to_s, namespace: namespace.constantize
    end
  end
end

# changed files do not get auto-reloaded 👇
initializer "avo.autoload" do |app|
  # This undoes Rails' previous nested directories behavior in the `app` dir.
  # More on this: https://github.com/fxn/zeitwerk/issues/250
  ActiveSupport::Dependencies.autoload_paths.delete("#{Rails.root}/app/avo")
  Rails.autoloaders.main.push_dir("#{Rails.root}/app/avo", namespace: Avo)
end

Because I'm using Rails.autoloaders.main.push_dir on both statements, this makes me think that push_dir auto-reloads only the root directory.

Is this valid behavior, or am I reading things wrong?

Thank you,
Adrian

Update: When I'm speaking about changes and not auto-reloaded, I mean, I'm changing something in Avo::Resources::User (ex: add/remove a field), and Avo won't see that change until after the Rails server is restarted.

[fxn]

Hey @adrianthedev!

So, you are doing things correctly, only Rails < 7.1 does not have builtin support for custom root directories and assumes ActiveSupport::Dependencies.autoload_paths are the autoload paths. In particular, it configures the file system watcher to only monitor those. Since the directory is (correctly) being deleted from that collection, we need to also tell Rails to watch it.

Could you please add

app.config.watchable_dirs["#{Rails.root}/app/avo"] = [:rb]

to that initializer?

This is all very ugly, a consequence of the fact that we are bending a bit what Rails < 7.1 supports during this transition period. But won't be necessary in Rails 7.1 (I have the WIP in a branch, not yet in main).

[adrianthedev]

🙌 I confirm that it fixed my issue.

Got it! That makes perfect sense. I overlooked the fact that we are deleting it from the autoload paths. The fact that they are also removed from watchable dirs was hidden from me.

This is something that was very difficult for me to grasp with zeitwerk/autoloading/reloading.
How does autoloading work in Ruby?
When does zeitwerk come into play?
Who does the directory watching and refreshing? Is is Rails, is it zeitwerk?
Should I manually reload/eager-load the code?
Are the there performance implications?
Why do people still use gems like listen to manually reload it (referring to this tweet)?

These questions popped into my head at different times in my learning path. You don't have to answer these questions, I just want to give you a peek into my though process (and I know you may want to speak at a conference about this 😅).

Thank you again for your help and most importantly, for explaining how things work.

[fxn]

@adrianthedev that is excellent feedback. I do need to work on documenting some of that, those wondering about what is happening should have answers in the docs. The new talk I am preparing on how Zeitwerk works that will present in Bucharest goes also in that direction.

Let me answer your questions anyway.

How does autoloading work in Ruby?

Let's consider

module M
  autoload :X, "x" # (1)
end

M::X # (2)

In (2), M is defined, but M::X is not. When executing (2), the constant resolution algorithm asks M: "do you have a constant called :X? M has an autoload for :X and performs a require "x", where "x" is the second argument in (1). If all goes well, that file does define M::X, and execution resumes.

When does zeitwerk come into play?

In a project managed by Zeitwerk, once you execute its setup method, the loader scans the file system, camelizes file names, and defines autoloads as in (1) on your behalf. So, Ruby is the one autoloading. This is key, Zeitwerk does not perform the ultimate, real code autoload, it is Ruby. And this is a key idea in the project, because as we saw before, autoloading is builtin in Ruby, and it is integrated with the resolution algorithms. That is why autoloading with Zeitwerk matches Ruby semantics.

Who does the directory watching and refreshing? Is is Rails, is it zeitwerk?

Zeitwerk has API to reload, but it has to be called by the parent project. Zeitwerk does not watch the file system. In the case of Rails applications, Rails is the one creating listen instances.

There are some configuration options for reloading in Rails, but the bottom line is that when Rails determines a reload has to be executed, then it invokes the reload method of the main autoloader.

Should I manually reload/eager-load the code?

In Rails, eager loading is determined by config.eager_load in the environments configuration. Reloading is driven by Rails except in the console, where you manually need to execute reload! if you want to reload (because reloading behind the scenes while you are in a session would be confusing).

Are the there performance implications?

Indeed, Zeitwerk works only with absolute paths. That is, the second argument in (1) above is always an absolute path. Therefore, there is no $LOAD_PATH lookups in projects managed by Zeitwerk. Also, you don't have the same require repeated in several places forcing Ruby to verify the file was loaded and returning false, there is only 1 require call per file (1), no more.

On the other hand, if you do not eager load, Zeitwerk does not scan the whole project tree, it only scans the first level, and descends in the used subtrees on demand.

Why do people still use gems like listen to manually reload it (referring to this tweet)?

Because if your web framework does not have an integration like Rails does, it is not watching any files. You need to understand when does a reload can be executed (e.g., in between requests), and program that yourself.

In ordinary gems there is no reload, because there is no process running. If you are developing a library and change something, you run the test suite.

Please feel free to ask more!

[adrianthedev]

I definitely understand things better now. Thank you for answering these questions 🙌