Reloading of local gems

[davidmilo]

I have a rails 7.1.3 app which is using local gem. My folder structure is:

  my_rails_app/
  my_local_gem/

Reloading in development mode does not work.

I tried different advices
https://format-express.dev/blog/embed-a-gem-in-a-rails-project-and-enable-autoreload
https://stackoverflow.com/a/71001392
but none worked for me.

Any ideas how to achieve this? Is there a simple way to tell zeitwerk to reload some code from some gem paths?

[fxn]

Your gems are designed to be independent and have their own autoloaders? Or do you want them to be managed by the main autoloader of the parent application?

[davidmilo]

If I were to choose, i would say it is better that gems handle their own auto-loading so If I am only working on the gem, I do not have to rely on parent application autoloading setup.

I am open to both to be honest. Maybe you could provide some example how I could get it work in both cases?

[fxn]

The option in which gems can work independently is doable but it is a bit tricky.

On the other hand, the option in which they are written to be used by a parent Rails application is very easy if you make them engines, because engines are automatically integrated in autoloading/reloading/eager loading and file watching without you having to do anything.

Happy to help in any of the two cases.

[davidmilo]

If you could show me both examples, that would be very helpful

1. Local gem with loading maintained by rails app (but not as an engine)
2. Independent gem having it's own loading. (either using zeitwerk or manual file loading in gem would be fine)

[fxn]

The first one should be easy:

1. DO NOT put the gems in the Gemfile.
2. Add the full path of each gem's lib directory to config.autoload_paths and config.eager_load_paths in config/application.rb.

That is all. I have not tested it, but should work. If gems have a version.rb file that defines VERSION, you may need an inflection. The autoloading guide explains how to do this.

(2) is more detailed and I'll need some time.

[bkuhlmann]

Hey. 👋 I'm interested in this as well, especially the second approach (i.e. allowing an embedded gem to have it's own Zeitwerk loader) and then teaching the Rails application's Zeitwerk loader to leverage the gem's loader. My structure is slightly different where I'm wanting to use the Rail's lib folder to the gem in hopes that I'll eventually extract it into a proper gem dependency outside of the application:

my_rails_app             # The main application.
my_rails_app/lib/my_gem  # My gem with it's own loader.

[fxn]

@bkuhlmann I'd discourage you from doing that. The lib directory has Rails-specific folders and files, and that is mixed with something considered to be external at the same level. I'd recommend that you use config.autoload_lib (or copy its implementation if not in Rails 7.1). And if the code becomes extracted to a gem some day, then you make it a gem independent of the Rails application.

Zeitwerk does not support chaining loaders on purpose, I don't want to evolve the API in that direction because that is trickier than you guys may realize. Tricky in the application-logic sense, not in the implementation sense. Specially when there are cross-references at the file level.

@davidmilo how does your application and your gems refer to each other's constants? Are there shared mixins or superclasses or top-level cross-references?

[bkuhlmann]

OK, that's good to know, thanks. I ended up using basic require's in the gem implementation and using config.autoload_lib as you mentioned. Definitely miss being able to use Zeitwerk's gem loader for the gem instead of the manual requires but make sense.

[fxn]

@bkuhlmann oh, with config.autoload_lib you don't need any require (better delete them all). The Rails autoloader manages lib with that setting. If the day arrives in which you extract the gem for real, then just instantiate a loader using for_gem, configure any inflection rules if you need them, and done. The gem is ready because it worked without requires before.

[bkuhlmann]

True and normally that'd be fine. Apologies, but I didn't explain the full story. The situation is that I'm only using the following helpers for testing purposes:

my_rails_app             # Uses: `spec/rails_helper.rb`
my_rails_app/lib/my_gem  # Uses: `spec/spec_helper.rb`

The reason for this distinction is that I'm enforcing a clear separation of concerns because I don't want the gem to ever know about Rails (or be tempted to throw in Rails objects). Originally, I was thinking it'd be neat to define a Zeitwerk gem loader for my_rails_app/lib/my_gem but that doesn't work with Rails as you mentioned. So I'm using require in the gem while Rails autoloads the rest. This works great for fast/pure Ruby specs while still allowing the entire Rails test suite to test everything without load or circular dependency issues.

[fxn]

@bkuhlmann Technically, you can have a gem loader there if you will.

This is a routine situation, in a given Rails application instance there may be 7 loaders managing their trees, 2 from the application itself, and 5 from gem dependencies (you probably know this, just building the argument). It is cool to have at most one loader in the process with reloading enabled, and an arbitrary number of different loaders managing other projects with reloading disabled.

For that, if lib is in the autoload paths, just tell main to ignore the directory with the gem:

# config/application.rb
Rails.autoloaders.main.ignore("#{Rails.root}/lib/my_gem")

and then setup a regular loader in the gem with for_gem or whatever. With that ignore, as far as the Rails application is concerned, that subtree does not belong to the project.

My recommendation was related to the fact that now, in lib there is stuff that belongs to the app mixed with external stuff. It can be confusing. But that is subjective, and it works, so your call.

The tricky part comes when you want other loaders to also reload. For example, if you enable reloading in the gem, and then connect the loaders in some way or another, and the application uses gem constants in config/initializers, that will work on boot, but changes in a reloaded gem won't have an effect there. Initializers do not run on reloads. If your gem enhances Active Record and adds a mixin to ActiveRecord::Base, reloading won't change that object already pushed to the ancestor chain. Etc. Those are the kind of things that get tricky, and they are subtle.

[bkuhlmann]

Ah! This is slick, thanks. 🙇 Based on your advice, I refactored as follows:

Rails Application Configuration

# my_rails_app/config/application.rb
autoloaders.main.ignore Rails.root.join("lib/demo")

Gem Setup

# my_rails_app/lib/demo/setup.rb
require "zeitwerk"

Zeitwerk::Loader.new.then do |loader|
  loader.ignore __FILE__
  loader.tag = "demo"
  loader.push_dir "#{__dir__}/.."
  loader.setup
end

# Main namespace.
module Demo
end

RSpec Helper

# my_rails_app/spec/spec_helper.rb
require "demo/setup" # <-- Load the gem for pure Ruby specs only (Rails will inherit this).

RSpec.configure do |config|
  # Your configuration details go here.
end

I don't know if you recommend using something different than setup.rb but -- when the day arrives to extract the gem -- it would take minimal effort to rename setup.rb as demo.rb. I also want to acknowledge the trickiness of reloading the gem, as you mentioned, so have taken note of this for further exploration in the future.

💡 Based on this discussion -- and apologies to David for accidentally taking this discussion in a slightly divergent direction -- I think sharing the use of autoloaders.main.ignore related to unborn gems within a Rails application would be hugely insightful for folks to learn about in the Zeitwork documentation.

[fxn]

Why not have lib/demo/lib/demo.rb from day 1?

[bkuhlmann]

Actually, yes, that's a good call out, thanks. Kind of fell asleep at the wheel on that detail. 😅 Even better, this means you can use Gemsmith to easily generate new unborn/embedded gems right in your Rails application in the future without any additional customization (well, minus a little tweaking of the generated test suite) because Gemsmith is enabled with Zeitwerk support by default. 🎉

[fxn]

I'd still consider putting demo somewhere else, for example under vendor or under gems. That way, the Rails application only needs to have the gem's lib directory in $LOAD_PATH, but they are clearly apart. Or Bundler can load their entrypoint with :path.

Regarding the loader, you have to think this case is not special, really, Zeitwerk is designed to be used by gems too and there are hundreds already using it. It even has a special for_gem constructor I'd recommend to use. Your demo gem is local, but that does not matter, gems loaded by Bundler are also on disk, if you think about it.

[bkuhlmann]

I'd still consider putting demo somewhere else

Yeah, I'm going to experiment with this more. All good points.

Regarding the loader, you have to think this case is not special

True. In fact, I use Zeitwork for nearly all my gems and each has it's own loader I can inspect for debugging purposes. What I didn't realize, until having this discussion with you, was grabbing hold of autoloaders.main in Rails. That was a great revelation so thanks for that.

[davidmilo]

Sorry for the late reply. I appreciate your help on this a lot.

@davidmilo how does your application and your gems refer to each other's constants? Are there shared mixins or superclasses or top-level cross-references?

• Gem never references rails app
• Rails app only references main namespace/class of the gem. <GemNamespace>::<GemName>.something()

The first one should be easy:

1. DO NOT put the gems in the Gemfile.
2. Add the full path of each gem's lib directory to config.autoload_paths and config.eager_load_paths in config/application.rb.

That is all. I have not tested it, but should work. If gems have a version.rb file that defines VERSION, you may need an inflection. The autoloading guide explains how to do this.

I have tried your "simple approach" and I landed on one problem. Removing my local gem from rails app Gemfile gave me errors because dependent gems specified in gem's .gemspec were not available since rails app would only install what is in the rails Gemfile and would not know about those dependencies. It would however be nice to find a way to keep the local gem in the rails Gemfile to avoid this problem. I did workaround by putting dependent gems in the Gemfile of the rails app but that is not very suistainable.

[fxn]

@davidmilo have you tried adding

gemspec path: 'path/to/my_gem.gemspec'

to the Gemfile?

[davidmilo]

@davidmilo have you tried adding

gemspec path: 'path/to/my_gem.gemspec'

to the Gemfile?

Just tried it. It breaks the reloading 😢

[fxn]

@davidmilo which reloading? Could you share exactly what you have configured in the app and the gem?

[davidmilo]

Changes in the gem are not reloaded in the rails app when I use gemspec in rails Gemfile.

Let me setup a test repo.

[davidmilo]

This is what I started with:
https://github.com/davidmilo/local_gem_reloading

Here I tried to remove gem from gemfile and add the paths in application.rb davidmilo/local_gem_reloading#1

Here I tried to add gem to gemfile through gemspec and add the paths in application.rb davidmilo/local_gem_reloading#2

[fxn]

Thanks for the example!

• Problem with a regular gem declaration is that my_namespace-my_gem/lib/my_namespace/my_gem.rb is required by Bundler, and therefore not managed by the autoloader.
• Something similar happens if we only set a gemspec declaration.
• Problem with a gem declaration with require: false is that Bundler evaluates the version file because the gemspec loads it. As a side-effect, MyNamespace::MyGem is defined, and therefore my_namespace-my_gem/lib/my_namespace/my_gem.rb is ignored by the loader (the constant it would define is already defined externally, same idea).

Yeah, this is related to the gem being loaded with path, regular gems do not even ship with gemspecs.

Let me think about this.

[fxn]

The way to implement this approach is the following:

1. Declare the gem dependency with an ordinary gem declaration and require: false.
2. The gemspec should not load gem code. For example, it could hard-code the version.
3. The entry point should not have a require_relative for the version file as the sample app has. There is autoloading in place, there shouldn't be any require or require_relative to load code from the gem itself.
4. The Rails application should add the gem's lib directory to the autoload and eager load paths (technically eager load paths is enough, but I find more clear to push to the two of them).
5. The Rails application should have a rule to inflect VERSION, otherwise eager loading will err.

[davidmilo]

I tried this (davidmilo/local_gem_reloading#3) but reloading of the gem code doesn't work. Did I miss anything?

I am also puzzled how http gets actually required? It is not mentioned in the gem Gemfile(.lock) and neither in rails app Gemfile(.lock)

---

I think that this looks like lot of hacking of the gem. I would still like to in the future push this gem and be used by others. Maybe we could explore the other options you mentioned originally "Independent gem having it's own loading. (either using zeitwerk or manual file loading in gem would be fine)". Maybe we should explore setup where we have:

• independent gem (using zeitwerk internally for loading)
• independent rails app which adds gem in the gemflie with path

---

I actually started to look into this as I was writing this message and I experiment with some suggestions in the links I mentioned in my original post. I ended up making it work.

0. I converted my example to use 2 namespaces to test that scenario - MyRootNamespace::MyNamespace::MyGem
1. Convert Gem to use zeitwerk and enable reloading conditionally based on ENV variable:

# lib/my_root_namespace/my_namespace/my_gem.rb
require "zeitwerk"
loader = Zeitwerk::Loader.for_gem_extension(MyRootNamespace::MyNamespace)
loader.enable_reloading if ENV["LOCAL_GEM_RELOADING"]
loader.setup

2. During rails app initialisation:

• find the specific gem loader based on the tag
• setup file watcher to call reload

  gem_path = Rails.root.join('../', "my_root_namspace-my_namespace-my_gem")
  gem_loader = Zeitwerk::Registry.loaders.find { |loader|
    loader.tag == "MyRootNamespace::MyNamespace-my_gem"
  }

  file_watcher = ActiveSupport::FileUpdateChecker.new(gem_path.glob('**/*')) do
    gem_loader.reload
  end
 
  # Plug it to Rails to be executed on each request
  Rails.application.reloaders << Class.new do 
    def initialize(file_watcher)
      @file_watcher = file_watcher
    end
 
    def updated?
      @file_watcher.execute_if_updated
    end
  end.new(file_watcher)

3. I did a bit of generalisation of that code to get something which would work for any gem (considering it follows naming conventions, etc ..)

  def setup_reloading_for_local_zeitwerk_gem(gem_name, local_gems_folder)
    gem_path = Pathname.new(File.join(local_gems_folder, gem_name, "lib"))

    parts = gem_name.split("-")
    gem_entry = parts.delete(parts.last)
    gem_namespaces = parts.map(&:camelize)
    
    loader_tag = "#{[gem_namespaces.join("::"), gem_entry.underscore].join('-')}"

    gem_loader = Zeitwerk::Registry.loaders.find { |loader| loader.tag == loader_tag }

    file_watcher = ActiveSupport::FileUpdateChecker.new(gem_path.glob('**/*')) do
      gem_loader.reload
    end
   
    Rails.application.reloaders << Class.new do 
      def initialize(file_watcher)
        @file_watcher = file_watcher
      end
   
      def updated?
        @file_watcher.execute_if_updated
      end
    end.new(file_watcher)
    true
  end

# development.rb

setup_reloading_for_local_zeitwerk_gem("my_root_namespace-my_namespace-my_gem", Rails.root.join("../"))

What do you think?

Maybe I will clean this up and publish it as a mini gem.

Not sure if this is a best or safest way to find a loader by tag?

    loader_tag = "#{[gem_namespaces.join("::"), gem_entry.underscore].join('-')}"
    gem_loader = Zeitwerk::Registry.loaders.find { |loader| loader.tag == loader_tag }

Is it safe to use file checker/reloaders like this or are there any quirks I am not aware?

[davidmilo]

davidmilo/local_gem_reloading@e064426

[fxn]

@davidmilo since this needs more work and it is a Rails-specific ticket (Rails is who coordinates reloading and has the relevant APIs, beyond the scope of Zeitwerk itself), I have moved it to rails/rails#51185. Closing here.