Loading shadowed files (QUESTION)

[catmando]

I have a use case where it is needed to allow shadow files to be loaded. For details on why see below.

To do this I am using the loader's on_load callback to detect when the primary file is loaded, then checking to see if there is a shadowed file in the appropriate directory, and then calling require on the shadowed file.

This seems to work fine (and I have been doing a similar thing in classic mode) but I am wondering if there are any negative implications within Zeitwerk.

BTW: It was very nice implementing this in Zeitwerk as it required no monkey patches (as I had to do in classic mode.). Seems like a great step forward, thanks!

WHY AM I DOING THIS (IN CASE YOU CARE)

The Hyperstack Framework (a client-server isomorphic extension to Rails) allows for shadowed Ruby files.

This allows a model definition to have a server-only part and a client-and-server part. This is generally needed in some models
and services for efficiency and security.

The Hyperstack directory structure (within a Rails app) looks like this:

--app
   |- hyperstack
      |- components
      |- models
        - some_model.rb
        ....
      |- ...
   |- models
        - some_model.rb
        ....

In this case, the definitions in app/hyperstack/models/some_model.rb will be available on both the server and client (i.e. browser) while the definitions in app/models/some_model.rb will only be seen (per standard rails) on the server.

For details see: Hyper Model Documentation

[fxn]

Interesting!

Couple of questions come to mind.

Do you really need app/hyperstack/models in the autoload paths? Would it make sense to configure the autoloader to ignore that directory?

I am wondering because if that was possible, the main file would be better defined as the only one managed by Zeitwerk. Otherwise it depends on the order of configuration of the root directories.

Also, could be intentional to ignore..., but depends on usage.

Another question would be related to idempotence. If you require the shadowed file, on reload that require won't interpret the file again.

Let me share these first thoughts and we can iterate from there :).

[catmando]

Thank you so much for you attention to this.

1). Both directories need to be autoloaded.

Perhaps a simple but dumb example will make it clear why:

# app/hyperstack/models/sample.rb
class Sample < ApplicationRecord
end

# app/models/sample.rb
class Sample < ApplicationRecord
  def self.super_secret_server_side_method
    true
  end
end

Here the Sample model will be accessible on the server as normal, but will also be compiled (via Opal) for the client. So all the normal Rails ActiveRecord methods will be visible on both server and client. For example you might say Sample.create on the client to create a new record.

But for whatever reason we want super_secret_server_side_method to only be accessible on the server. This is typically either because the method(s) have complex computations that are best done on the server, or for security reasons, the code should not be exposed or executed on the client.

Thus we can optionally split definitions across two files. This by the way also applies to other directories.

2). Reloading

Yes this was what I was afraid of. However when a reload occurs does the on_load callback get called again? If so then all should still work, as that is how I know to look for the shadowed files. If on_load is not called, is there another hook that could be used to tell us that file is getting reloaded?

[fxn]

But, in your example, app/hyperstack/models/sample.rb is not autoloaded by Zeitwerk, it is manually autoloaded by your callback. That is the key observation. Question would be, do you have anything in app/hyperstack/models that is not also present in app/models and needs to be autoloaded by Zeitwerk regularly?

Yes, on_load is called when the class is loaded again after a reload too.

[fxn]

Hmm, after reading again the first message, perhaps I got it backwards. Which of the two sample.rb is considered to be the primary file?

[catmando]

the app/hyperstack/ directory is first, so it is "shadowing" any other directories in app.

So the order that things are loaded in the example are:

app/hyperstack/models/sample.rb
when this loaded the on_load handler then discovers that
app/models/sample.rb also exists, and so does a require of that file

and btw if on_load is called then everything should work (I think)

[fxn]

OK, then the actual question is: do you need app/models to be in the autoload paths?

and btw if on_load is called then everything should work (I think)

In principle, Kernel#load would be the way to go.

[fxn]

Hey, do you need more help with this?

[catmando]

@fxn apologies, I was out hiking! I think its all set, but I didn't understand you comment about Kernel#load. Are you saying use that instead of require inside the on_load handler? I am using require now, and all is well but I will try load and see if there is any difference.

Thanks again for all the help, and making such a nice robust API.

[catmando]

FYI for reference here is the handler, in case it helps someone else, or you spot a problem I might have missed:

Note this works for Rails >= 5.0 / classic or zeitwerk.

# require "hyperstack/server_side_auto_require.rb" in your hyperstack initializer
# to autoload shadowed server side files that match files
# in the hyperstack directory

if Rails.configuration.try(:autoloader) == :zeitwerk
  Rails.autoloaders.each do |loader|
    loader.on_load do |_cpath, _value, abspath|
      ActiveSupport::Dependencies.add_server_side_dependency(abspath) do |load_path|
        loader.send(:log, "Hyperstack loading server side shadowed file: #{load_path}") if loader&.logger
        require("#{load_path}.rb")
      end
    end
  end
end

module ActiveSupport
  module Dependencies
    HYPERSTACK_DIR = "hyperstack"
    class << self
      alias original_require_or_load require_or_load

      # before requiring_or_loading a file, first check if
      # we have the same file in the server side directory
      # and add that as a dependency

      def require_or_load(file_name, const_path = nil)
        add_server_side_dependency(file_name) { |load_path| require_dependency load_path }
        original_require_or_load(file_name, const_path)
      end

      # search the filename path from the end towards the beginning
      # for the HYPERSTACK_DIR directory.  If found, remove it from
      # the filename, and if a ruby file exists at that location then
      # add it as a dependency

      def add_server_side_dependency(file_name, loader = nil)
        path = File.expand_path(file_name.chomp(".rb"))
                   .split(File::SEPARATOR).reverse
        hs_index = path.find_index(HYPERSTACK_DIR)

        return unless hs_index # no hyperstack directory here

        new_path = (path[0..hs_index - 1] + path[hs_index + 1..-1]).reverse
        load_path = new_path.join(File::SEPARATOR)

        return unless File.exist? "#{load_path}.rb"

        yield load_path
      end
    end
  end
end

[fxn]

@catmando awesome re hiking! :)

Sure, let me explain. Let's consider the Sample model.

1. The model is somehow loaded for the first time.
2. The callback notices it has to decorate it.
3. Evaluates the companion file with Kernel#require.

Cool, now we reload. The original Sample constant is removed from Object.

1. Sample is referenced an loaded again.
2. The callback notices it has to decorate it.
3. Evaluates the companion file with Kernel#require.

Now, (3) is no longer able to evaluate the file because Kernel#require is idempotent, so Sample is not extended anymore on reloads.

On the other hand, if (3) is done with Kernel#load, it always executes the file.

(Note that require_or_load is private interface, and only used by classic.)