[Feature Request] Dependency graph introspection

[Epigene]

Hi, we have a use-case for packwerk that could significantly cut down on CI work needed for monolithic apps.
If we could tell which pack a changed file in a merge request belongs to and which packs depend on the changed pack,
it would be possible to run CI for only the affected part of the code, not necessarily all domain.
For example, changes in front-end code would not necessitate running DB querier specs, because there's no
way queriers were affected.

Let's consider this example app:

app/
  domain/
    thing.rb
    package.yml
  io/
    thing_reader.rb
    package.yml
package.yml

The root pack lists both app/domain and app/io as dependencies, because we need both for the app to function.
app/domain has no dependencies, it is the core of our business. app/io depends on app/domain - contains front-end stuff etc.

Given a list of changed files in a merge request, we would love to be able to ask packwerk two things:

1. What package(s) are the files in?
2. What packages, followed all the way to dependency tree branch ends, are downstream of changed packages?

So for example if we have an MR that changes app/domain/thing.rb:

1. What package(s) are the files in? - ['app/domain']
2. What packages are downstream? - ['.', 'app/io']

Whereas if app/io/thing_reader.rb gets changed:

1. What package(s) are the files in? - ['app/io']
2. What packages are downstream? - ['.'] # root pack will always be downstream

Perhaps to answer the question "which pack specs need to be ran?" merging both of these lists is preferable,
so here's a proposed API:

# implicitly runs on root and in 'self only' mode - lists all packs in the project
$ packwerk packs
'.'
'app/domain'
'app/io'

$ packwerk packs "app/io/thing_reader.rb"
'app/io'

$ packwerk packs --dependents "app/io/thing_reader.rb"
'.'

$ packwerk packs --affected "app/io/thing_reader.rb"
'.'
'app/io'

[alexevanczuk]

Hey @Epigene we actually do a version of this in our CI suite using ParsePackwerk https://github.com/rubyatscale/parse_packwerk

There are two important things to note here:

1. You'd need to include violations in the graph traversal to more accurately predict the blast radius of a change
2. There are likely ways that one package impacts another through a mechanism that packwerk cannot pick up, so you may just want to make sure you have a mechanism to manage this (e.g. always running all tests on main, even if you don't run them all on feature branches).

I think to do this, we just need to do a breadth first search of a package's ancestors. Our implementation actually doesn't yet look at violations Here's our implementation:

require 'parse_packwerk'

class AffectedPackagesHelper
  def initialize(modified_packs)
    @packs = modified_packs
    @memo = {}
  end

  def dependent_packs
    @packs.each do |pack|
      inbound_dependencies_traversal(pack)
    end
    return @memo.keys
  end

  # Using BFS traversal, finds list of explicitly dependent packages defined
  # by each pack's `package.yml` and adds it to the memoized traversed packages
  # array @memo. 
  #
  # @param String the name of a pack in `packs/some_pack` format
  def inbound_dependencies_traversal(package_name)
    return if @memo.key?(package_name)

    queue = [package_name]

    while !queue.empty?
      curr_package = queue.pop
      # Our implementation doesn't yet look at violations, but it should look at 
      # ParsePackwerk.all.select { |package| ParsePackwerk::DeprecatedReferences.for(package).violations.any?{|v| v.to_package_name == curr_package }
      inbound_dependencies = @memo[curr_package] ||
                     ParsePackwerk.all.select { |package| package.dependencies.include?(curr_package) }.map(&:name)
      if !@memo.key?(curr_package)
        @memo[curr_package] = inbound_dependencies
      end

      inbound_dependencies.each do |dependency|
        if !@memo.key?(dependency)
          queue << dependency
        end
      end
    end
  end
end

Want to let me know if this technique works for you locally? I'd love to be able to support more CI tools like this in the packwerk ecosystem and want to work out the kinks and figure out what/where the API should live.

[Epigene]

@alexevanczuk Thanks for the quick reply, I'll give parse_packwerk a look.
I agree with both points you make, about needing to include violations (contents of pack deprecated_references.yml) in traversal and having a final-safety CI pass on everything once in a while.

[alexevanczuk]

Awesome, let me know what you come up with @Epigene ! I love this effort because allowing packwerk to open doors to conditional builds (as we refer to them) for fast (and thus cheaper) builds is a huge opportunity. Long-term, we hope that the same concept can allow for conditional deployments – that is, deploying a package separately once we've ascertained programmatically that it can stand in its own application (i.e. all dependencies fully specified) and all consumers are talking with it via a network-boundary-agnostic API (this is one of many reasons why we feel privacy enforcement is so important).