Consider using compiled ETS match specs for event filtering

[michalmuskala]

This is just a wild idea, but could provide a very powerful and flexible API.

ETS allows making "standalone" match specs created with :ets.match_spec_compile/1 and executed with :ets.match_spec_run/2 - this is used by DETS for executing match specs and for example by the Version module for matching version requirements. Match specs are pretty efficient as they are executed on an internal small VM that runs inside BEAM itself, called PAM (Yo, Dawg).

I could imagine an API that allows you to subscribe with a match spec to events and would allow you to filter and possibly transform them before they would be delivered to you, reducing the amount of flying messages at the cost of extra work for the sender.

For example, here's a simple API for a PubSub based on elixir's Registry and this idea:

defmodule RegistryPubSub do
  def child_spec(opts) do
    Registry.child_spec(Keyword.merge(opts, keys: :duplicate))
  end

  def subscribe_match_spec(name, channel, spec) do
    Registry.register(name, channel, :ets.match_spec_compile(spec))
  end

  def publish(name, channel, events) do
    Registry.dispatch(name, channel, fn entries ->
      for {pid, spec} <- entries, event <- :ets.match_spec_run(events, spec), do: send(pid, event)
      :ok
    end)
  end
end

With some macro magic, I could imagine an API like:

require RegistryPubSub
RegistryPubSub.subscribe(MyServer, "foo_channel") do
  {:created, topic, _details} -> {:created, topic}
  {:deleted, %{name: name}} -> {:deleted, name}
end

I haven't evaluated the performance of this solution in practice, but I think it's very promising and allows for filtering events at the producer instead of filtering them at the receiver with a very powerful mechanism - basically almost all of pattern matching, since match specs are almost like making patterns first class citizens.

[josevalim]

@michalmuskala Just to clarify: the dispatching here is not process based. The code here is closer to Erlang/OTP's logger implementation than the Registry PubSub. In a nutshell, we store handlers on ets and for every event we read those handlers and invoke them.

I guess that, if we want to allow event filtering based on the metadata, then we could employ this technique for performance, but it is not something we are worried about right now.

Thoughts?

[arkgil]

Yes, if we want to allow filtering based on metadata then this looks promising, since we could probably use a familiar pattern-matching syntax for that. Good call, @michalmuskala! 👍 Having said that, we don't allow that now and I'd rather focus on validating API in some real project and a prototype where we actually gather metrics and push them somewhere. What do you think?