File-based snapshots

[zachallaun]

Mneme has focused on pattern-matching and automatically generating patterns. There are some situations, though, where it is not convenient (or possible) to store the generated value in the test source. For instance, you cannot currently benefit from auto_assert in a loop unless the generated pattern holds for every input (might as well use assert).

I propose the introduction of a new snapshot/1 macro that could be used with auto_assert to save the value in a separate snapshot file that will be used the next time the test runs.

# test/foo_test.exs
auto_assert snapshot(:my_snapshot) <- some_call()

When run the first time, Mneme generates the full pattern (good for value comparison, which will be used here) and stores it in a snapshot file. By default, that would be:

# test_snapshots/foo_test/my_snapshot.exs
[
  src: "test/foo_test.exs",
  expr: some_call(),
  val: :some_value
]

An example with a loop, currently not possible with Mneme:

for {name, input} <- inputs do
  auto_assert snapshot(name) <- some_call(input)
end

Additional implementation notes:

• Unlike pattern-patching assertions, snapshot comparisons are done with ==.
• Will live in Mneme.Snapshot which exports a snapshot/1 macro. This means it can be used with regular ExUnit assertions, e.g. assert snapshot(:foo) == some_call(), but without auto-updating (the test will just fail).
• Not imported by default, need to import Mneme.Snapshot. Can be requireed automatically by use Mneme, though.
• When used without a name, a generated one is added on the first run, e.g. snapshot("gen_f6XH2p").
• Snapshot names must be unique within the same test file to differentiate.
• Config option :snapshot_path controls the directory that snapshots are stored in (defaults to test_snapshots).

Undecided:

• Should the macro be snapshot() or snapshot!()? The bang kind of makes sense especially if used with assert (not auto_assert) because it will raise if the snapshot isn't present. It's also rather mutate-y in general and is dealing with an external file, etc. I'm actually leaning towards snapshot!().
• Some task for cleaning up unused snapshot files. The tricky thing is that for snapshots named at runtime, we have to ensure that all of them are run before marking anything as "stale". I don't think anything should ever be deleted automatically, though -- should be something like mix mneme.snapshots.clean.