More flexible matching, ordering, cardinality, and responses

[cdsmith]

Currently, monad-mock requires an exact list of the actions that will be performed by the test case. I think it's fairly well understood that, in practice, this leads to over-assertion and brittle tests. By contrast, well-developed mock frameworks in mainstream programming languages have converged around a design that lets a programmer:

1. Match arguments using predicates, rather than exact values.
2. Specify whether mocked actions must happen in a guaranteed order, in any order, some number of times, etc.
3. Perform arbitrary actions when responding to a mocked call, including looking at parameters, setting up additional expectations (e.g., if you open a file handle, you must close it). This isn't so important if the exact sequence of actions is listed, but it's more important when expectations are more flexible.

The effect of this is to move closer to a middle ground between mocks and fakes, where a user can write tests that pass for a larger range of correct implementations, while asserting more precisely which properties they care about.

I've been experimenting today with extending the techniques in monad-mock to provide some of these benefits. An example that I think is at least a bit compelling is https://code.world/haskell#P1ruEyf0CvftpB5zOiN2kUw

copyFile :: MonadFS m => FilePath -> FilePath -> m ()
copyFile a b = do
  writeFile "progress.txt" "Starting..."
  txt <- readFile a
  cfg <- readFile ".config"
  writeFile "progress.txt" "... read the source file"
  writeFile b $
    if cfg == "transform: upper" then map toUpper txt else txt
  writeFile "progress.txt" "Finished."

main :: IO ()
main = hspec $ do
  describe "copyFile" $ do
    let isConfigFile = startsWith "." `andAlso` hasSubstring "config"

        setupConfig tform =
          mock $
            whenever $
              ReadFile_ isConfigFile :=> \_ -> return ("transform: " ++ tform)

        ignoreProgress =
          mock $
            whenever $
              WriteFile_ (isEqual "progress.txt") isAny :=> \_ -> return ()

        ignoreCopy = do
          mock $ whenever $ ReadFile "foo.txt" |-> ""
          mock $
            whenever $
              WriteFile_ (isEqual "bar.txt") isAny :=> \_ -> return ()

    it "copies the file correctly" $
      example $
        runMockT $ do
          setupConfig "none"
          ignoreProgress

          mock $ expect $ ReadFile "foo.txt" |-> "lorem ipsum"
          mock $ expect $ WriteFile "bar.txt" "lorem ipsum" |-> ()

          copyFile "foo.txt" "bar.txt"

    it "performs the transform" $
      example $
        runMockT $ do
          setupConfig "upper"
          ignoreProgress

          mock $ expect $ ReadFile "foo.txt" |-> "lorem ipsum"
          mock $ expect $ WriteFile "bar.txt" "LOREM IPSUM" |-> ()

          copyFile "foo.txt" "bar.txt"

    it "doesn't read the config file more than once" $
      example $
        runMockT $ do
          ignoreProgress
          ignoreCopy

          mock $
            expectN (atMost 1) $
              ReadFile_ isConfigFile :=> \_ -> return "transform: none"

          copyFile "foo.txt" "bar.txt"

    it "reports progress in progress.txt" $
      example $
        runMockT $ do
          setupConfig "none"
          mock $
            inSequence
              [ expect $ WriteFile "progress.txt" "Starting..." |-> (),
                expect $ ReadFile "foo.txt" |-> "",
                expect $ WriteFile "progress.txt" "... read the source file" |-> (),
                expect $ WriteFile_ (isEqual "bar.txt") isAny :=> \_ -> return (),
                expect $ WriteFile "progress.txt" "Finished." |-> ()
              ]

          copyFile "foo.txt" "bar.txt"

I don't know how to keep this backward compatible, but except for the generated instances, it's close. One simply must replace :-> with |->, and move the list of function calls from an argument of runMockT to mock calls in the body of the test. To retain the behavior of the old version, you should use inSequence in the mock calls, but honestly it's probably better not to, unless there's a reason to assert things about the sequence, so you should probably use allOf or multiple mock calls instead.

There were some places I changed a bit more than necessary, perhaps: such as removing the redundant argument to runMockT, and changing the Action class to Mockable and parameterizing over the constraint. But honestly, I think it's so much nicer and more consistent this way.

This is just a prototype. But I wanted to get some early feedback about whether this is a direction you're interested in going with monad-mock, or if this seems better suited to a new package.

[cdsmith]

Some details:

1. Currently, if there is more than one active expectation that matches an action, it is an error. However, it's fairly common in mainstream mock frameworks to set a default response with broad matchers, and then override it with more specific matches. This is sometimes handled there by ad hoc rules about the "most specific" rule matching, but it might be better to be explicit about it, and let priorities be attached to matches.
2. Another place where this library is substantially different from mainstream mock frameworks is that there are no default actions, so if you don't set an expectation, an action is always an error. It wouldn't be too hard to add something to Mockable that sets up default (low priority... see above) rules that provide defaults for some actions... especially those that return (). I'm sure there will be controversy over whether that's the right thing to do. (gMock, for example, makes the choice that actions that return void always silently succeed unless you set up another expectation for them, and then failing to match that expectation is an error regardless of the default behavior. That's an ugly and arbitrary rule.) Maybe it's better to let people just define their own initializers to set up default expectations.

[cdsmith]

@lexi-lambda Is this library dead?

I see the last commit was 2017, and no response to issues and pull requests for a few years. If so, I'll strongly consider releasing a new library instead. But I'm heavily influenced by monad-mock here, so wanted to check before doing something as drastic as a fork.

[cdsmith]

As I've continued to work on this, my local code has diverged far enough from monad-mock that it no longer makes sense to look at it as a prototype for changes to monad-mock. Instead, it seems to want to be a new project. I'd still love to hear from you if you're interested in discussing my implementation and the different choices I've made, but I will close this issue.