cabal-doctest

A Setup.hs helper for running doctests.

Simple example

For the typical use case, namely a .cabal file with a single library containing doctests, adapting the simple example will be sufficient.

To use this library in your Setup.hs, you should specify a custom-setup section in your .cabal file. For example:

custom-setup
 setup-depends:
   base >= 4 && <5,
   Cabal,
   cabal-doctest >= 1 && <1.1

Note: The Cabal dependency is only needed for cabal-install < 2.4 (see issue haskell/cabal#4288).

You'll also need to specify build-type: Custom at the top of the .cabal file. Now put this into your Setup.hs file:

module Main where

import Distribution.Extra.Doctest (defaultMainWithDoctests)

main :: IO ()
main = defaultMainWithDoctests "doctests"

Given your test-suite is called doctests, when you build your project, this Setup will generate a Build_doctests module. (Note: If your test-suite goes by another name, like foo, this creates a Build_foo module.) To use the generated module in a testsuite, simply do like so:

module Main where

import Build_doctests (flags, pkgs, module_sources)
import Data.Foldable (traverse_)
import System.Environment.Compat (unsetEnv)
import Test.DocTest (doctest)

main :: IO ()
main = do
    traverse_ putStrLn args -- optionally print arguments
    unsetEnv "GHC_ENVIRONMENT" -- see 'Notes'; you may not need this
    doctest args
  where
    args = flags ++ pkgs ++ module_sources

(The System.Environment.Compat module is from the base-compat package. This package already in the transitive closure of doctest's dependencies. System.Environment.unsetEnv was added with GHC 7.8 so, if you don't need to support versions of GHC older than 7.8, you can use System.Environment from base instead.)

Example with multiple .cabal components

cabal-doctest also supports more exotic use cases where a .cabal file contains more components with doctests than just the main library, including:

doctests in executables,
doctests in internal libraries (if using Cabal-2.0 or later).

Unlike the simple example shown above, these examples involve named components. You don't need to change the Setup.hs script to support this use case. However, in this scenario Build_doctests will generate extra copies of the flags, pkgs, and module_sources values for each additional named component.

Simplest approach is to use x-doctest-components field, for example:

x-doctest-components: lib lib:internal exe:example

In that case, the test driver is generally:

module Main where

import Build_doctests (Component (..), components)
import Data.Foldable (for_)
import System.Environment.Compat (unsetEnv)
import Test.DocTest (doctest)

main :: IO ()
main = for_ components $ \(Component name flags pkgs sources) -> do
    print name
    putStrLn "----------------------------------------"
    let args = flags ++ pkgs ++ sources
    for_ args putStrLn
    unsetEnv "GHC_ENVIRONMENT"
    doctest args

There is also a more explicit approach: if you have an executable named foo, then separate values named flags_exe_foo, pkgs_exe_foo, and module_sources_exe_foo will be generated in Build_doctests. If the name has hyphens in it (e.g., my-exe), then cabal-doctest will convert those hyphens to underscores (e.g., you'd get flags_my_exe, pkgs_my_exe, and module_sources_my_exe). Internal library bar values will have a _lib_bar suffix.

An example testsuite driver for this use case might look like this:

module Main where

import Build_doctests
       (flags,            pkgs,            module_sources,
        flags_exe_my_exe, pkgs_exe_my_exe, module_sources_exe_my_exe)
import Data.Foldable (traverse_)
import System.Environment.Compat (unsetEnv)
import Test.DocTest

main :: IO ()
main = do
    unsetEnv "GHC_ENVRIONMENT"
    -- doctests for library
    traverse_ putStrLn libArgs
    doctest libArgs

    -- doctests for executable
    traverse_ putStrLn exeArgs
    doctest exeArgs
  where
    libArgs = flags            ++ pkgs            ++ module_sources
    exeArgs = flags_exe_my_exe ++ pkgs_exe_my_exe ++ module_sources_exe_my_exe

See this example for more details.

Additional configuration

The cabal-doctest based Setup.hs supports few extensions fields in pkg.cabal files to customise the doctest runner behaviour, without customising the default doctest.hs.

test-suite doctests:
  if impl(ghc >= 8.0)
    x-doctest-options: -fdiagnostics-color=never
  x-doctest-source-dirs: test
  x-doctest-modules: Servant.Utils.LinksSpec

  ...

x-doctest-options Additional arguments passed into doctest command.
x-doctest-modules Additional modules to doctest. May be useful if you have doctest in test or executables (i.e not default library complonent).
x-doctest-src-dirs Additional source directories to look for the modules.

Notes

Recent versions of Cabal (for instance, 2.0) can choose to build a package's doctest test suite before the library. However, in order for cabal-doctest to work correctly, the library must be built first, as doctest relies on the presence of generated files that are only created when the library is built. See #19.

A hacky workaround for this problem is to depend on the library itself in a doctests test suite. See the simple example's .cabal file for a demonstration. (This assumes that the test suite has the ability to read build artifacts from the library, a separate build component. In practice, this assumption holds, which is why this library works at all.)
custom-setup section is supported starting from cabal-install-1.24. For older cabal-install's you have to install custom setup dependencies manually.
stack respects custom-setup starting from version 1.3.3. Before that you have to use explicit-setup-deps setting in your stack.yaml. (stack/GH-2094)
You can use x-doctest-options field in test-suite doctests to pass additional flags to the doctest.
For build-type: Configure packages, you can use defaultMainAutoconfWithDoctests function to make custom Setup.hs script.
If you use the default . in hs-source-dirs, then running doctests might fail with weird errors (ambiguous module errors). Workaround is to move sources under src/ or some non-top-level directory.
The extensions: field isn't supported. Upgrade your .cabal file to use at least cabal-version: >= 1.10 and use default-extensions or other-extensions.
If you use QuickCheck properties (prop>) in your doctests, the test-suite doctest should depend on QuickCheck and template-haskell. This is a little HACK: These dependencies aren't needed to build the doctests test-suite executable. However, as we let Cabal resolve dependencies, we can pass the resolved (and installed!) package identifiers to to the doctest command. This way, QuickCheck and template-haskell are available to doctest, otherwise you'll get errors like:

    Variable not in scope:
      mkName
        :: [Char]
           -> template-haskell-2.11.1.0:Language.Haskell.TH.Syntax.Name

    Variable not in scope:
      polyQuickCheck
        :: Language.Haskell.TH.Syntax.Name -> Language.Haskell.TH.Lib.ExpQ

From version 2, Stack sets the GHC_ENVRIONMENT variable, and GHC (as invoked by doctest) will pick that up. This is undesirable: cabal-doctest passes all the necessary information on the command line already, and can lead to ambiguous module errors as GHC will load the environment in addition to what cabal-doctest instructs it to.

Hence, cabal-doctest tells GHC to ignore package environments altogether on the command line. However, this is only possible since GHC 8.2. If you are using cabal-doctest with Stack 2 and GHC 8.0 or earlier and seeing ambiguous module errors or other mysterious failures, try manually unsetting GHC_ENVIRONMENT before invoking doctest.
If you are on Nix. doctest will not pick up your version of GHC if you don't point it towards it, and therefore will result in "cannot satisfy -package-id" errors. You will need to set NIX_GHC and NIX_GHC_LIBDIR within your environment in order for doctest to pick up your GHC. Put the following in shell.nix and run nix-shell.

# shell.nix
{ pkgs ? import <nixpkgs> {} }:
let
  myHaskell = (pkgs.haskellPackages.ghcWithHoogle (p: with p; [
    # Put your dependencies here
    containers
    hslogger
  ]));
in
pkgs.mkShell {
  name = "myPackage";
  
  # These environment variables are important. Without these,
  # doctest doesn't pick up nix's version of ghc, and will fail
  # claiming it can't find your dependencies
  shellHook = ''
    export NIX_GHC=${myHaskell}/bin/ghc
    export NIX_GHC_LIBDIR=${myHaskell}/lib/ghc-8.10.7
  '';
  buildInputs = with pkgs; [ 
    myHaskell
  ];
}

Copyright