Migration guide for users coming from `doctest`

[martijnbastiaan]

As stated in the README of this project, doctest-parallel is a backwards-incompatible fork of doctest. Although none of your actual tests need to be rewritten, you need to integrate it differently into your project. Depending on your project you may also need to adjust some $setup blocks.

Comparing doctest and doctest-parallel

To understand what needs to be changed, it's useful to understand the differences between doctest and doctest-parallel. Once understood, fixing tests is hopefully a case of following error messages and adjusting minor things. In very rough terms, doctest operates as follows:

1. It parses a project's source files using the GHC API. It then extracts comments and detects doctests.
2. It starts a GHCi session - reinterpreting the whole project.
3. For every module:
   • It "opens" a module. Because this is run against interpreted code, all binders and imports are available, and all LANGUAGE pragmas are enabled.
   • For every test in the module: execute $setup, execute test.

Although there's overlap, this differs a bit from doctest-parallel:

1. Same as doctest: it parses a project's source files using the GHC API. It then extracts comments and detects doctests.
2. For every module Foo:
   • It starts a GHCi session
   • It runs import Foo. Because this is run against compiled code, only exposed binders are available and no LANGUAGE pragmas are set.
   • Same as doctest: For every test in the module: execute $setup, execute test.

Migrating

1. Integrate your project with doctest-parallel using the example project.
2. If you encounter "symbol not in scope" errors, make sure you import them in a $setup block.
3. If you encounter issues related to language pragmas use :set -XTheExtension.
4. If you encounter issues related to plugins use :set -fplugin The.Plugin
5. If you encounter issues related to ambiguous symbols, consider hiding the Prelude
6. If you encounter issues related to non-exported symbols or non-exposed modules, read this section of the README.

[andreasabel]

So if I understand this correctly, if I want to doctest functions that are not exported from a module, I basically have to give up and use the original doctest, or restructure my project to use Internal modules everywhere.

Because adding imports to $setup does not help with symbols that are not exported:

2. If you encounter "symbol not in scope" errors, make sure you import them in a $setup block.

Maybe there could be a bold banner on this migration guide that you cannot doctest non-exported symbols.

[martijnbastiaan]

You cannot test non-exported binders, that's correct. This is mentioned in the README too:

Generally, doctest-parallel cannot test binders that are part of non-exposed modules, unless they are re-exported from exposed modules.

[martijnbastiaan]

It says it pretty clearly, even in bold, in this issue too:

Because this is run against compiled code, only exposed binders are available and no LANGUAGE pragmas are set.

[martijnbastiaan]

You could consider conditionally exporting it based on a Cabal flag: false by default, but set in cabal.project. This should work for local development and CI.

[andreasabel]

You could consider conditionally exporting it based on a Cabal flag: false by default, but set in cabal.project. This should work for local development and CI.

Ah, yes this would be a path forward with less effort: simply put all export lists of modules that have doctests under a conditional (e.g. flag) and then have two build trees, one for running the tests and one for building the deployed version. A CI that runs the doctests would have to be hand-knitted (since haskell-ci does not allow the configuration of flags afaik).
I might come back to try this in case doctest does not make progress towards GHC 9.4 soonish.

[martijnbastiaan]

Small price to pay for the many advantages IMO :).

You could have a cabal.project.local in your .ci/.github folder with the right flags set, which you move into the repo root on CI.