ipfs/aegir

Improving Aegir test speed

RonaldZielaznicki opened this issue · 2 comments

RonaldZielaznicki commented

TLDR;

Are y'all interested in some refactors?

That would...

  • reduce testing time (from 4000ms to 45ms per test)
  • utilize the extra time to introduce more tests for more descriptive coverage
  • save money on CI
  • remove side effects from ./src/utils.js (making it easier to add new commands)

PS: I will break code examples here into smaller more manageable chunks for commits / PR reviews. The linked code was made as a proof.

Is your feature request related to a problem? Please describe

When running the tests for Aegir, most tests take 4+ seconds to finish

Describe the solution you'd like

Ideally, Aegir should not take several minutes to run its test suite

Describe alternatives you've considered

  1. using command.parse in tests over execa
  2. remove reliance on outside modules through dependency injection (more context below)

Additional context

Using command.parse over execa

Running tests with execa forces additional overhead by opening a new node instance for each test.

By using command.parse we use the existing instance and dodge this overhead. The caveat/blocker is src/utils.js . It has side affects on import that make it difficult to test commands in isolation.

See: 506644e

Results

We gain almost 2 seconds back per test by doing this.

Default tests with execa and module dependencies
  build
    esm
      ✔ should build an esm project (4005ms)
    ts
      ✔ should build a typescript project (4031ms)
Tests run from command.parse
  build
    esm
      ✔ should build an esm project (2394ms)
    ts
      ✔ should build a typescript project (2287ms)

Removing reliance on outside modules from tests

Completely removing them wouldn't be great since we still want to make sure that it works from end to end.

But we could still pull some dependency injections to test aegir independently.

See: bfa18e4

Results

Running the typescript command on its own takes 2-3 seconds for a small project. If you look back to the tests run with command.parse you'll notice that's about how long running command.parse takes.

With some dependency injection, that type drops to a negligible value. Compared from the original 4 seconds, it drops down to 45ms.

Default tests with execa and module dependencies
  build
    esm
      ✔ should build an esm project (4005ms)
    ts
      ✔ should build a typescript project (4031ms)
Tests run from command.parse and dependency injections
  build
    esm
      ✔ should pass the appropriate options to injected dependencies (45ms)
      ✔ should build an esm project (2342ms)
    ts
      ✔ should build a typescript project (2280ms)
RonaldZielaznicki commented

I've since cleaned up the examples I made into more consumable chunks to be easier to understand.

Here is a draft pr (on my fork of aegir) to showcase the changes. If this seems like something y'all are interested in, I'll drop the PRs over here too.

https://github.com/RonaldZielaznicki/ipfs-aegir/pull/5

Here is what a test run looks like with these changes:

  build
    should pass the appropriate options to dependency
      tsc
        ✔ passes preferLocal (257ms)
        ✔ passes stdio (149ms)
      esbuild
        cli options
          ✔ passes build to metafile (158ms)
        default config options
          ✔ passes conditions (129ms)
          ✔ passes format (99ms)
          ✔ passes minify (108ms)
          ✔ passes sourcemap (104ms)
        derived options
          ✔ passes pascalCased globalName from package.name (110ms)
          ✔ passes entrypoint "/src/index.js" for none-typescript project (102ms)
          ✔ passes entrypoint "/dist/src/index.js" for typescript project (106ms)
          ✔ passes outfile (105ms)
          ✔ passes umdPre as js banner (102ms)
          ✔ passes umdPost as js footer (100ms)
RonaldZielaznicki commented

Closing due to a lack of input.