Implement process isolation support for tests

[horenmar]

Major features that could be implemented afterwards:

1. Catch should not go crazy if the test case uses fork or similar system call
2. Catch should allow running tests in isolated processes
3. Catch should allow testing for abort -- there should be a macro along the lines of REQUIRE_ABORT(expr), that launches separate process to check expr and succeeds if the launched process aborts
4. Catch should allow limited time for test suite execution

[pavelponomarev]

Really, really needed feature!

[Leandros]

👍

[ranty-fugue]

@philsquared Is this still under consideration? Would really like to see this feature! :-)

[Cloudef]

#pragma once

#include <functional>
#include <iostream>
#include <sstream>
#include <thread>
#include <unistd.h>
#include <sys/wait.h>
#include <fcntl.h>

struct catchpp_stdstream { int fd[3], target; std::stringstream ss; };

static inline bool
catchpp_fork_and_run(std::function<void(void)> fun) {
    struct catchpp_stdstream stream[] = {
        { { -1, -1, -1 }, STDOUT_FILENO, std::stringstream() },
        { { -1, -1, -1 }, STDERR_FILENO, std::stringstream() },
    };

    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        pipe(stream[i].fd);
        fcntl(stream[i].fd[0], F_SETFL, O_NONBLOCK);
        fcntl(stream[i].fd[1], F_SETFL, O_NONBLOCK);
    }

    pid_t pid;
    if ((pid = fork()) == 0) {
        for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
            dup2(stream[i].fd[1], stream[i].target);
            close(stream[i].fd[1]);
            close(stream[i].fd[0]);
        }
        fun();
        _exit(0);
    }

    int code;
    waitpid(pid, &code, 0);

    char buf[1024];
    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        close(stream[i].fd[1]);
        for (ssize_t r = 0; (r = read(stream[i].fd[0], buf, sizeof(buf))) > 0;) stream[i].ss.write(buf, r);
        close(stream[i].fd[0]);
    }

    std::cout << stream[0].ss.str();
    std::cerr << stream[1].ss.str();
    return WIFEXITED(code);
}

#define REQUIRE_ABORT(x) REQUIRE_FALSE(catchpp_fork_and_run(x))

static inline void
catchpp_catch_stdstreams(std::function<void(void)> fun, std::function<void(const std::stringstream &sout, const std::stringstream &serr)> then) {
    struct catchpp_stdstream stream[] = {
        { { -1, -1, -1 }, STDOUT_FILENO, std::stringstream() },
        { { -1, -1, -1 }, STDERR_FILENO, std::stringstream() },
    };

    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        pipe(stream[i].fd);
        fcntl(stream[i].fd[0], F_SETFL, O_NONBLOCK);
        fcntl(stream[i].fd[1], F_SETFL, O_NONBLOCK);
        stream[i].fd[2] = dup(stream[i].target);
        dup2(stream[i].fd[1], stream[i].target);
    }

    fun();

    char buf[1024];
    for (size_t i = 0; i < sizeof(stream) / sizeof(stream[0]); ++i) {
        fsync(stream[i].target);
        for (ssize_t r = 0; (r = read(stream[i].fd[0], buf, sizeof(buf))) > 0;) stream[i].ss.write(buf, r);
        close(stream[i].fd[0]);
        close(stream[i].fd[1]);
        dup2(stream[i].fd[2], stream[i].target);
        close(stream[i].fd[2]);
    }

    std::cout << stream[0].ss.str();
    std::cerr << stream[1].ss.str();
    then(stream[0].ss, stream[1].ss);
}

#define CATCH_STDSTREAM(x, y) catchpp_catch_stdstreams(x, y)

Something I've used. Fork is not really safe for various things though. Isolating tests to own processes would be the right way.

[robert-andrzejuk]

microsoft/GSL#831

[johnmcfarlane]

I think I've mentioned this in person but ASSERT_DEATH is the only thing keeping using GTest both at work and at home. It's vital to testing contract tests.

[rfbird]

I think I've mentioned this in person but ASSERT_DEATH is the only think keeping using GTest both at work and at home. It's vital to testing contract tests.

+1, we might have to swap over to gtest..

[escherstair]

+1 from my side too

[tsondergaard]

Here is an idea for a design:

I imagine that we extend the existing runner Catch::Session::run() with an undocumented command-line switch --managed where you can pass two file descriptors used for communication. Then we create an external runner called catch2-runner that you run e.g like this: catch2-runner test-executable . The catch2-runner will then create two pipes and launch test-executable with test-executable --managed fd1 fd2. It will then be able to send instructions to the test-executable via one of the fds and receive responses on the other. I imagine we make a simple protocol for communication - perhaps json based. I imagine the protocol will provide catch2-runner with commands like these:

• Request list all available tests with tags
• Run test

The test-executable running Catch::session::run() in managed mode will be able to respond with

• List of all available tests with tags
• Test competion with metadata - duration and test result

catch2-runner will execute the first command to get a list of all tests and after that it will issue Run test commands every time it receives a response back about a test completing.

This would give us the following benefits

• A crashing test will still give a full (junit) test report xml - only the failing test(s) will be flagged as crashing instead of what happens now which is leave an empty or missing xml report
• We can reliably capture stderr and stdout (I’d like them to be joined) and include that in the junit xml report for much improved usability in CI. stderr+stdout can be thrown away for all tests that don’t fail to avoid generating huge xml files
• We have a path for running multiple instances of test-executable in parallel. For large suites of tests this would be a significant win. All it would take is for catch2-runner to be able to launch multiple copies of test-executable and run different subset of tests in each.
• Ability to timeout individual tests.

Doing this would require some platform specific code and/or some dependencies, ie for creating pipes, starting and managing processes and perhaps a JSON library for the wire protocol.

[Raekye]

I have some code that calls std::terminate() if a requested/required allocation would be too large (I have exceptions disabled so throwing an exception is not an option). Would love this feature!

[Zerophase]

Any word on when this is being implemented? I have an assert in my code to make sure passed values match as an optimization to avoid a strlen call. Notifies others of proper usage, and gets optimized out on release builds. An exception is not exactly ideal here.

[bcaddy]

Any word on this? The lack of death tests is forcing me to switch to google test and I'd rather not.

[Bktero]

Same same! This feature would be highly appreciated :)

[tlifschitz]

Same same! This feature would be highly appreciated :)

[MiaoDX]

I had one rather "happy" night by looking at gtest and catch2 (I want one proper lib for clang-tidy integration):

Now, gtest thinks clang-tidy is giving false positive: google/googletest#2442

and catch2 currently has no EXPECT_DEATH

Both seems will keep as is for a while 0.0 😆