Support for CTest

[nbn22385]

Is your feature request related to a problem? Please describe.

I would like to suggest support for running CTest (unit test driver packaged with CMake by default) using the cmake4vim plugin, and having any test errors be parsed in the quickfix list (file/line #).

Describe the solution you'd like

A set of one or more commands for CTest integration:

• A new command, possibly named CTest that can accept optional arguments. See CTest documentation for common command line options here. Example: :CTest --verbose --rerun-failed. Running this command will output to the quickfix window and highlight the first test failure (file and line #).

• A new command, possibly named CMakeBuildAndCTest that will run CMakeBuild in its current form and, if successful, execute CTest as described above.

Describe alternatives you've considered

Created custom commands that modify makeprg to run the commands via make:

• Created a new command CTest: command! CTest let &makeprg = 'cd build && ctest --output-on-failure' <bar> make
• Created a new command CMakeBuildAndTest: command! CMakeBuildAndTest let &makeprg = 'cd build && make && ctest --output-on-failure' <bar> make
  • If the build fails the test will still run (and pass if the last set of test pass, false positive)
  • Did not get the desired quickfix output

Created a custom command that uses cmake4vim's CMakeBuild with a test target:

• command! -nargs=? -complete=custom,cmake4vim#CompleteTarget CMakeBuildAndTest call cmake4vim#CMakeBuild('--target test')
  • Now the target is set to test for the next time i want to run CMakeBuild so it only runs the test. I would need to run CMakeSelectTarget all to put the target back.

Created a custom function that uses cmake4vim's CMakeBuild in two ways:

function! Test() abort
 call cmake4vim#CMakeBuild('--target all')

 " if there are no build errors, run the tests
 let s:num_build_errors = len(filter(getqflist(), 'v:val.valid'))
 if s:num_build_errors == 0
   echo "Build succeeded!"

   call cmake4vim#CMakeBuild('--target test')

   let s:num_test_errors = len(filter(getqflist(), 'v:val.valid'))
   if s:num_test_errors == 0
     echo "Tests passed!"
   endif
 else
   echo "Build failed!"
 endif
endfunction

command! CMakeBuildAndCTest call Test()

• I couldn't get the test failure to report the correct output. Running ctest on the command line produces correct result. (this might be due to the async nature of running both commands in the same function, also i have not defined an errorformat for ctest results)
• The targets are hardcoded to "all" and "test"
• Cannot pass arguments to CTest by calling CMake with a target of "test"

Let me know if there are already ways to accomplish this with custom commands and/or <bar>ing together cmake4vim commands somehow.

[ilyachur]

Hi @nbn22385 ,

I didn't work with CTest.
But I thought that if I use test target, it means that CMake should automatically build changed files and run the tests. Am I not right?

Do you know how CTest works with different CMake generators? Is it independent on CMake generator type?

[nbn22385]

Hi @ilyachur , thanks for the response.

But I thought that if I use test target, it means that CMake should automatically build changed files and run the tests. Am I not right?

Running cmake --build . --target test (or make test or even ctest) does not trigger cmake to rebuild any changed source files, the last built version of the test will be run. You must run cmake --build . [--target all] (or make) to rebuild the source, then rerun the preferred test command. The rebuild+test behavior can be achieved with a workaround.

In addition, running tests without going through a call to ctest provides no way of passing ctest-specific arguments to get more fine-grained control over unit test output.

Do you know how CTest works with different CMake generators? Is it independent on CMake generator type?

Calling include(CTest) or enable_testing() in your project's CMakeLists.txt adds another build target, which is test for Makefile generators, or RUN_TESTS for integrated development environments (like Visual Studio). I don't know exactly how CTest is impacted by different generators, however I do believe it is independent of the generator that is used except for the Visual Studio example. I tested with Ninja and the ctest command works as expected.

I hope I was able to address your questions.

[ilyachur]

Hi @nbn22385 ,

Thank you for clarification. Now it is more clear for me.

In general if you have resources for contributing you can create a PR for this feature.

Or I will investigate this enhancement because I didn't use CTest.

Also "test proj" application should be extended in order to test this feature.

[nbn22385]

After some more research, you can pass arguments to ctest via cmake in the form of:

cmake --build <bld_directory> --target test -- ARGS="<ctest_args>"

Also I found that cmake files can be regenerated when calling ctest using the --build-and-test argument if the generator is specified:

ctest --build-and-test <path-to-source> <path-to-build> --build-generator <generator>

I am playing around with the code on a fork and can put up a PR if I get somewhere worth sharing. Seems like there are a handful of ways to get the desired behavior.

[ilyachur]

Hi @nbn22385 ,

I added CTest support in PR #61. This PR already was merged to master branch. Please verify that it works for your cases.

Thanks.

[nbn22385]

This is awesome, thank you!

I have experimented with the update and have a couple observations that I will put below. Feel free to close the issue if you feel they are not worthy of making any changes.

---

When running the following scenario:

1. CMake
2. CMakeBuild (runs target all)
3. CTest (runs target test)

The output is correct. However, if I make a code change and run:

1. CMakeBuild (runs target test, unless i specify all)

Only the tests from the prior build run again because the target is still set to test. This may be by design, as it might be unexpected behavior to silently change the target back to all. A way to remedy this would be to manually specify the target, unless you have a better suggestion:

1. CMakeBuild all (runs target all)
2. CTest (runs target test)
3. Make a code change
4. CMakeBuild all
5. CTest
6. repeat

---

Do you think it's a good idea to add the -bar attribute to the CMakeBuild command so that the commands can be chained together? For example (run a build, and if successful, run the tests):

:CMakeBuild | CTest -V

This would only make sense if the CMakeBuild generating a build error returns a failure status, so that the CTest command would not be run. I would like to be able to simulate the unix command make && ctest.

[ilyachur]

@nbn22385

Thank you for investigation. I created PR #63 which should solve the issue.

[ilyachur]

I have merged this PR.

[ilyachur]

I closed the issue. @nbn22385 feel free to reopen if you have any questions