check() does not stop on failed tests

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

This code

lucode2/R/check.R

Lines 31 to 40 in 009b94c

	########### Run tests ###########
	# run tests in a separate R session so test results are independent of anything set in the current R session
	testResults <- callr::r(function(lib) {
	withr::local_options(crayon.enabled = TRUE)
	return(devtools::test(lib))
	}, args = list(lib), show = TRUE)
	if (sum(testResults[["failed"]]) > 0 || any(testResults[["error"]])) {
	stop("Some tests failed, please fix them first.")
	}

doesn't work.

I have the tests of the remind2 package failing for some time (on some lusweave/pdflatex issue), and buildLibrary()happily ignore that. (Not that I'm eager to have remind2 fail on me several times a day, but … boyscout rule.)

Restarting R session...

> (lib <- getwd())
[1] "/home/pehl/PIK/pik-piam/remind2"
> { testResults <- callr::r(function(lib) {
+       withr::local_options(crayon.enabled = TRUE)
+       return(devtools::test(lib))
+   }, args = list(lib), show = TRUE)
+ 
+   if (sum(testResults[["failed"]]) > 0 || any(testResults[["error"]])) {
+       stop("Some tests failed, please fix them first.")
+   }
+ }
ℹ Loading remind2
Loading required package: magclass
Registered S3 method overwritten by 'quantmod':
  method            from
  as.zoo.data.frame zoo 
ℹ Testing remind2
✓ | F W S  OK | Context
|
Executing reportFEnvGDX2mif                                                     
\
| | 1       0 | convGDX2mif                                                     
x | 1       0 | convGDX2mif [289.6s]
────────────────────────────────────────────────────────────────────────────────
Failure (test-convGDX2mif.R:95:3): Test if REMIND reporting is produced as it should and check data integrity
file.exists(scenarioComparisonPath) is not TRUE

`actual`:   FALSE
`expected`: TRUE 
────────────────────────────────────────────────────────────────────────────────
|
══ Results ═════════════════════════════════════════════════════════════════════
Duration: 289.6 s

[ FAIL 1 | WARN 0 | SKIP 0 | PASS 0 ]
|
Frustration is a natural part of programming :)

>

The referenced fields do not exist

> testResults[["failed"]]
NULL
> testResults[["error"]]
NULL

so the condition is always FALSE and no error thrown.

It seems as if testResult is a list of all test files and their results, which again are lists of all test_that calls and their final or terminating conditions

> str(testResults)
List of 1
 $ :List of 7
  ..$ file   : chr "test-convGDX2mif.R"
  ..$ context: chr "convGDX2mif"
  ..$ test   : chr "Test if REMIND reporting is produced as it should and check data integrity"
  ..$ user   : num 286
  ..$ system : num 3.42
  ..$ real   : num 290
  ..$ results:List of 1
  .. ..$ :List of 6
  .. .. ..$ message    : chr "file.exists(scenarioComparisonPath) is not TRUE\n\n`actual`:   \033[32mFALSE\033[39m\n`expected`: \033[32mTRUE\033[39m "
  .. .. ..$ srcref     : 'srcref' int [1:8] 95 3 95 50 3 50 95 95
  .. .. .. ..- attr(*, "srcfile")=Classes 'srcfilecopy', 'srcfile' <environment: 0x561dcb2d9e50> 
  .. .. ..$ trace      : NULL
  .. .. ..$ start_frame: int 49
  .. .. ..$ end_frame  : num 51
  .. .. ..$ test       : chr "Test if REMIND reporting is produced as it should and check data integrity"
  .. .. ..- attr(*, "class")= chr [1:4] "expectation_failure" "expectation" "error" "condition"
 - attr(*, "class")= chr "testthat_results"

Something like

> any(sapply(testResults, 
+            function(t) { 
+                sapply(t$result, inherits, what = "error") 
+                }
+            ))
[1] TRUE

might work.