CI builds are not failing like they ought to when build steps are failing

[hlovdal]

Currently the latest travis build contains:

$ pwsh -command "Invoke-Expression .\tests\Tests.ps1"
New-Item: /home/travis/build/cake-build/resources/tests/Tests.ps1:6
Line |
   6 |      New-Item -Type Directory -Path $testresultFolder -Force
     |                                     ~~~~~~~~~~~~~~~~~
     | Cannot bind argument to parameter 'Path' because it is an
     | empty string.

Execute tests for build.ps1 in powershell
Join-Path: /home/travis/build/cake-build/resources/tests/Tests.ps1:11
Line |
  11 |  … ormat NUnitXml -OutputFile (Join-Path -Path $testresultFolder -ChildP …
     |                                                ~~~~~~~~~~~~~~~~~
     | Cannot bind argument to parameter 'Path' because it is an
     | empty string.

The command "pwsh -command "Invoke-Expression .\tests\Tests.ps1"" exited with 0.

So even though something fails in the script it does return a non-zero result code. In the unix world this is solved by simply adding a set -e to make the script exit on first error with an error code. With powershell there is no similar simple solution.

For internal errors, like the "cannot bind ..." error above, there is something that sort of resembles set -e but without the elegance:

# From https://stackoverflow.com/a/44810914/23118
Set-StrictMode -Version Latest
$ErrorActionPreference = "Stop"
$PSDefaultParameterValues['*:ErrorAction'] = 'Stop'
function ThrowOnNativeFailure {
    if (-not $?) {
        throw 'Native Failure'
    }
}

although that is not complete, in order to catch errors from external commands you have to painstakingly make sure that every single external command invocation is wrapped through an exec function:

# Getting powershell to exit whenever some of the commands fail is by far not as
# straight forward as "set -e" is in bash. Further info:
# https://stackoverflow.com/questions/11450153/powershell-analogue-of-bashs-set-e
# https://stackoverflow.com/questions/10666101/lastexitcode-0-but-false-in-powershell-redirecting-stderr-to-stdout-gives
# https://stackoverflow.com/questions/9948517/how-to-stop-a-powershell-script-on-the-first-error
# https://rkeithhill.wordpress.com/2009/08/03/effective-powershell-item-16-dealing-with-errors/
# https://github.com/PowerShell/PowerShell-RFC/pull/261/files
#
# TL;DR: wrap all external commands inside the exec function, e.g.
#    exec { docker build -t mytag . }

# From https://stackoverflow.com/a/48999101/23118
#
# Taken from psake https://github.com/psake/psake (MIT License)
<#
.SYNOPSIS
  This is a helper function that runs a scriptblock and checks the PS variable $lastexitcode
  to see if an error occcured. If an error is detected then an exception is thrown.
  This function allows you to run command-line programs without having to
  explicitly check the $lastexitcode variable.
.EXAMPLE
  exec { svn info $repository_trunk } "Error executing SVN. Please verify SVN command-line client is installed"
#>
function Exec {
    [CmdletBinding()]
    param(
        [Parameter(Position = 0, Mandatory = 1)][scriptblock]$cmd,
        [Parameter(Position = 1, Mandatory = 0)][string]$errorMessage = ("Error executing command {0}" -f $cmd)
    )
    & $cmd
    if ($lastexitcode -ne 0) {
        throw ("Exec: " + $errorMessage)
    }
}

Similarly, the appveyor builds have green status even though everything currently is failing:

Image: Visual Studio 2017                          4 of 4 failed
Image: Ubuntu                                      2 of 2 failed

They are failing due to build.ps1 missing, so copy from dotnet-framework like I did for the travis build, although fixing that is the minor issue here.

The major issue is that the build scripts should fail when anything goes wrong.

PS I am only reporting this, I do not intend to work on fixing this.