[BUG]: Pipeline fails since upgrade to agent 3.227.2

Question

[BUG]: Pipeline fails since upgrade to agent 3.227.2

Kralizek opened this issue a year ago · 6 comments

What happened?

A pipeline started failing without any changes. These are the relevant parts of the pipeline.

  - script: dotnet tool restore
    displayName: Restore tools
...
  - script: dotnet test backend/MySolution.sln --logger "junit;LogFilePath=$(Build.SourcesDirectory)/outputs/tests/{assembly}/TestResults.xml;MethodFormat=Full" --no-build --no-restore --collect:"XPlat Code Coverage" --results-directory "$(Build.SourcesDirectory)/outputs/coverage"
    displayName: Run tests
...
  - script: dotnet coverage merge outputs/coverage/ -r -f cobertura -o $(coverageFilePath)
    displayName: Merge coverage files

Here is the dotnet-tools.json file

{
  "version": 1,
  "isRoot": true,
  "tools": {
    "dotnet-coverage": {
      "version": "17.8.6",
      "commands": [
        "dotnet-coverage"
      ]
    }
  }
}

Here is the relevant output on the pipelines:

Starting: Restore tools
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.229.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
dotnet tool restore
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/3194b80f-5743-47cf-9f3e-82b8450a245e.sh
Skipping NuGet package signature verification.
Tool 'dotnet-coverage' (version '17.8.6') was restored. Available commands: dotnet-coverage

Restore was successful.
Finishing: Restore tools

Starting: Run tests
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.229.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
dotnet test backend/MySolution.sln --logger "junit;LogFilePath=/home/vsts/work/1/s/outputs/tests/{assembly}/TestResults.xml;MethodFormat=Full" --no-build --no-restore --collect:"XPlat Code Coverage" --results-directory "/home/vsts/work/1/s/outputs/coverage"
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/2c32f82d-4c69-491b-a7b3-2a1137680838.sh

[...]

Attachments:
  /home/vsts/work/1/s/outputs/coverage/a9a1ad8e-9c1a-4015-bea8-db6028dd1390/coverage.cobertura.xml
  /home/vsts/work/1/s/outputs/coverage/d0960b3d-8718-409a-b095-d4ca80346951/coverage.cobertura.xml
Finishing: Run tests

Starting: Merge coverage files
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.229.0
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
Script contents:
dotnet coverage merge outputs/coverage/ -r -f cobertura -o outputs/coverage/cobertura.xml
========================== Starting Command Output ===========================
/usr/bin/bash --noprofile --norc /home/vsts/work/_temp/2006242c-3ddd-47a5-a891-f25d4a750dae.sh
Run "dotnet tool restore" to make the "dotnet-coverage" command available.
##[error]Bash exited with code '1'.
Finishing: Merge coverage files

Versions

Looking at the pipeline logs, I noticed that the agent version changed from 3.277.1 (working) to 3.277.2 (broken).

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

No response

Relevant log output

No response

Answer 1 · 2023-10-16T11:04:46.000Z

Hi @Kralizek, thank you for the feedback. T
he only changes between version 3.227.1 and 3.227.2 is the PR which seems irrelevant to you. I checked on local env and can't reproduce the error.
Could you please check the runner Image and runner provider

Also, please ensure that some of your steps do not modify/remove env variables as well as dotnet itself.

Answer 2 · 2023-10-16T15:30:58.000Z

@DmitriiBobreshev this is the log of the Initialize Job step

Starting: Initialize job
Agent name: 'Hosted Agent'
Agent machine name: 'fv-az162-989'
Current agent version: '3.227.2'
Operating System
Ubuntu
22.04.3
LTS
Runner Image
Image: ubuntu-22.04
Version: 20231005.3.0
Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20231005.3/images/linux/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20231005.3
Runner Image Provisioner
2.0.310.1
Current image version: '20231005.3.0'
Agent running as: 'vsts'
Prepare build directory.
Set build variables.
Download all required tasks.
Downloading task: AzureKeyVault (1.228.0)
Downloading task: CmdLine (2.229.0)
Downloading task: PowerShell (2.229.1)
Downloading task: UseDotNet (2.228.0)
Downloading task: PublishTestResults (2.229.1)
Downloading task: Docker (2.221.0)
Downloading task: SlackNotification (6.0.2)
Checking job knob settings.
   Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES) 
   Knob: AgentToolsDirectory = /opt/hostedtoolcache Source: ${AGENT_TOOLSDIRECTORY} 
   Knob: AgentPerflog = /home/vsts/perflog Source: ${VSTS_AGENT_PERFLOG} 
   Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE) 
   Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT) 
   Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT) 
Finished checking job knob settings.
Start tracking orphan processes.
Finishing: Initialize job

Answer 3 · 2023-10-16T15:32:56.000Z

For sake of completeness, the pipeline installs and uses the .NET SDK 8 but it had successful runs even after .NET 8 RC2 was releases so I didn't think it was a factor

Starting: Install .NET 8 SDK
==============================================================================
Task         : Use .NET Core
Description  : Acquires a specific version of the .NET Core SDK from the internet or the local cache and adds it to the PATH. Use this task to change the version of .NET Core used in subsequent tasks. Additionally provides proxy support.
Version      : 2.228.0
Author       : Microsoft Corporation
Help         : https://aka.ms/AA4xgy0
==============================================================================
Tool to install: .NET Core sdk version 8.x.
Found version 8.0.100-rc.2.23502.2 in channel 8.0 for user specified version spec: 8.x
get-os-distro: Error: Distribution specific OS name and version could not be detected: UName = Linux
Version 8.0.100-rc.2.23502.2 was not found in cache.
Getting URL to download .NET Core sdk version: 8.0.100-rc.2.23502.2.
Detecting OS platform to find correct download package for the OS.
/home/vsts/work/_tasks/UseDotNet_b0ce7256-7898-45d3-9cb5-176b752bfea6/2.228.0/externals/get-os-distro.sh
Primary:linux-x64
Legacy:-x64
Detected platform (Primary): linux-x64
Detected platform (Legacy): -x64
Downloading: https://download.visualstudio.microsoft.com/download/pr/9144f37e-b370-41ee-a86f-2d2a69251652/bc1d544112ec134184a5aec7f7a1eaf9/dotnet-sdk-8.0.100-rc.2.23502.2-linux-x64.tar.gz
Extracting downloaded package /home/vsts/work/_temp/09719952-b627-4acc-84ae-c896d9beea04.
Extracting archive
/usr/bin/tar xC /home/vsts/work/_temp/99345cda-b81e-4423-9b35-554aa8434855 -f /home/vsts/work/_temp/09719952-b627-4acc-84ae-c896d9beea04
Successfully installed .NET Core sdk version 8.0.100-rc.2.23502.2.
Creating global tool path and pre-pending to PATH.
Finishing: Install .NET 8 SDK

Answer 4 · 2023-10-16T16:29:09.000Z

@Kralizek Seems like the problem is not exactly in the dotnet version but in the dotnet-coverage version.

I was able to reproduce the error with dotnet 8 + dotnet-coverage 17.8.6 but seems like it happened because of the dotnet-coverage version after I changed the package version from 17.8.6 to the latest 17.9.1, everything started working fine.

Could you please try to set dotnet-coverage to 17.9.1 in your dotnet-tools.json file?

Answer 5 · 2023-10-16T17:30:29.000Z

@DmitriiBobreshev thanks a lot!

upgrading dotnet-coverage to 17.9.1 actually solved the issue. but I am not super happy that a new tool version can break a build. It makes pinning a version useless.

Answer 6 · 2023-10-17T07:27:00.000Z

@Kralizek, Closing the issue. The problem is not related to the agent/tasks but to dotnet.
Reproduced with clean env on debian:

$: dotnet --info
 .NET SDK:
 Version:   8.0.100-rc.2.23502.2
 Commit:    0abacfc2b6
$: dotnet tool list 
 Package Id           Version      Commands             Manifest
 -----------------------------------------------------------------------------------------
 dotnet-coverage      17.8.6       dotnet-coverage      /home/dmybor/net/dotnet-tools.json
$: dotnet coverage
 Run "dotnet tool restore" to make the "dotnet-coverage" command available.

Please feel free to reopen if you have additional questions, but I believe it is better to create a ticket in the dotnet repository if you want to understand the bug's behaviour.