nkdAgility/azure-devops-migration-tools

[Bug]: Pipelines migration: AzureDevOpsPipelineProcessor System.NullReferenceException when REST API does not return repo properties of a build pipeline

Bart80-1 opened this issue · 5 comments

Bart80-1 commented

Version

  • I confirm that I am using the latest version

Source Version

Azure DevOps Service

Target Version

Azure DevOps Service

Relevant configuration

{
  "ChangeSetMappingFile": null,
  "Source": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://dev.azure.com/source_org/",
    "Project": "source_proj",
    "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "Prompt",
    "PersonalAccessToken": "",
    "PersonalAccessTokenVariableName": "",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    },
    "CollectionName": "https://dev.azure.com/nkdagility-preview/"
  },
  "Target": {
    "$type": "TfsTeamProjectConfig",
    "Collection": "https://dev.azure.com/target_org/",
    "Project": "target_proj",
    "ReflectedWorkItemIDFieldName": "Custom.ReflectedWorkItemId",
    "AllowCrossProjectLinking": false,
    "AuthenticationMode": "AccessToken",
    "PersonalAccessToken": "<PAT>",
    "PersonalAccessTokenVariableName": "",
    "LanguageMaps": {
      "AreaPath": "Area",
      "IterationPath": "Iteration"
    },
    "CollectionName": "https://dev.azure.com/nkdagility-preview/"
  },
  "FieldMaps": [],
  "GitRepoMapping": null,
  "LogLevel": "Information",
  "CommonEnrichersConfig": null,
  "Processors": [
    {
      "$type": "WorkItemMigrationConfig",
      "Enabled": false,
      "ReplayRevisions": true,
      "PrefixProjectToNodes": false,
      "UpdateCreatedDate": true,
      "UpdateCreatedBy": true,
      "WIQLQueryBit": "AND  [Microsoft.VSTS.Common.ClosedDate] = '' AND [System.WorkItemType] NOT IN ('Test Suite', 'Test Plan','Shared Steps','Shared Parameter','Feedback Request')",
      "WIQLOrderBit": "[System.ChangedDate] desc",
      "LinkMigration": true,
      "AttachmentMigration": true,
      "AttachmentWorkingPath": "c:\\temp\\WorkItemAttachmentWorkingFolder\\",
      "FixHtmlAttachmentLinks": false,
      "SkipToFinalRevisedWorkItemType": false,
      "WorkItemCreateRetryLimit": 5,
      "FilterWorkItemsThatAlreadyExistInTarget": false,
      "PauseAfterEachWorkItem": false,
      "AttachmentMaxSize": 480000000,
      "AttachRevisionHistory": false,
      "LinkMigrationSaveEachAsAdded": false,
      "GenerateMigrationComment": true,
      "WorkItemIDs": null,
      "MaxRevisions": 0,
      "UseCommonNodeStructureEnricherConfig": false,
      "NodeBasePaths": null,
      "AreaMaps": {
		"source_proj": "target_proj\\source_proj"
	  },
      "IterationMaps": {
		"source_proj":"target_proj\\source_proj"
	  },
      "MaxGracefulFailures": 0,
      "SkipRevisionWithInvalidIterationPath": false,
      "SkipRevisionWithInvalidAreaPath": false,
      "ShouldCreateMissingRevisionPaths": true
    },
	{
	  "$type": "AzureDevOpsPipelineProcessorOptions",
	  "Enabled": true,
	  "MigrateBuildPipelines": true,
	  "MigrateReleasePipelines": true,
	  "MigrateTaskGroups": true,
	  "MigrateVariableGroups": true,
	  "MigrateServiceConnections": true,
	  "BuildPipelines": null,
	  "ReleasePipelines": null,
	  "RepositoryNameMaps": {},
	  "ProcessorEnrichers": null,
	  "SourceName": "Source",
	  "TargetName": "Target"
	}	
  ],
  "Version": "14.4",
  "workaroundForQuerySOAPBugEnabled": false,
  "WorkItemTypeDefinition": {
    "Product Backlog Item": "User Story",
	"Impediment": "Issue"
  },
  "Endpoints": {
    "InMemoryWorkItemEndpoints": [
      {
        "Name": "Source",
        "EndpointEnrichers": null
      },
      {
        "Name": "Target",
        "EndpointEnrichers": null
      }
    ], 
	"AzureDevOpsEndpoints": [
		{
		  "name": "Source",
		  "$type": "AzureDevOpsEndpointOptions",
		  "Organisation": "https://dev.azure.com/source_org/",
		  "Project": "source_proj",
		  "AuthenticationMode": "AccessToken",
		  "AccessToken": "<PAT>",
		  "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
		  "EndpointEnrichers": null          
		},
		{
		  "Name": "Target",
		  "$type": "AzureDevOpsEndpointOptions",
		  "Organisation": "https://dev.azure.com/target_org/",
		  "Project": "target_proj",
		  "AuthenticationMode": "AccessToken",
		  "AccessToken": "<PAT>",
		  "ReflectedWorkItemIdField": "Custom.ReflectedWorkItemId",
		  "EndpointEnrichers": null
		}
    ]	
  }
}

Relevant log output

[10:26:08 INF] [v14.4.6] 2 of 2 source BuildDefinition(s) are going to be migrated..
[10:26:08 FTL] [v14.4.6] Error while running AzureDevOpsPipelineProcessor
System.NullReferenceException: Object reference not set to an instance of an object.
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.MapRepositoriesInBuidDefinition(IEnumerable`1 sourceRepositories, IEnumerable`1 targetRepositories, BuildDefinition definitionToBeMigrated) in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 393
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<CreateBuildPipelinesAsync>d__17.MoveNext() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 312
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.<MigratePipelinesAsync>d__9.MoveNext() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 95
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MigrationTools.Processors.AzureDevOpsPipelineProcessor.InternalExecute() in D:\a\1\s\src\MigrationTools.Clients.AzureDevops.Rest\Processors\AzureDevOpsPipelineProcessor.cs:line 49
   at MigrationTools.Processors.Processor.Execute() in D:\a\1\s\src\MigrationTools\Processors\Processor.cs:line 106

What happened?

Above error happened when trying to migrate 2 build pipelines.
After I did some digging in VS debug I found out that this line causes the exception (AzureDevopsPipelineProcessor.cs Line 306):
var sourceConnectedServiceId = definitionToBeMigrated.Repository.Properties.ConnectedServiceId;

It supposes that the definition to be migrated pipeline's repository property has a property called "Properties". But the AzD Rest API did not return one. See Powershell code:

$pl3=Invoke-RestMethod -uri "https://dev.azure.com/source_org//source_proj/_apis/build/definitions/3" -Method Get -Headers $headerssrc
PS C:\Users\usr> $pl3.repository


id                 : <ID>
type               : TfsGit
name               : source_proj
url                : https://dev.azure.com/source_org/source_proj/_git/source_repo
defaultBranch      : refs/heads/main
clean              : 
checkoutSubmodules : False

I was able to work around it by tinkering with the trigger settings in the source pipelines (for those interested: see https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/pipeline-options-for-git?view=azure-devops&tabs=yaml ). But I guess this could be improved?
Tx :)

Debug in Visual Studio

  • Visual Studio Debug
tomfrenzel commented

Hi @Bart80-1

thanks for already debugging the error! This exact same issue has been around for a while but i never got it reproduced.
Can you let me know how the yaml looked at the beginning when the issue happened and how it looked when you got the migrator to work?

Bart80-1 commented

Hi @Bart80-1

thanks for already debugging the error! This exact same issue has been around for a while but i never got it reproduced. Can you let me know how the yaml looked at the beginning when the issue happened and how it looked when you got the migrator to work?

Hi,

Do you mean the yaml of the pipeline? I did not make any changes in that.

tomfrenzel commented

Oh alright, i thought you changed the triggers inside the yaml file.

I saw there is an UI to change triggers of the Pipeline so you probably mean that right?
If so, please let me know how those triggers were configured initially and how they were configured at the end where the error didn't occur anymore

Bart80-1 commented

Oh alright, i thought you changed the triggers inside the yaml file.

I saw there is an UI to change triggers of the Pipeline so you probably mean that right? If so, please let me know how those triggers were configured initially and how they were configured at the end where the error didn't occur anymore

Yes, I meant that I made changes in the Triggers section of the pipeline.
I don't remember exactly 100% but if I recall correctly, in the Triggers section, Pipeline tab, "Get sources" step, I changed the "Clean" combobox from True to False, saved the change (without queuing), then changed it to True again. The result was that the "Properties" property of the Repository item were present, where before they weren't.
The ability to simulate this issue will depend on what MS sends back in the API call. I guess the most reliable way to get the situation in which the Properties property is not present is to have a newly created pipeline - and maybe, in a newly created Devops project...

tomfrenzel commented

Thanks for letting me know! Unfortunately, tho, I wasn't able to reproduce it on my side. I've been trying a whole lot of different settings but I always get the properties object.
I'll probably make to code more failsafe regarding this behaviour so the migrator doesn't just crash but rather log an error

  • 👍1