microsoft/winget-cli-restsource

Add-WinGetManifest "Failed to retrieve a proper manifest. Verify and try again." / Get-WinGetManifest "Attempt to convert YAML to JSON failed."

Closed this issue · 2 comments

3limin4t0r commented

Scenario

I've created a WinGet manifest with wingetcreate new '{installer-url}', which resulted in some YAML files. The resulting file structure looks like this:

{manifest-path}\
├╴{package-id}.yaml
├╴{package-id}.installer.yaml
└╴{package-id}.locale.en-US.yaml

When trying to add the manifest to my WinGet source I encountered the exception:

Failed to retrieve a proper manifest. Verify and try again.

Issue

Add-WinGetManifest always throws the exception "Failed to retrieve a proper manifest. Verify and try again."

Add-WingetManifest -FunctionName '{function-name}' -Path '{manifest-path}' -Verbose
# VERBOSE: Populating RepositorySourceLocation property for module Az.Resources.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Resources.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Resources.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Accounts.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Accounts.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Websites.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Websites.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Functions.
# VERBOSE: Populating RepositorySourceLocation property for module Az.Functions.
# VERBOSE: Validating connection to azure, will attempt to connect if not already connected.
# VERBOSE: Connected to Azure
# VERBOSE: No Subscription Name or Subscription Id provided. Will test connection to default Azure Subscription
# VERBOSE: Determines the Azure Function Resource Group Name
# WARNING: App settings have been redacted. Use the Get-AzFunctionAppSetting cmdlet to view them.
# VERBOSE: Verifying that the Azure Resource {function-name} exists..
# WARNING: App settings have been redacted. Use the Get-AzFunctionAppSetting cmdlet to view them.
# VERBOSE: Azure Resources:
#            Azure Function Exists:       True
#            Azure Resource Group Exists: True
# VERBOSE: Retrieving a copy of the app Manifest file for submission to WinGet source.
# VERBOSE: Path pointed to a directory, found 0 JSON files, and 3 YAML files.
# VERBOSE: Single YAML has been found in the specified directory.
# VERBOSE: YAML Files have been found in the target directory. Building a JSON manifest with found files.
# VERBOSE: Other
# VERBOSE: Attempt to convert YAML to JSON failed.
# VERBOSE: Returned Manifest from YAML file:
# VERBOSE: Returning (0) manifests based on search.
# VERBOSE:
# 
# 
# 
# Exception: Documents\PowerShell\Modules\Microsoft.WinGet.Source\1.1.20230906\Library\Add-WinGetManifest.ps1:97
# Line |
#   97 |              throw "Failed to retrieve a proper manifest. Verify and t …
#      |              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#      | Failed to retrieve a proper manifest. Verify and try again.

Steps to reproduce

I've followed the "PowerShell pre-Requisites" instructions listed in Create a Windows Package Manager REST source.

I installed the PowerShell module somewhat following the instructions. I want the module to be able to automatically load, so I've extracted the .zip file to my Documents\PowerShell\Modules\ folder instead. (I did also try to 100% accurately follow the "Download and install the PowerShell module" instructions, but this yielded the same exception.) For easy reproduction I've written the installation instructions as PowerShell script, so you can easily install the module the same way:

Installation steps 
# Assuming you've completed the PowerShell pre-Requisites
$PSModuleName = 'Microsoft.WinGet.Source'
$ReleaseVersion = '1.1.20230906'
$PSModuleZipName = 'WinGet.RestSource-Winget.PowerShell.Source'

$PSModulesPath = "$([Environment]::GetFolderPath("MyDocuments"))\PowerShell\Modules"
$PSModuleRoot = "$PSModulesPath\$PSModuleName\$ReleaseVersion"
$DownloadsPath = (New-Object -ComObject Shell.Application).NameSpace('shell:Downloads').Self.Path

# Download PowerShell module zip file
Invoke-WebRequest `
  -Uri "https://github.com/microsoft/winget-cli-restsource/releases/download/$ReleaseVersion/$PSModuleZipName.zip" `
  -OutFile "$DownloadsPath\$PSModuleZipName.zip"

# Extract the zip contents to Documents\PowerShell\Modules\Microsoft.WinGet.Source\{version}\*
New-Item -Path $PSModulesPath -Name $PSModuleName -ItemType directory
Expand-Archive -LiteralPath "$DownloadsPath\$PSModuleZipName.zip" -DestinationPath "$PSModulesPath\$PSModuleName"
Rename-Item -LiteralPath "$PSModulesPath\$PSModuleName\$PSModuleZipName" -NewName $ReleaseVersion

# Unblock files downloaded from the internet
Get-ChildItem -LiteralPath $PSModuleRoot -Recurse | Unblock-File

# Update version in module definition file to match the path (see issue #255)
(Get-Content -LiteralPath "$PSModuleRoot\$PSModuleName.psd1") `
  -replace "ModuleVersion = '0\.1\.0'", "ModuleVersion = '$ReleaseVersion'" `
  | Set-Content -LiteralPath "$PSModuleRoot\$PSModuleName.psd1"

After the installation I restarted my PowerShell window and ran Connect-AzAccount and Set-AzContext again.

I then created a new WinGet source using the command:

New-WinGetSource `
  -Name '{function-name}' `
  -ResourceGroup '{resource-group-name}' `
  -Region 'westeurope' `
  -ImplementationPerformance 'Demo' `
  -ShowConnectionInstructions `
  -Verbose

Up until this point everything worked as it was supposed to.

I then tried to upload a manifest with the command:

Add-WingetManifest -FunctionName '{function-name}' -Path '{manifest-path}' -Verbose

This threw the exception "Failed to retrieve a proper manifest. Verify and try again." as mentioned above.

Expected behaviour

I expect the the command Add-WingetManifest to upload an installer manifest to my WinGet source.

Actual behaviour

Add-WingetManifest throws an exception.

Debugging

I've had a look at the line the exception is pointing to and saw the following code:

winget-cli-restsource/Tools/PowershellModule/src/Library/Add-WinGetManifest.ps1

Lines 91 to 98 in af2275e

###############################
## Gets the content from the Package Manifest (*.JSON, or *.YAML) file for posting to REST source.
Write-Verbose -Message "Retrieving a copy of the app Manifest file for submission to WinGet source."
$ApplicationManifest = Get-WinGetManifest -Path $Path
if(!$ApplicationManifest) {
Write-Verbose "$ApplicationManifest`n`n`n"
throw "Failed to retrieve a proper manifest. Verify and try again."
}

Meaning that Get-WinGetManifest does not return any output. When running the command by itself we get the following output:

Get-WinGetManifest -Path '{manifest-path}' -Verbose
# VERBOSE: Path pointed to a directory, found 0 JSON files, and 3 YAML files.
# VERBOSE: Single YAML has been found in the specified directory.
# VERBOSE: YAML Files have been found in the target directory. Building a JSON manifest with found files.
# VERBOSE: Other
# VERBOSE: Attempt to convert YAML to JSON failed.
# VERBOSE: Returned Manifest from YAML file:
# VERBOSE: Returning (0) manifests based on search.

Like shown above this does not throw an exception, but simply lists "Attempt to convert YAML to JSON failed." as verbose output and doesn't return anything.

I then had a look in Get-WinGetManifest.ps1 to see what causes this output and found the following:

winget-cli-restsource/Tools/PowershellModule/src/Library/Get-WinGetManifest.ps1

Lines 274 to 299 in af2275e

## If the path resolves to a YAML file
".yaml" {
## Directory - *.yaml files included within.
if($WinGetDesktopAppInstallerLibLoaded) {
Write-Verbose -Message "YAML Files have been found in the target directory. Building a JSON manifest with found files."
if($Json){
Write-Verbose "JSON"
$Return += [Microsoft.WinGet.RestSource.PowershellSupport.YamlToRestConverter]::AddManifestToPackageManifest($Path, $JSON.GetJson());
}
else{
Write-Verbose "Other"
try {
$Return += [Microsoft.WinGet.RestSource.PowershellSupport.YamlToRestConverter]::AddManifestToPackageManifest($Path, "");
}
catch {
Write-Verbose "Attempt to convert YAML to JSON failed."
}
}
Write-Verbose -Message "Returned Manifest from YAML file: $($Return.PackageIdentifier)"
}
else {
Write-Error -Message "Unable to process YAML files. Re-import the module to reload the required dependencies." -Category ResourceUnavailable
}
}

In the given scenario $WinGetDesktopAppInstallerLibLoaded was truthy and no -Json parameter was provided so $Json was falsy. You can see "Other" and "Attempt to convert YAML to JSON failed." in the output above. Meaning that the following line is the offender:

$Return += [Microsoft.WinGet.RestSource.PowershellSupport.YamlToRestConverter]::AddManifestToPackageManifest($Path, "");

However this stil does not really explain what the issue is because the try-catch catches all exceptions, obfuscating the real exception as a result. After commenting the try-catch code we get the following results:

Get-WinGetManifest -Path '{manifest-path}' -Verbose
# VERBOSE: Path pointed to a directory, found 0 JSON files, and 3 YAML files.
# VERBOSE: Single YAML has been found in the specified directory.
# VERBOSE: YAML Files have been found in the target directory. Building a JSON manifest with found files.
# VERBOSE: Other
# MethodInvocationException: Documents\PowerShell\Modules\Microsoft.WinGet.Source\1.1.20230906\Library\Get-WinGetManifest.ps1:286
# Line |
#  286 |  …             $Return += [Microsoft.WinGet.RestSource.PowershellSupport …
#      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#      | Exception calling "AddManifestToPackageManifest" with "2" argument(s): "Unable to load DLL 'WinGetUtil.dll' or
#      | one of its dependencies: Kan opgegeven module niet vinden. (0x8007007E)"
# VERBOSE: Returned Manifest from YAML file:
# VERBOSE: Returning (0) manifests based on search.

This seems to reveal the real problem:

Unable to load DLL 'WinGetUtil.dll' or one of its dependencies: Kan opgegeven module niet vinden. (0x8007007E)

Dutch "Kan opgegeven module niet vinden" translates to "Cannot find specified module" in English.

I downloaded Windows Process Monitor (aka procmon) to check where the system was looking for this file. The module directory is checked two times:

Screenshot Windows Process Monitor

screenshot procmon

Library\WinGet.RestSource.PowershellSupport\WinGetUtil.dll             | NAME NOT FOUND
Library\WinGet.RestSource.PowershellSupport\win-x64\WinGetUtil.dll.dll | PATH NOT FOUND

The actual file is located at (depending on architecture):

Library\WinGet.RestSource.PowershellSupport\runtimes\win-x64\native\WinGetUtil.dll
Library\WinGet.RestSource.PowershellSupport\runtimes\win-x86\native\WinGetUtil.dll

So I'm guessing the Library\WinGet.RestSource.PowershellSupport\runtimes\{architecture}\native is not correctly added as load path somehow.

Dirty solution (for users)

For those stumbling upon this issue, you can fix it by copy/pasting the files from Library\WinGet.RestSource.PowershellSupport\runtimes\{architecture}\native\* into Library\WinGet.RestSource.PowershellSupport\* since the system does look there.

Windows x64 PowerShell script 
# see installation instructions for value of $PSModuleRoot
Get-ChildItem "$PSModuleRoot\Library\WinGet.RestSource.PowershellSupport\runtimes\win-x64\native" -File `
  | Copy-Item -Destination "$PSModuleRoot\Library\WinGet.RestSource.PowershellSupport" -ErrorAction Ignore # skip conflict

After copying the files, restart your PowerShell, run Connect-AzAccount and Set-AzContext again. Now Add-WinGetManifest should work.

Environment

PowerShell x64 7.4.5

[winget --info]
Windows Package Manager v1.8.1911
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.22631.4317
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.23.1911.0

[wingetcreate --version]
WingetCreateCLI 1.6.5.0+7dfac18035d9884aa5e6eae0bcb59883b4ec4dc0
denelon commented

We need to update the library to the latest schema version to fix this. The work is in progress.

yao-msft commented

This should be fixed with latest 1.9.1 release.