Program 'winget.exe' failed to run: The file cannot be accessed by the system.At line:1 char:1 [Running Virtual Machine via SSH]

[ranavale]

Brief description of your issue

I created a VM and I installed winget using latest release on its github page, then I connected to my VM from VSCode SSH remote connection. I am trying to run the winget from my local remote-VSCode (connected to VM via SSH).
Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Steps to reproduce

run comment "winget"

Expected behavior

winget to work
This is output when I login to my VM and run winget

Actual behavior

Program 'winget.exe' failed to run: The file cannot be accessed by the system. At line:1 char:1 [Running Virtual Machine via SSH]

Environment

Winget Package

Windows Package Manager v1.0.11692
Copyright (c) Microsoft Corporation. All rights reserved.

Windows specifications:

Edition: Windows 10 Enterprise for Virtual Desktops
Version: 21H1
OS Build: 19043.1237

App Installer

Version: 1.12.11692.0

[jedieaston]

This is a similar issue to #995.

If it helps the team narrow it down, it doesn't just happen with winget.exe, but with any App Execution Alias (Windows Terminal registers wt.exe and it doesn't work either). It happens over PowerShell Remoting and SSH (via the Windows built-in OpenSSH Server), and it happens even if winget works completely fine on the local system from a shell. It also happens if you are running Windows inside a docker container and attempting to execute winget via docker exec.

This is probably not a winget problem then, but @denelon can you ask whoever manages the App Execution Aliases stuff to look into it? It basically blocks using winget remotely unless you get creative.

Proofsies:

PS E:\> Enter-LabPSSession wingettest

[wingettest]: PS C:\Users\Administrator\Documents> whoami 
wingettest\administrator

[wingettest]: PS C:\Users\Administrator\Documents> winget --info
Program 'winget.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python3
Program 'python3.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> python
Program 'python.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> wt 
Program 'wt.exe' failed to run: The file cannot be accessed by the system.
    + CategoryInfo          : ResourceUnavailable: (:) [], ApplicationFailedException
    + FullyQualifiedErrorId : NativeCommandFailed
 

[wingettest]: PS C:\Users\Administrator\Documents> # Aw, beans!

[JohnMcPMS]

@jedieaston , my guess is that this is because PowerShell is attempting to read the file before launching it. The way that AEA is implemented in the OS makes the shells not work if they attempt to read it, since it isn't the actual file (nor any kind of file really).

There are a few solutions:

1. We change a few things about the winget package so that the actual binary is launchable directly. As I recall, there was some work done in more recent Windows builds to enable that. That would possibly enable a workaround that an Invoke-WinGet PowerShell function could be written to handle. All speculative on my part; only helps those who seek out the solution but I imagine this type of remoting falls into "people who seek out solutions" territory anyway. Only works on "newer" (TBD) OS.
2. PowerShell (and docker, and anyone else) updates to not attempt to read from the "executable" or at least handle a failure to do so by simply executing it. Requires multiple teams to do work; will keep coming up.
3. The OS is updated to allow the file to be read through the AEA. Only works in some future timeframe, but would hopefully solve the issue for everything. I think might actually be possible (I was party to the original implementation but am no longer on the owning team). Requires getting that work approved, and it is not a bug level change.

Our changes for 1 are actually pretty easy, but again it would only (potentially) make something like this work:

C:\Program Files\WindowsApps\Microsoft.DesktopAppInstaller_1.16.12581.0_x64__8wekyb3d8bbwe> winget.exe install foo

And not the default path winget command.

[jedieaston]

Obviously 3 would be wonderful, but if we have to settle for 1 that’d still be good enough to get winget working for now in these scenarios.

By TBD, do you mean that the OS changes necessary for 1 aren’t in stable yet? The only thing I know of that rings a bell with that is Invoke-CommandInDesktopPackage and that doesn’t seem to work either.

[JohnMcPMS]

Agreed on 3, I think there is a case for it but if it is going to take tens of hours of work then there needs to be a justification that meets the bar. Need to ensure that it is actually the problem first, then we can put it on the backlog. But even if it was a go today, I don't think we would see it released for at least a year and then it wouldn't be serviced back to older Windows versions. We need the short term solution too.

By TBD I meant more "to be looked up because I remember seeing the PR but I have no idea when it was released, or if it even has been yet". This was work specifically to enable creating the process directly on the package file in certain instances. Normally packaged processes need to be launched by special API to enable all of packagy goodness (don't @ me), but this provided the ability to CreateProcess against the file and have it invoke the special API for you.

[Deleted user]

ah, I see everyone is busy with App Execution Aliases , but that's not the issue.
the issue is MSIX packaged apps. MSIX packages can't be run over SSH or WinRM , even worse one user can't launch the app as another user. MSIX doesn't support per machine class registration.

[jedieaston]

But if I SSH as myself into the system, the MSIX apps I've installed as myself work. The AEAs don't though.

[Deleted user]

TL;DR : win32 OpenSSH port might want to implement AEAs/NTFS reparse points handling mechanism. I believe, @JohnMcPMS you as a MS employee, your words/winget teams words will have more weight to the win32-openssh team than ours. people even have pointed out where the fix might be needed : https://github.com/PowerShell/openssh-portable/blob/0b73c4636d38f0c69424721d52d0e7752db99c81/contrib/win32/win32compat/fileio.c#L1124

Does it ? 🤔
No MSIX (or Appx) packages regardless of installation medium (loose appxmanifest.xml installation or through AppInstaller installation or directly through Store Installation) run over SSH, here on my machine.

here's a screenshot of me from a non-elevated shell sshing to myself and trying to launch a MSIX package over ssh (eg: winget)

crap, let me dig a little

Update: same result regardless of specified locations or AEAs. It turns out AEA is the MSIX Problem, which could be anything among Symbolik Link/Junction/Mount Point/Reparse Point. is indeed a Special NTFS Reparse Point. specified path(%program files%\windowsapps\CrapFolder\crap.exe launches fine over SSH)

[Masamune3210]

Here you are trying to use a AEA, proving the point. Winget should work if you drill down into the correct package location and find the specific exe

[Deleted user]

Here you are trying to use a AEA

yeah, that's right ~\appdata\local\microsoft\windowsapps is the AEA folder.

[Masamune3210]

Oooh i see, I didnt realize that AEA's were plopped in a specific folder, figured they were just namespace links or some other magic. Forgive the comment

[jedieaston]

@denelon Do you know how winget is executed during the Installation Validation step of the winget-pkgs pipeline? Presumably it isn't via PowerShell Remoting or SSH due to this bug.

[JohnMcPMS]

@mshaikhcool , do you have an issue that I could comment on? I don't really understand why these remoting shell type operations need to read the target "exe", but I can certainly comment on what could be done in this case. I'm not sure that the internal data of the AEA reparse point are public, but the type certainly is (IO_REPARSE_TAG_APPEXECLINK).

@jedieaston , @chuckferring would know for sure but I think it is part of a script run on the machine as remoting would not be allowed for security reasons.

[ChuckFerring-zz]

@jedieaston, you are correct. During validation the script is run on the validation machine. It is not done via PowerShell remoting or SSH.

[matsmcp]

May I suggest
4. Repackage with a classic MSI/EXE instead of using "modern" packaging?
It would also fix the Issue with non internet connected installations at the same time

[roland-5]

I can confirm, with new preview version, I was able to use ssh on virtual machine with windows, update and install applications.

[denelon]

Can anyone else confirm? If this is fixed, I would like to close this issue.

[ranavale]

I am still seeing the issue,
(Note: I am connecting to my VM using VSCode SSH feature)

[roland-5]

I use Arch Linux. As my terminal Alacritty, use KVM for virtualization, and installed Windows 11 and use this for ssh server in Windows 11 VM.

Hmm. I created new VM with Windows 11 and using in this VM Windows terminal I was able to ssh log in to first Windows 11 VM and without problem installed some programs (vim, libreoffice).

EDIT.
OK. I didn't saw you exactly use VSCode. I previously tried installing remotely apps, and have the same warning like you. As I see, VSCode have this problem still. Sorry for misunderstanding.

[denelon]

@ranavale does this fail only with SSH in VSCode? I'm not sure what the differences are between that and the ssh server @roland-rollo is using.

[jedieaston]

It fails with any ssh/remoting/docker exec session due to the issue that @JohnMcPMS described around AEAs not working. Extracting AppInstallerCLI.exe and the requisite dependencies (resources, dlls, etc) are the only way I've gotten it to work so far.

[A9G-Data-Droid]

I'm confused by this whole AEA problem when it seems like filesystem hardlinks would be the solution. Why invent some new type of file link that doesn't work when filesystem hardlinks have worked as advertised since the early days of Unix?

[Deleted user]

You will be able understand once you start to see the world without your "Unix and Unix Clones" Glass.

see #1838 (reply in thread)

mainly, it's an issue because of the lack of interest of SSH/WinRM/Docker client programs.

[DylanWoodrow]

I am also still having this issue with build 1.17.3411.0 on 21H2. Notably when running Powershell as admin.

[roland-5]

Hmm, I tried this at computers with Windows 10 at work, and they have this same error message as in ranavale and ghost screenshots.
EDIT.
I tried few Windows 10 builds, and on everyone is this problem. On Windows 11 is not problem.

[dawin-knipp]

@DylanWoodrow can confirm, happens to me even on my local machine, no ssh / remoting involved. When running as admin it fails, when running as currently logged in user, it works (but can't make real use of winget as regular user - UAC on every updated package). Also doesn't matter if I use powershell or cmd, same result (with different error message: The system cannot execute the specified program).

[Deleted user]

@dawin-knipp That's one of the major limitations of the MSIX packaging format, has nothing to do with winget-itself. see #1841 (comment)

[A9G-Data-Droid]

@dawin-knipp That's one of the major limitations of the MSIX packaging format, has nothing to do with winget-itself. see #1841 (comment)

So can we get an MSI installer? It's a hard requirement until MSIX packages are fixed.

[Deleted user]

IDK. That's been asked like a thousand times before. To be brutally honest, they are reluctant to ditch MSIX.

[jedieaston]

It's not really MSIX that is the problem, it's the use of a App Execution Alias. You can create MSIX packages that plop "real" executables in the right places, but that comes with other tradeoffs (for one, you'll have a bunch of copies of winget installed under every profile instead of one in the system directory).

[A9G-Data-Droid]

Naturally, the installer would need to be configured properly. There needs to be two options:

1. Per user install
2. Admin install

If you're an Admin trying to update all software in the system, you obviously need admin rights and a system install. The current user install may have it's uses but I don't know what they are. Doesn't work for my use case, which is to use it like a Windows package manager. (Wasn't that the point?)

[JohnMcPMS]

Since other things lead me to come back to this issue, I will add that I have changed the name of the executable file in the package to be winget.exe (a while ago, so it should be in the latest public release) and thus potential option 1 for solving the issue. Yes, you would need to be remoted in on an account that has the package registered, but beyond that I think that directly running winget.exe from its location in the package should work. Unfortunately, that requires knowing the location of the package, but that is fairly easily solved:

PS> $wingetPath = Join-Path (Get-AppxPackage Microsoft.DesktopAppInstaller).InstallLocation "winget.exe"
PS> Invoke-Expression "& '$wingetPath'"
Windows Package Manager (Preview) v1.4.2161-preview
...

Not quite so clean as winget.exe or actual PowerShell, but you could wrap it in a function or script.

[Luiz-Monad]

You will be able understand once you start to see the world without your "Unix and Unix Clones" Glass.

see #1838 (reply in thread)

mainly, it's an issue because of the lack of interest of SSH/WinRM/Docker client programs.

what a ruse .
"unix of church glass", everything is a untyped stream of text without any metadata. bacause parsing text is fun and we are in 1960...

That was about the lack of powershell support (waiting for it). I love powershell, I even install it on linux because I don't like working with texts and "awk" and stupid unstructured data tools.
In an ideal world, an application would be a group of classes that are grouped into either a graphical user interface or a command line-rich software like powershell, no one should deal with unstructured text.

I'm confused by this whole AEA problem when it seems like filesystem hardlinks would be the solution.

It would ideally be a symlink , hardlinks are hazardous. NTFS allow symlinks since Vista (https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createsymboliclinka) .
I know that UWP seems to run on what's basically a "chroot" (to use Unix terms, its a "silo"), so there must be a reason for AEA

What ended up happening is AEA is basically a ".lnk" file and those only work on "explorer.exe".
Who would have tough they would make the same mistake again... but here we are.

At least make the start shell API work on those then (I mean, this API https://learn.microsoft.com/en-us/windows/win32/api/processthreadsapi/nf-processthreadsapi-createprocessw). Perhaps it does work and its my system that's hosed, or perhaps I need to install the APPX on the system-wide account, who knows, all I know is that explorer is able to launch it, so I must also be able to launch it using the "ShellExecute".

That elevation is connecting to powershell over OpenSSH.

I think the real problem is not only knowing the real directory by entering the reparse point, but also permissions.

Perhaps a simpler solution is to provide "shims" that do the proper calls to the proper RPC service.
So shells don't require any modification, just don't use the WEA misfeature.
https://www.tiraniddo.dev/2019/09/overview-of-windows-execution-aliases.html

Just throw them in some user's public folder and ask the user to set its path to that. (incidentally "choco" seems to work)

[A9G-Data-Droid]

These per-user-installed UWP packages are analogous to flatpak or snaps in Linux. It seems disastrous to deploy the package manager itself in this fashion. In both a flatpak or a snap you first install the package manager at the system level, using the system package manager. Microsoft seems to have ignored all prior art here and failed to come up with a workable alternative.

You can't skip the system level package manager. That needs to come first. Then you can install the UWP package manager (the windows store stuff) using the system package manager. All the problems I have seen with winget stem from this disconnect. They could be solved by a system level package manager that comes with the OS and can be used to install\repair winget.

[exoosh]

Ran into the exact same issue right now when winget worked last week.

I can rule out that this is about the execution aliases, because WinDbgX.exe and wt.exe work as expected. For further context, I run this from an elevated (using SuRun) prompt with a member of the BUILTIN\Administrators alias, but not BUILTIN\Administrator.

The exact same scenario worked last week, but that was before the latest Windows Update run.

---

Edit ＃1: aaaand here we go:

Faulting application name: svchost.exe_AppXSvc, version: 10.0.19041.4355, time stamp: 0x9ce47784
Faulting module name: appxdeploymentserver.dll, version: 10.0.19041.5072, time stamp: 0x559cd44d
Exception code: 0xc0000005
Fault offset: 0x00000000001fd08b
Faulting process id: 0x4624
Faulting application start time: 0x01db3b25c64d9a03
Faulting application path: C:\windows\system32\svchost.exe
Faulting module path: c:\windows\system32\appxdeploymentserver.dll
Report Id: 01d68cfc-1d6c-4ded-b09c-a25faf16342e
Faulting package full name: 
Faulting package-relative application ID: 

For reference, that's a STATUS_ACCESS_VIOLATION. So the issue isn't the execution alias or the permissions or any such thing in my case.

Edit ＃2: Alright and since this service starts on-demand it's always a fresh process anyway, so the fault is somewhere in the program (or in my Windows image).

Edit ＃3: the typical routine of sfc /scannow and dism followed by a reboot also don't resolve it:

C:\>dism /Online /Cleanup-Image /ScanHealth

Deployment Image Servicing and Management tool
Version: 10.0.19041.3636

Image Version: 10.0.19045.5131

[==========================100.0%==========================] No component store corruption detected.
The operation completed successfully.

C:\>dism /Online /Cleanup-Image /RestoreHealth

Deployment Image Servicing and Management tool
Version: 10.0.19041.3636

Image Version: 10.0.19045.5131

[==========================100.0%==========================] The restore operation completed successfully.
The operation completed successfully.

PS: I think I may never quite fathom why a basic application like winget had to be packaged up as store app which caused it to assume some weird Xaml-package dependency and having to go through hoops with execution aliases. This could literally have been a native application that runs without any pain on a wide range of Windows versions (including Server editions) and has no prerequisites other than system DLLs and an internet connection.

[n5ke]

@exoosh we had this exact problem happen widely at work about 5 days ago. Most likely relates to some windows update. A workaround that seems to work was taking ownership of $env:userprofile\AppData\Local\Microsoft\WindowsApps, adding permissions and rebooting

takeown /F  $env:userprofile\AppData\Local\Microsoft\WindowsApps /A /R
icacls  $env:userprofile\AppData\Local\Microsoft\WindowsApps /grant Administrators:F /T

Then rebooting and either reinstalling App Installer ( https://aka.ms/getwinget ) or running repair-WingetPackageManager

Install-Module Microsoft.WinGet.Client
repair-WingetPackageManager -force -latest -verbose -AllUsers

So in one way or another permissions do seem to factor in

[exoosh]

@n5ke thanks for chiming in. This is helpful. I was aware of the last two commands and knew that the SDs on these directories are generally rather restrictive, but I hadn't tried taking ownership to alleviate this issue.

As far as I can tell taking ownership just had the purpose to make icacls subsequently succeed. Correct? Any downsides to granting BUILTIN\Administrators these permissions on $env:userprofile\AppData\Local\Microsoft\WindowsApps? I suppose for the reinstalled app it makes no difference, but there may be different apps in that directory.

[n5ke]

Yeah had to take ownership to be able to change permissions. With my limited windows understanding, I think it's just a matter of sacrificing some protection against some malware that might target/use windows apps in some cases. I can't think of a scenario where it would cause breakage and haven't seen any breakage in practice. It also shouldn't be a problem saving and then restoring them if need be, I think this would do it:

icacls $env:userprofile\AppData\Local\Microsoft\WindowsApps /save \path\to\backup-acl-file /t
icacls $env:userprofile\AppData\Local\Microsoft\WindowsApps /restore \path\to\backup-acl-file

You are right applying to only WindowsApps\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe and WindowsApps\winget.exe might be enough, didn't try this, it's definitely better to minimize the changes if that works

[edit] icacls doesn't backup ownership, this should serve better

# Backup:
Get-ChildItem -Path $env:userprofile\AppData\Local\Microsoft\WindowsApps -Recurse | ForEach-Object { $_ | Get-Acl } | Export-Clixml -Path C:\bkp.xml

# Restore:
Import-Clixml -Path C:\bkp.xml | ForEach-Object { Set-Acl -Path $_.Path -AclObject $_ }