msys2/msys2-installer

Docker build hangs indefinitely after installing MSYS2

relapids opened this issue · 25 comments

relapids commented

Related (?) issue: #58

Thought I had the same issue but it turns out apparently not... Copying my message from there here below.

==========================

Same thing started happening to me yesterday with no changes on my end (been working for months up until now). Was definitely working two days ago since I have a script to force-rebuild my image daily. I see this on multiple machines and OS versions.

Tried both isolation modes (hyperv and process) and neither are currently working for me. One slight difference to the original reporter is that I'm using servercore:ltsc2022.

Still trying to diagnose further but unfortunately it's difficult to get diagnostic information from Windows containers for these sorts of issues. Trying to investigate using the information here: https://learn.microsoft.com/en-us/virtualization/windowscontainers/troubleshooting

EDIT:

Actually, I think I may have misunderstood the original issue in this thread and conflated it with mine. In my case the processes all appear to run successfully and terminate normally, but then the RUN step never finishes and just hangs indefinitely when finalizing the layer. I'm guessing this is probably a distinct issue and I misclassified it as the same as the one here due to the timing.

Strange though that it's happening on multiple machines and seems to coincide with the latest installer release. I tried rolling back to an earlier installer but that didn't help. Likely because whatever is causing the issue gets updated to the same version as in the latest installer anyway, but that's just a guess on my part so far - I need to do additional testing.

lazka commented

thanks

relapids commented

So I've done some more testing and if I blacklist msys2-runtime (via IgnorePkg in pacman.conf) then I'm able to get through the MSYS2 install and install other packages in subsequent layers, though obviously that's not really a viable workaround as it puts me in an unsupported state which I assume will cause other issues (and likely get worse as more time passes).

Is there some sort of flag I can enable to get verbose diagnostic information related to msys2-runtime to try and help narrow down the source of the problem?

lazka commented

Thanks for testing, yeah, looks like the latest msys2-runtime update broke something only in docker, which is why CI didn't catch it :(

  • 👍1
relapids commented

Thanks. At least now I know it's not just me. Let me know if there's anything I can do to help investigate. I'm not really familiar with debugging of MYS2 internals but happy to try and learn to lend a hand.

lazka commented

Please try again now

relapids commented

@lazka Wow that was fast. It appears to be working now. Thank you!

Out of curiosity, what did you change?

lazka commented

@lazka Wow that was fast. It appears to be working now. Thank you!

Thanks for testing.

Out of curiosity, what did you change?

I wrote up my findings here: https://cygwin.com/pipermail/cygwin/2022-December/252711.html

  • ❤️1
relapids commented

@lazka It appears this has regressed again some time today. Any ideas? I see this on multiple machines again (both locally and my CI server) so I don't think it's a local issue. Was able to work around it with the same hack as last time (blocking update of msys2-runtime).

Biswa96 commented

Can you get the version of msys2-runtime package - which one fails and which one does not ?

relapids commented

This is what I see when I block the update:
warning: msys2-runtime: ignoring package upgrade (3.4.3-2 => 3.4.3-3)

So presumably 3.4.3-2 is the working one and 3.4.3-3 the broken one.

relapids commented

By the way I'm currently pulling the nightly installer, but I just tried switching to the one tagged 2022-12-16 and the overall result appears to be the same.

Though the previous version is different in this case (which seems reasonable/expected):
warning: msys2-runtime: ignoring package upgrade (3.3.6-6 => 3.4.3-3)

Digging back through previous CI logs I see the failure was occurring when upgrading to 'msys2-runtime-3.4.2-2' though, and in the run where it started working it looks like no update occurred at all.

Perhaps the issue being resolved was an incorrect conclusion originally because I was testing with the nightly installer, and since no update occurred after the attempted fix, that resolved the issue simply because the offending code was no longer being executed anymore (since there was no update), but now that there's an update it's back again.

lazka commented

I see, yeah these might be two unrelated issues. The ASLR change made our nightly builds fail which in turn meant installing the latest build and updating resulted in a runtime update.

relapids commented

Did some more testing yesterday, and this also reproduces with the latest pacman update for me. I checked the Github Actions here (for the Docker builds in msys2-installer) and the latest runs are passing but also doesn't appear to be triggering an update of msys2-runtime or pacman. I wonder if that's the reason it seems to work fine there too (since afaict it always builds the installer first and uploads it, so the Docker builders pull an installer that's up-to-date with the latest runtime/pacman).

For what it's worth this is the test case I'm using (without the workarounds to IgnorePkg) which I ripped from your CI documentation to make sure there was nothing I was introducing in my own Dockerfile to cause the issue:

FROM mcr.microsoft.com/windows/servercore:ltsc2022

RUN powershell -Command \
  $ErrorActionPreference = 'Stop'; \
  $ProgressPreference = 'SilentlyContinue'; \
  (New-Object System.Net.WebClient).DownloadFile('https://github.com/msys2/msys2-installer/releases/download/nightly-x86_64/msys2-base-x86_64-latest.sfx.exe', 'msys2.exe'); \
  .\msys2.exe -y -oC:\; \
  Remove-Item msys2.exe; \
  function msys() { C:\msys64\usr\bin\bash.exe @('-lc') + @Args; } \
  msys ' '; \
  msys 'pacman --noconfirm -Syuu'; \
  msys 'pacman --noconfirm -Syuu'; \
  msys 'pacman --noconfirm -Scc';

If I add the following after the first-run it starts working (and I can install/update other packages).

  msys \"echo >> /etc/pacman.conf\"; \
  msys \"echo '[options]' >> /etc/pacman.conf\"; \
  msys \"echo 'IgnorePkg = msys2-runtime' >> /etc/pacman.conf\"; \
  msys \"echo 'IgnorePkg = pacman' >> /etc/pacman.conf\"; \

How I'm building:

docker build -t msys2-installer -f Dockerfile.2022 --no-cache .

Requires either a msys2-runtime or a pacman update to reproduce. Haven't noticed the same problem with other packages. E.g. I can add steps to install GCC or whatever - I tried many different packages - just fine as long as msys2-runtime/pacman are blacklisted, and updates to the pre-installed curl/libcurl also work so it's not an install vs update difference.

Quite confused, but will keep trying to narrow it down.

  • 👍1
lazka commented

No update, but I can at least confirm the issue is reproducible in CI (this case installed a version leading to an update): https://github.com/msys2/msys2-installer/actions/runs/3953154465

  • 👍1