[Bug]: rdeps.py dont seems work anymore.

Question

[Bug]: rdeps.py dont seems work anymore.

sorasoras opened this issue a year ago · 23 comments

sorasoras commented a year ago

Describe the bug

A clear and concise description of what the bug or problem is.

To Reproduce

Precise version of rocBLAS installed or rocBLAS commit hash if building from source.
Steps to reproduce the behavior:

build rocblas 5.5 from source for windows
See error on logfile
log.txt

Answer 1 · 2023-12-05T18:19:09.000Z

Not sure if you saw examples of details in the issue template but adding some more information will help.
You say doesn't seem to work anymore, so when did it work for you?
I see drives C: D: and W: all referred to as well as proxy errors, but could be windows updates make the hard coded vcpkg release tag (in rdeps.py) incompatible. You can try with newer vcpkg releases.
Can you confirm if you set your env VCPKG_PATH and don't use a vpkg install from visual studio that you updated?
Also good to know your visual studio version along with vcpkg version.
I can try to reproduce with latest visual studio version and the 5.5 branch tip, on Windows 11 latest patch.

Answer 2 · 2023-12-05T20:12:21.000Z

But the mirrors is no longer available
404 Not Found

Answer 3 · 2023-12-05T20:44:15.000Z

I'll see if I can find out what is up with the mirror list issue but for 5.5 you want to match this vcpkg:
git clone -b 2022.05.10 https://github.com/microsoft/vcpkg
The -b flag gives you release tag 2022.05.10 that should give you download lists for the version that was tested as working. If you just clone you get an unreleased version, not to say all later releases have been tested but is the original non-mirror download site retired or just temporarily down needs to be checked. Make sure you are not blocking it directly by fetch/ping test.
The old vcpkg tag could also have problems with newer toolchain....

Answer 4 · 2023-12-05T21:22:48.000Z

I dont think switch to 2022.05.10 vcpkg help.
logs.txt
The only workaround i can think of is to download them one by one and load it as cache.

Also,
I just find this out at here "microsoft/vcpkg#35359 (comment)"
I guess this is related?

Answer 5 · 2023-12-05T21:51:29.000Z

Okay yes looks like the same issue, guess the dependency msys2 download is too stale if they remove them after 1.75 years after any update.
That isn't very nice if you want something to keep building for 2 years at least. You can try later release tags of vcpkg but I would start moving forward from 2022.05.10 as new dependencies could also break things. Let us know and I'll look too for a newer candidate.

Answer 6 · 2023-12-06T17:25:54.000Z

Bad news is that the first vcpkg release that fixes the gtest to use a still available mingw package is 2023.06.20 but that breaks backward compatibility in msgpack as the new versions change target names. So it looks like manual installation of components by version will be required, the required components to install are all listed in rdeps.xml so just moving gtest forward is what I'll try next. vcpkg releases or commits won't imply any build reproducibility if distributions availability may drop anytime after.

Answer 7 · 2023-12-06T20:26:16.000Z

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents:
{
"dependencies": [
"gtest",
"msgpack",
"openblas"
],
"builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44",
"overrides": [
{
"name": "gtest",
"version": "1.13.0"
}
]
}
So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch:
%VCPKG_PATH%\vcpkg --triplet x64-windows install
Then the same as before to build
python rmake.py -c -a ...
I need to look into how to patch this old release as the branch is normally frozen

Answer 8 · 2023-12-07T11:02:35.000Z

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents: { "dependencies": [ "gtest", "msgpack", "openblas" ], "builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44", "overrides": [ { "name": "gtest", "version": "1.13.0" } ] } So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch: %VCPKG_PATH%\vcpkg --triplet x64-windows install Then the same as before to build python rmake.py -c -a ... I need to look into how to patch this old release as the branch is normally frozen

Errors occurred while parsing W:\git\vcpkg\vcpkg.json
$ (a manifest): missing required field 'name' (an identifier)
$ (a manifest): expected a versioning field (example: version-string)
See https://github.com/Microsoft/vcpkg/tree/master/docs/users/manifests.md for more information.

vcpkg.json
am I missing something in vcpkg.json?

Nevermind, I have to use latest ver vcpkg to load this vcpkg.

Answer 9 · 2023-12-07T16:43:30.000Z

how shuold i do？ I has already been done the vcpkg.json. But run python rdeps.py are identical errors.

Answer 10 · 2023-12-07T16:48:33.000Z

So to start with commit of 2022.05.10 and add new gtest only to avoid mingw change you can use a local file vcpkg.json with contents: { "dependencies": [ "gtest", "msgpack", "openblas" ], "builtin-baseline": "14e7bb4ae24616ec54ff6b2f6ef4e8659434ea44", "overrides": [ { "name": "gtest", "version": "1.13.0" } ] } So running vcpkg at location of this new file builds the deps (e.g. root dir). I tested and works with VS 17.8.2 and building 5.5 branch: %VCPKG_PATH%\vcpkg --triplet x64-windows install Then the same as before to build python rmake.py -c -a ... I need to look into how to patch this old release as the branch is normally frozen
logs2.txt

I make the change with the json you provide, it does not seems to resolve the issue

microsoft/vcpkg#35516
This issue I look up kind of similar to this one.

Answer 11 · 2023-12-07T16:51:25.000Z

rocblas.txt That is log.
..

Answer 12 · 2023-12-07T16:57:07.000Z

Sorry that json must have been run against vckpg 2023.06.23 on my workstation, they must have changed the parser. I'll see if I can find a json for the original branch vcpkg 2022.05.10 version that you can you try, or clone 2023.06.23 and retry.
You can manually install the 3 dependencies and point to them via CMAKE_PREFIX_PATH, or try vcpkg install with what I tested with your vcpkg verison. These are what get found by json and 06.23:

gtest:x86-windows -> 1.13.0 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\gtest\3f70f42192aacc5f8242c14d959d02a802f7559c
msgpack:x86-windows -> 3.3.0#2 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\msgpack\33927863a8eae2ea7c816389e7ef3dbdcc00382a
openblas:x86-windows -> 0.3.20 -- C:\develop\github\vcpkg\buildtrees\versioning_\versions\openblas\abd8d8b0d0286d963399d79e3e8b6f7f8e12117b

Answer 13 · 2023-12-07T17:24:40.000Z

Okay with 2022.05.10 it appears it doesn't accept the newer json package overrides (requires other tags as well to parse but the newer dependency versions aren't allowed) so please try that json with 2023.06.23 or later vcpkg commit and let me know how it goes. I don't think we can convince mingw to restore the older versions so seems like we will have to give updates when older versions drop off availability.

Answer 14 · 2023-12-07T17:34:37.000Z

Okay with 2022.05.10 it appears it doesn't accept the newer json package overrides (requires other tags as well to parse but the newer dependency versions aren't allowed) so please try that json with 2023.06.23 or later vcpkg commit and let me know how it goes. I don't think we can convince mingw to restore the older versions so seems like we will have to give updates when older versions drop off availability.

I was able to download all stuff except "mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst"
log3.txt
I found the package via google but I don't how to manually load onto pkg. vckpg 2023.06.23 no longer available so i use 23/6/15

https://mirrors.piconets.webwerks.in/msys2-mirror/mingw/i686/mingw-w64-i686-crt-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst

Answer 15 · 2023-12-07T17:41:49.000Z

This is my version
E:\llama.cpp\Rocm\rocBLAS>vcpkg version
vcpkg package management program version 2023-09-15-ac02a9f660977426b8ec6392919fbb1d51b10998

Answer 16 · 2023-12-08T16:28:53.000Z

@mononoSaya okay is that is from vcpkg-tool right? 9-15 is likely too new so requires msgpack changes that are incompatibl with Tensile used in rocBLAS in 5.5
Probably best if we stick with the vcpkg repo, not the tool vcpkg-tool repo to keep it simple, so this is what should still exist there:
git clone -b 2023.06.20 https://github.com/microsoft/vcpkg
Then bootstrap that one as per bootstrap-vcpkg.bat and set env VCPKG_PATH to where you installed.
Putting the json above in rocblas root directory that then you should get the required versions with

%VCPKG_PATH%\vcpkg --triplet x64-windows install

Otherwise you just need to use manual methods to install the versions listed earlier. The overriding change is client testing for gtest so doesn't change the rocblas and tensile library build.

Answer 17 · 2023-12-08T19:04:15.000Z

@mononoSaya okay is that is from vcpkg-tool right? 9-15 is likely too new so requires msgpack changes that are incompatibl with Tensile used in rocBLAS in 5.5 Probably best if we stick with the vcpkg repo, not the tool vcpkg-tool repo to keep it simple, so this is what should still exist there: git clone -b 2023.06.20 https://github.com/microsoft/vcpkg Then bootstrap that one as per bootstrap-vcpkg.bat and set env VCPKG_PATH to where you installed. Putting the json above in rocblas root directory that then you should get the required versions with

%VCPKG_PATH%\vcpkg --triplet x64-windows install

Otherwise you just need to use manual methods to install the versions listed earlier. The overriding change is client testing for gtest so doesn't change the rocblas and tensile library build.

Thanks, I was able pass all deploy and compiled .

Answer 18 · 2023-12-09T10:30:05.000Z

logfailed.txt
I encounter another bug I think.
I was able to get one compile success for one gpu arch. Then, I try to target another arch, but i encounter another bug, but
"CMake Error at C:/vcpkg/scripts/buildsystems/vcpkg.cmake:853 (_find_package):
By not providing "Findmsgpack.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "msgpack", but
CMake did not find one.

Could not find a package configuration file provided by "msgpack" with any
of the following names:

msgpackConfig.cmake
msgpack-config.cmake

Add the installation prefix of "msgpack" to CMAKE_PREFIX_PATH or set
"msgpack_DIR" to a directory containing one of the above files. If
"msgpack" provides a separate development package or SDK, be sure it has
been installed.
Call Stack (most recent call first):
build/release/virtualenv/Lib/site-packages/Tensile/Source/lib/CMakeLists.txt:100 (find_package)"

thanks.

Answer 19 · 2023-12-09T20:55:56.000Z

O nevermind, I was just not able compile for gfx1032/gfx1035 but gfx1031/gfx1010 are fine.

Answer 20 · 2023-12-13T09:15:03.000Z

Will gfx1035 support ever land?

Answer 21 · 2023-12-13T15:45:25.000Z

Unsupported gfx will not work as GPU targets for building. The msgpack error is independent of this and due to a breaking change in newer msgpack versions, the changed cmake targets will have to be accomodated by fixes in our (Tensile ROCm/Tensile#1836) CMake code in later releases. It can be avoided with the custom listed dependency versions provided earlier (and specific vcpkg version). While this is closed I will still try and determine how to provide dependency patches so that rdeps.py can be used once again for earlier and future releases.

@ekg while I agree with the request to support more gfx I can not say if direct support will ever come. Only on Linux you can try to use an override to make the gfx1035 report itself as the supported gfx1030 using environment variable, HSA_OVERRIDE_GFX_VERSION=10.3.0
While this may work it is not officially supported, Please add your voice to ROCm/ROCm#2654 as this issue is a building only topic for dependencies.

Answer 22 · 2023-12-14T00:38:33.000Z

Will gfx1035 support ever land?

On Ubuntu 23.10 or Debian Unstable, you can install librocblas-dev to get a version of rocBLAS that has been tested on gfx1035. Those results suggest that Torre's HSA_OVERRIDE_GFX_VERSION=10.3.0 recommendation is likely to work for users of the AMD binary packages and other Linux distributions. Unfortunately, the methods used to enable gfx1035 on Linux are not applicable to Windows.

Answer 23 · 2024-05-13T13:49:17.000Z

Download manually from here and put inside to downloads folder inside vcpkg folder. It works without any other tricks.
(both or one of these files is giving errors : msys-mingw-w64-i686-libwinpthread-git-9.0.0.6373.5be8fcd83-1-any.pkg.tar.zst , msys-mingw-w64-i686-pkg-config-0.29.2-3-any.pkg.tar.zst)
https://sandwich.sh/archive/other/pineapple-src/externals/vcpkg/downloads/index.html