Duplicated BTFs files

Question

Duplicated BTFs files

mauriciovasquezbernal opened this issue 3 years ago · 2 comments

mauriciovasquezbernal commented 3 years ago

This is rather not an issue but something that could be improved.

I found out that there are some BTF files with the exact same content for different kernel versions:

$ fdupes -r ./archive/ 
./archive/centos/8/x86_64/4.18.0-147.3.1.el8_1.x86_64.btf
./archive/centos/8/x86_64/4.18.0-147.5.1.el8_1.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1062.4.2.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1062.4.3.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1160.2.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1160.2.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1160.24.1.el7.x86_64.failed
./archive/centos/8/x86_64/4.18.0-240.el8.x86_64.failed

./archive/centos/7/x86_64/3.10.0-1160.41.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1160.45.1.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-327.36.2.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-327.36.3.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-327.3.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-327.4.4.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-1062.1.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-1062.1.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-693.1.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-693.2.2.el7.x86_64.btf

./archive/centos/7/x86_64/3.10.0-514.6.1.el7.x86_64.btf
./archive/centos/7/x86_64/3.10.0-514.6.2.el7.x86_64.btf

./archive/ubuntu/18.04/x86_64/4.15.0-106-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-108-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1020-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1022-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1047-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1048-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1050-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1033-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1034-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-115-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-117-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1014-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1018-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1045-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1047-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1022-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1024-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-121-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-122-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-123-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-124-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-109-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-111-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-112-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1103-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1104-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-69-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-70-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-129-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-130-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-132-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-134-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1034-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1035-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-24-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-29-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1028-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1030-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-40-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-42-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1028-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1029-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-45-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-47-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1039-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1040-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-51-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-52-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-53-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-54-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1093-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1095-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1077-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1079-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1086-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1087-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-156-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-158-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1023-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1025-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-74-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-76-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1080-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1082-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1059-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1061-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1073-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1076-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1018-aws.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1020-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1096-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1097-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1091-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1092-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-84-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-86-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-44-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-45-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-139-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-140-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1057-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1058-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1110-azure.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1111-azure.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1035-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1036-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-60-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-62-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-64-generic.btf

./archive/ubuntu/18.04/x86_64/5.4.0-1048-azure.btf
./archive/ubuntu/18.04/x86_64/5.4.0-1049-azure.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1011-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1016-aws.btf

./archive/ubuntu/18.04/x86_64/4.15.0-151-generic.btf
./archive/ubuntu/18.04/x86_64/4.15.0-153-generic.btf

./archive/ubuntu/18.04/x86_64/4.15.0-1099-aws.btf
./archive/ubuntu/18.04/x86_64/4.15.0-1101-aws.btf

./archive/ubuntu/18.04/x86_64/5.4.0-59-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-60-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-62-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-64-generic.btf
./archive/ubuntu/18.04/x86_64/5.4.0-65-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1020-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1022-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-26-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-28-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-29-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1045-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1047-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1022-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1024-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1015-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1017-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1034-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1035-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-40-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-42-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-45-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-47-generic.btf

./archive/ubuntu/20.04/x86_64/5.8.0-1040-azure.btf
./archive/ubuntu/20.04/x86_64/5.8.0-1041-azure.btf
./archive/ubuntu/20.04/x86_64/5.8.0-1042-azure.btf

./archive/ubuntu/20.04/x86_64/5.8.0-25-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-28-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-29-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-52-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-53-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-54-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1023-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1025-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-31-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-33-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1059-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1061-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1018-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1020-aws.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1021-aws.btf

./archive/ubuntu/20.04/x86_64/5.4.0-84-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-86-generic.btf

./archive/ubuntu/20.04/x86_64/5.8.0-34-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-36-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-38-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-40-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-41-generic.btf
./archive/ubuntu/20.04/x86_64/5.8.0-43-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1016-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1019-azure.btf

./archive/ubuntu/20.04/x86_64/5.11.0-1017-azure.btf
./archive/ubuntu/20.04/x86_64/5.11.0-1019-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1035-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1036-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-1048-azure.btf
./archive/ubuntu/20.04/x86_64/5.4.0-1049-azure.btf

./archive/ubuntu/20.04/x86_64/5.4.0-37-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-39-generic.btf

./archive/ubuntu/20.04/x86_64/5.4.0-59-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-60-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-62-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-64-generic.btf
./archive/ubuntu/20.04/x86_64/5.4.0-65-generic.btf

I think this could be optimized to reduce the storage needs by replacing the duplicated ones with a symlink. The feasibility and importance of this optimization is related to the adopted storage solution -> #12.

Answer 1 · 2022-08-30T17:28:57.000Z

I believe this can't currently be done because a symlink to the files would make the raw permlink not to work if someone tries to get it from github URL directly (which is a current use case), so I'm closing this, please do reopen if you see fit/need.

Answer 2 · 2022-09-07T08:32:29.000Z

@rafaeldtinoco are you sure about that? here's a test where the file bar is a symlink to the file foo and both work through GitHub raw urls:
https://github.com/itaysk/actionstest
https://raw.githubusercontent.com/itaysk/actionstest/main/foo
https://raw.githubusercontent.com/itaysk/actionstest/main/bar