Kray-G/kinx

Repository size is too big and could be reduced with git-filter-repo

Closed this issue · 3 comments

mingodad commented

The first impression that one has when trying to clone/build this project i that it's too big in disk size terms for a scrimping language, then after looking around I found this project https://github.com/newren/git-filter-repo to help trim down this repository and did a crude first try by removing the following folders src/extlib, fonts, examples/typesetting and lib/katex with the command bellow with a reduction is size from 233MB to 18MB, but although the repository history seems to be ok the Makefile need to be fixed to build this project again (not done).

git-filter-repo \
    --replace-refs delete-no-add \
    --path-glob 'src/extlib/' \
    --path-glob 'fonts/' \
    --path-glob 'lib/katex/' \
    --path-glob 'examples/typesetting/' \
    --invert-paths --force

And got this result:

kinx-filtered$ du -h
28K	./bench
264K	./include
4,0K	./.git/logs/refs/remotes
4,0K	./.git/logs/refs/heads
12K	./.git/logs/refs
20K	./.git/logs
12K	./.git/info
768K	./.git/filter-repo/analysis
944K	./.git/filter-repo
4,0K	./.git/branches
52K	./.git/hooks
4,0K	./.git/refs/tags
4,0K	./.git/refs/remotes
4,0K	./.git/refs/heads
16K	./.git/refs
108K	./.git/objects/info
4,8M	./.git/objects/pack
4,9M	./.git/objects
6,1M	./.git
16K	./examples/c
44K	./examples/pdf
32K	./examples/esc
512K	./examples
8,0K	./bin
20K	./utility/src
316K	./utility
548K	./template/theme/standard
552K	./template/theme
576K	./template
44K	./docs/Reference/Object
8,0K	./docs/Reference/Exception
60K	./docs/Reference
1,6M	./docs/utility/images
1,7M	./docs/utility
12K	./docs/benchmark
12K	./docs/spec/others
32K	./docs/spec/definition/native
96K	./docs/spec/definition
8,0K	./docs/spec/lib/function
40K	./docs/spec/lib/primitive
116K	./docs/spec/lib/basic
168K	./docs/spec/lib
100K	./docs/spec/statement/expression
180K	./docs/spec/statement
8,0K	./docs/spec/command
16K	./docs/spec/spectest
52K	./docs/spec/algorithm
536K	./docs/spec
100K	./docs/licenses
2,7M	./docs
12K	./.vscode
28K	./src/optimizer
80K	./src/extlib/onig/test
340K	./src/extlib/onig/m4
56K	./src/extlib/onig/sample/.deps
144K	./src/extlib/onig/sample
1,6M	./src/extlib/onig/autom4te.cache
76K	./src/extlib/onig/src
3,5M	./src/extlib/onig
68K	./src/extlib/kc-json/dist
72K	./src/extlib/kc-json
3,6M	./src/extlib
200K	./src/exec/code
204K	./src/exec
32K	./src/disasm/common
32K	./src/disasm/arch/mips
76K	./src/disasm/arch/x86
40K	./src/disasm/arch/arm
152K	./src/disasm/arch
24K	./src/disasm/gen
12K	./src/disasm/spec
292K	./src/disasm
408K	./src/disasm-x64/libudis86
416K	./src/disasm-x64
1,2M	./src/jit
6,7M	./src
24K	./lib/std/net
228K	./lib/std
8,0K	./lib/exec/3rdparty
84K	./lib/exec
316K	./lib
24K	./.github/workflows
28K	./.github
8,0K	./bucket
18M	.

And this is with the original repository:

du -h
28K	./bench
264K	./include
16K	./.git/logs/refs/remotes/origin
20K	./.git/logs/refs/remotes
8,0K	./.git/logs/refs/heads
32K	./.git/logs/refs
40K	./.git/logs
8,0K	./.git/info
768K	./.git/filter-repo/analysis
772K	./.git/filter-repo
4,0K	./.git/branches
52K	./.git/hooks
8,0K	./.git/refs/tags
16K	./.git/refs/remotes/origin
20K	./.git/refs/remotes
8,0K	./.git/refs/heads
40K	./.git/refs
4,0K	./.git/objects/info
146M	./.git/objects/pack
146M	./.git/objects
147M	./.git
16K	./examples/c
44K	./examples/pdf
32K	./examples/esc
512K	./examples
8,0K	./bin
20K	./utility/src
316K	./utility
548K	./template/theme/standard
552K	./template/theme
576K	./template
44K	./docs/Reference/Object
8,0K	./docs/Reference/Exception
60K	./docs/Reference
1,6M	./docs/utility/images
1,7M	./docs/utility
12K	./docs/benchmark
12K	./docs/spec/others
32K	./docs/spec/definition/native
96K	./docs/spec/definition
8,0K	./docs/spec/lib/function
40K	./docs/spec/lib/primitive
116K	./docs/spec/lib/basic
168K	./docs/spec/lib
100K	./docs/spec/statement/expression
180K	./docs/spec/statement
8,0K	./docs/spec/command
16K	./docs/spec/spectest
52K	./docs/spec/algorithm
536K	./docs/spec
100K	./docs/licenses
2,7M	./docs
12K	./.vscode
28K	./src/optimizer
100K	./src/extlib/libssh2/include
2,0M	./src/extlib/libssh2/x64/gcc
344K	./src/extlib/libssh2/x64/vs2017
2,3M	./src/extlib/libssh2/x64
2,4M	./src/extlib/libssh2
32K	./src/extlib/ansicolor
528K	./src/extlib/libxml2/include/libxml2/libxml
532K	./src/extlib/libxml2/include/libxml2
536K	./src/extlib/libxml2/include
9,9M	./src/extlib/libxml2/x64/gcc
12M	./src/extlib/libxml2/x64/vs2017
22M	./src/extlib/libxml2/x64
22M	./src/extlib/libxml2
1,8M	./src/extlib/openssl/include/openssl
1,8M	./src/extlib/openssl/include
14M	./src/extlib/openssl/x64/gcc
7,2M	./src/extlib/openssl/x64/vs2017
21M	./src/extlib/openssl/x64
23M	./src/extlib/openssl
32K	./src/extlib/onig/windows
8,0K	./src/extlib/onig/cmake
252K	./src/extlib/onig/test
340K	./src/extlib/onig/m4
56K	./src/extlib/onig/sample/.deps
220K	./src/extlib/onig/sample
1,6M	./src/extlib/onig/autom4te.cache
2,4M	./src/extlib/onig/src
224K	./src/extlib/onig/doc
48K	./src/extlib/onig/harnesses
6,7M	./src/extlib/onig
264K	./src/extlib/libharu/include
2,7M	./src/extlib/libharu/x64/gcc
2,7M	./src/extlib/libharu/x64/vs2017
5,3M	./src/extlib/libharu/x64
5,6M	./src/extlib/libharu
8,9M	./src/extlib/sqlite
224K	./src/extlib/libcurl/include/curl
228K	./src/extlib/libcurl/include
1,6M	./src/extlib/libcurl/x64/gcc
1,8M	./src/extlib/libcurl/x64/vs2017
3,3M	./src/extlib/libcurl/x64
3,6M	./src/extlib/libcurl
68K	./src/extlib/kc-json/dist
36K	./src/extlib/kc-json/util
16K	./src/extlib/kc-json/sample
56K	./src/extlib/kc-json/src
248K	./src/extlib/kc-json
120K	./src/extlib/zip/include
884K	./src/extlib/zip/x64/gcc
748K	./src/extlib/zip/x64/vs2017
1,6M	./src/extlib/zip/x64
1,8M	./src/extlib/zip
68K	./src/extlib/winiconv
3,7M	./src/extlib/duktape
78M	./src/extlib
200K	./src/exec/code
204K	./src/exec
32K	./src/disasm/common
32K	./src/disasm/arch/mips
76K	./src/disasm/arch/x86
40K	./src/disasm/arch/arm
152K	./src/disasm/arch
24K	./src/disasm/gen
12K	./src/disasm/spec
292K	./src/disasm
408K	./src/disasm-x64/libudis86
416K	./src/disasm-x64
1,2M	./src/jit
81M	./src
24K	./lib/std/net
228K	./lib/std
8,0K	./lib/exec/3rdparty
84K	./lib/exec
316K	./lib
24K	./.github/workflows
28K	./.github
8,0K	./bucket
233M	.
Kray-G commented

@mingodad Thanks for a good info!
It looks a good solution, and the disk size should be decreased.
On the other hand, I think I should use it very carefully, so I have to plan to make a procedure to do it step by step first.

I'm also going to try to do it when I have time, and thanks a lot.

Kray-G commented

I did it for fonts, examples/typesetting, and lib/katex. The size was a little decreased.
I will do it again for src/extlib after separated to another repo.

Kray-G commented

The size has finally shrunk and now it is only 9.71MB.
I feel this objective was achieved.

Thank you very much for good info.