Affect the decompression speed

Question

Affect the decompression speed

Closed this issue a year ago · 14 comments

lateautumn233 commented 2 years ago

Use pv

Do not use

I don't know why it has such a big impact

Answer 1 · 2023-07-16T00:13:50.000Z

Hello
What kind of system is this running on? (architecture and distribution)

Is splice() available, and if so, does the "-C" option make any difference?

Does passing different values to "-B" make any difference?

Answer 2 · 2023-07-16T00:19:36.000Z

Hello What kind of system is this running on? (architecture and distribution)

Is splice() available, and if so, does the "-C" option make any difference?

Does passing different values to "-B" make any difference?

I will conduct more tests

Answer 3 · 2023-07-16T00:54:25.000Z

splice() available

pv test.tar.zst -c | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:00:55 [37.3MiB/s] [=======================================>] 100%

tar "-I zstd -T0" -axf test.tar.zst --checkpoint-action="ttyout=Hit %T\r"
Hit R: 2088028160 (2.0GiB, 232MiB/s))

pv -B
How should I test it?

Answer 4 · 2023-07-16T09:05:32.000Z

It's "pv -C" / "pv --no-splice" to turn off splice, not "pv -c" (which is "pv --cursor").

With "pv -B", try different values to see how they affect the transfer rate. For example "pv -B 1024", "pv -B 1048576", etc.

The manual says that using "-B" implies "-C" automatically, so you don't need to try both, assuming you're using PV 1.6.20.

Answer 5 · 2023-07-16T13:35:35.000Z

It's "pv -C" / "pv --no-splice" to turn off splice, not "pv -c" (which is "pv --cursor").

With "pv -B", try different values to see how they affect the transfer rate. For example "pv -B 1024", "pv -B 1048576", etc.

The manual says that using "-B" implies "-C" automatically, so you don't need to try both, assuming you're using PV 1.6.20.

uname -a

Linux localhost 5.10.136-android12-9-00021-g821df8f5bd36-ab9585204 #1 SMP PREEMPT Thu Feb 9 13:08:46 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

pv --no-splice

time pv test.tar.zst --no-splice | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.00GiB 0:01:01 [19.9MiB/s] [=======================================> ] 99% ETA 0:00:00

2.01GiB 0:01:01 [33.5MiB/s] [=======================================>] 100%

________________________________________________________
Executed in   61.46 secs    fish           external
   usr time   11.07 secs    1.25 millis   11.07 secs
   sys time   77.15 secs    0.96 millis   77.15 secs

pv -C

time pv test.tar.zst -C | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.30GiB 0:00:41 [39.6MiB/s] 1.87GiB 0:00:58 [5.92MiB/s] [=====================================>   ]
93% ETA 0:00:04

2.01GiB 0:01:12 [28.2MiB/s] [=======================================>] 100%

________________________________________________________
Executed in   73.16 secs    fish           external
   usr time   12.87 secs    0.00 micros   12.87 secs
   sys time   76.53 secs  942.00 micros   76.53 secs

Answer 6 · 2023-07-16T13:59:46.000Z

Sorry I wasn't clear about -B implying -C, please try with -B for various values such as "-B 1024" or "-B 1048576". That may improve performance.

Can you recommend a way I can build or access a similar system to yours, to reproduce the behaviour? Does it do the same on other aarch64 systems such as a Pi 4 for example?

Answer 7 · 2023-07-16T14:26:08.000Z

Sorry I wasn't clear about -B implying -C, please try with -B for various values such as "-B 1024" or "-B 1048576". That may improve performance.

Can you recommend a way I can build or access a similar system to yours, to reproduce the behaviour? Does it do the same on other aarch64 systems such as a Pi 4 for example?

I'm sorry I forgot to tell you
This is an Android device
You can test on termux

Answer 8 · 2023-07-18T11:59:39.000Z

Can you try with "-B 1024", "-B 4096", "-B 65536", "-B 1048576", see what difference each of those make?

Answer 9 · 2023-07-18T13:33:08.000Z

root@localhost ~# ./pv -B 1024 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.26GiB 0:00:40 [29.5MiB/s] [========================>                ] 62% ETA 0:00:23
2.01GiB 0:01:07 [-6.66e+24yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 4096 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:14 [-1.30e+24yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 65536 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:18 [-5.07e+25yiB/s] [==================================>] 100%            

root@localhost ~# ./pv -B 1048576 test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:15 [5.59MiB/s] [=======================================>] 100%            

root@localhost ~# strace -o pv.log ./pv test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
1.89GiB 0:00:24 [9.76MiB/s] [=====================================>   ] 94% ETA 0:00:01
1.90GiB 0:00:26 [8.31MiB/s] [=====================================>   ] 94% ETA 0:00:01
1.92GiB 0:00:28 [10.7MiB/s] [=====================================>   ] 95% ETA 0:00:01
2.01GiB 0:00:36 [-2.95e+24yiB/s] [==================================>] 100%

I don't understand

Answer 10 · 2023-07-18T13:43:32.000Z

Which version of pv was that? ("./pv -V") The nonsensical negative final value should have gone away in 1.7.0.

On your last example you did "strace -o pv.log ./pv test.tar.zst | ..." and it was much faster, is it still much faster if you remove the strace so it's just "./pv test.tar.zst | ..."?

Answer 11 · 2023-07-18T13:53:22.000Z

Yes, it's much faster after I added stace

root@localhost ~ [128]# ./pv -V                                                        
pv 1.6.20
- Copyright 2015 Andrew Wood <andrew.wood@ivarch.com>

Web site: http://www.ivarch.com/programs/pv.shtml

This program is free software, and is being distributed under the
terms of the Artistic License 2.0.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@localhost ~# ./pv test.tar.zst  | tar "-I zstd -T0" -x --checkpoint-action="ttyout=Hit %T\r"
2.01GiB 0:01:17 [5.53MiB/s] [=======================================>] 100%

Answer 12 · 2023-07-18T14:22:07.000Z

So it is faster with strace. That is very strange.

Can you try with pv 1.7.0 please?

If you repeat your tests with "-T1" passed to zstd instead of "-T0", what difference does that make?

Answer 13 · 2023-07-18T14:30:37.000Z

root@localhost ~# ./pvnew -V
pv 1.7.0 
- Copyright 2015 Andrew Wood <andrew.wood@ivarch.com>

Web site: http://www.ivarch.com/programs/pv.shtml

This program is free software, and is being distributed under the
terms of the Artistic License 2.0.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@localhost ~# ./pvnew test.tar.zst  | tar "-I zstd -T1" -x                         
2.01GiB 0:00:48 [42.2MiB/s] [=======================================>] 100%

Answer 14 · 2023-08-16T17:04:53.000Z

This project has moved to Codeberg so the issue is open there instead: https://codeberg.org/a-j-wood/pv/issues/59