tinkerbell/osie

reduce osie size

nathangoulding opened this issue · 14 comments

nathangoulding commented

We should remove unnecessary packages, perhaps package x86_64/aarch64 architectures independently, and generally work to slim down OSIE substantially.

mmlb commented

We can take a look at doing this by both optimizing the Dockerfile some and also squashing the image at export time.

fransvanberckel commented

Why are these osie tar.gz files only tar'ed and not gz'ed? And isn't that's what gets loaded to the worker's time after time? The may be a good reason. The vmlinuz-x86_64 and initramfs-x86_64 are compressed, as expected. And modloop-x86_64 has squashfs.

$ tar -zxvf osie-runner-x86_64.tar.gz
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
$ file osie-runner-x86_64.tar.gz
osie-runner-x86_64.tar.gz: POSIX tar archive
$ file osie-x86_64.tar.gz
osie-x86_64.tar.gz: POSIX tar archive
$ file osie-aarch64.tar.gz
osie-aarch64.tar.gz: POSIX tar archive
$ ls -l
-rw-r--r-- 1 root root 679796224 Jul 15 06:11 osie-aarch64.tar.gz
-rw-r--r-- 1 root root 258395648 Jul 15 06:11 osie-runner-x86_64.tar.gz
-rw-r--r-- 1 root root 742394880 Jul 15 06:11 osie-x86_64.tar.gz

fransvanberckel commented

It's rules mk j2 at build/$v.tar.gz: ${packages} ? What part in the code does the extract ?

-- tar -cO $(sort $(subst build/,,$^)) | pigz >$(@f).tmp

fransvanberckel commented

A-ha, pigz does gzip, in parallel. ;-) But what about if these tar's get loaded as uncompressed to the worker's? Rather they become too early unpigz'ed, somewhere, somehow?

mmlb commented

@fransvanberckel the osie docker images are saved here https://github.com/tinkerbell/osie/blob/master/rules.mk.j2#L229. This issue is mainly about trimming up the source. I think a separate issue for actually compressing the osie-$arch.tar.gz makes sense.

MaxPeal commented

can we switch compressing with gz to xz -9?
it would be reduced it by ca 20% to 1.4GB

mmlb commented

Hmm I looked into it once iirc, it wasn't that big of a savings at the time and the -9 sure did take a long time. I'd be interested in seeing comparison with pigz -9 too. IIRC squashing the docker images is a good low hanging fruit/first try. For compression I'd be interested in seeing comparison of time/space for default and -9 pigz vs xz and also I think maybe compressed osie images vs not too.

fransvanberckel commented

@mmlb Also check the container content. Check the init script. Saw quite a few files in the package cache directory. Maybe you can think of something better, with the same result in the end?

mmlb commented

@fransvanberckel Do you mean the embedded alpine repo?

fransvanberckel commented

@mmlb Yes, but please check if there are more cached data files. I know we need it! But maybe you can think of a better solution,

mmlb commented

Not from the initramfs file I think. osie's ubuntu image is pretty huge and can definitely be slimmed down I think which is what this issue is about. There's OSIE everything and osie the ubuntu image. We should probably figure a better name for the osie ubuntu image.

MaxPeal commented

@mmlb some of my results

ls -lah latest*.tar*

-rw-r--r-- 1 root root 1.7G Sep 23 13:18 latest-11.tar.gz
-rw-r--r-- 1 root root 1.5G Sep 23 12:49 latest-6.tar.xz
-rw-r--r-- 1 root root 1.7G Sep 23 12:55 latest-9.tar.gz
-rw-r--r-- 1 root root 1.4G Sep 23 12:53 latest-9.tar.xz
-rw-r--r-- 1 root root 1.7G Sep 21 22:00 latest-org.tar.gz
-rw-r--r-- 1 root root 2.9G Sep 21 22:00 latest.tar

ls -la latest*.tar*

-rw-r--r-- 1 root root 1753268971 Sep 23 13:18 latest-11.tar.gz
-rw-r--r-- 1 root root 1538011504 Sep 23 12:49 latest-6.tar.xz
-rw-r--r-- 1 root root 1786131393 Sep 23 12:55 latest-9.tar.gz
-rw-r--r-- 1 root root 1464892148 Sep 23 12:53 latest-9.tar.xz
-rw-r--r-- 1 root root 1790151103 Sep 21 22:00 latest-org.tar.gz
-rw-r--r-- 1 root root 3046121472 Sep 21 22:00 latest.tar

sha1sum latest.tar latest-org.tar.gz

0a233ee1dec4fbdc2479da33573c974fa75c5543 latest-org.tar.gz
b8fb96eff7756598c60ed9c0df8d9c58017fc99f latest.tar

time pv latest.tar | pixz -k -9 > latest-9.tar.xz

2.84GiB 0:02:34 [18.8MiB/s] [=====================================================>] 100%
real 3m39.864s
user 41m53.912s
sys 2m32.816s

time pv latest.tar | pixz -k -9 -e > latest9-e.tar.xz

2.84GiB 0:03:42 [13.1MiB/s] [=========================================================>] 100%

real 5m51.489s
user 42m0.444s
sys 0m23.751s

mmlb commented

@MaxPeal I ran my own set of tests on a c3.small.x86. Along with pigz vs pixz I also compared current fat container images vs docker-squash'ed and got the following:

[nix-shell:~/osie-compression-tests]$ ls -alh *.tar
-rw-r--r-- 1 manny users 2.5G Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar'
-rw-r--r-- 1 manny users 2.9G Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master.tar'

[nix-shell:~/osie-compression-tests]$ for f in *.tar; do echo pigz $f; pigz <$f >$f.pigz; echo pigz9 $f; pigz -9 <$f >$f.pigz9; echo pixz $f; pixz <$f >$f.pixz; echo pixz9 $f; pixz -9 <$f >$f.pixz9; done

[nix-shell:~/osie-compression-tests]$ ls -alh *.tar*
-rw-r--r-- 1 manny users 2.5G Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar'
-rw-r--r-- 1 manny users 1.5G Sep 30 21:50 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pigz'
-rw-r--r-- 1 manny users 1.5G Sep 30 21:50 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pigz9'
-rw-r--r-- 1 manny users 1.3G Sep 30 21:51 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pixz'
-rw-r--r-- 1 manny users 1.3G Sep 30 21:53 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pixz9'
-rw-r--r-- 1 manny users 2.9G Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master.tar'
-rw-r--r-- 1 manny users 1.7G Sep 30 21:54 'osie-v0-n=331,c=eef3be9,b=master.tar.pigz'
-rw-r--r-- 1 manny users 1.7G Sep 30 21:54 'osie-v0-n=331,c=eef3be9,b=master.tar.pigz9'
-rw-r--r-- 1 manny users 1.5G Sep 30 21:55 'osie-v0-n=331,c=eef3be9,b=master.tar.pixz'
-rw-r--r-- 1 manny users 1.4G Sep 30 21:58 'osie-v0-n=331,c=eef3be9,b=master.tar.pixz9'

[nix-shell:~/osie-compression-tests]$ ls -al *.tar*
-rw-r--r-- 1 manny users 2616954880 Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar'
-rw-r--r-- 1 manny users 1561621585 Sep 30 21:50 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pigz'
-rw-r--r-- 1 manny users 1558106026 Sep 30 21:50 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pigz9'
-rw-r--r-- 1 manny users 1361739492 Sep 30 21:51 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pixz'
-rw-r--r-- 1 manny users 1328465892 Sep 30 21:53 'osie-v0-n=331,c=eef3be9,b=master-squashed-images.tar.pixz9'
-rw-r--r-- 1 manny users 3045191680 Sep 30 21:25 'osie-v0-n=331,c=eef3be9,b=master.tar'
-rw-r--r-- 1 manny users 1790075187 Sep 30 21:54 'osie-v0-n=331,c=eef3be9,b=master.tar.pigz'
-rw-r--r-- 1 manny users 1786061106 Sep 30 21:54 'osie-v0-n=331,c=eef3be9,b=master.tar.pigz9'
-rw-r--r-- 1 manny users 1535481344 Sep 30 21:55 'osie-v0-n=331,c=eef3be9,b=master.tar.pixz'
-rw-r--r-- 1 manny users 1464545424 Sep 30 21:58 'osie-v0-n=331,c=eef3be9,b=master.tar.pixz9'

With the following times:

[manny@mmlb-dev:/run/user/1000/osie-compression-tests]$ hyperfine --export-markdown timing.md -L compression 'pigz,pigz -9,pixz,pixz -9' '{compression} <osi
e-v0-n=331,c=eef3be9,b=master.tar >/dev/null' 
Benchmark #1: pigz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null
  Time (mean ± σ):      9.607 s ±  0.530 s    [User: 148.870 s, System: 0.703 s]
  Range (min … max):    8.662 s …  9.944 s    10 runs
 
Benchmark #2: pigz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null
  Time (mean ± σ):     23.594 s ±  0.078 s    [User: 345.041 s, System: 0.806 s]
  Range (min … max):   23.465 s … 23.694 s    10 runs
 
Benchmark #3: pixz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null
  Time (mean ± σ):     86.073 s ±  0.204 s    [User: 1302.374 s, System: 2.255 s]
  Range (min … max):   85.714 s … 86.416 s    10 runs
 
Benchmark #4: pixz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null
  Time (mean ± σ):     128.704 s ±  0.757 s    [User: 1541.209 s, System: 6.948 s]
  Range (min … max):   127.679 s … 130.159 s    10 runs
 
Summary
  'pigz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null' ran
    2.46 ± 0.14 times faster than 'pigz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null'
    8.96 ± 0.49 times faster than 'pixz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null'
   13.40 ± 0.74 times faster than 'pixz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null'
Command Mean [s] Min [s] Max [s] Relative
pigz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null 9.607 ± 0.530 8.662 9.944 1.00
pigz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null 23.594 ± 0.078 23.465 23.694 2.46 ± 0.14
pixz <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null 86.073 ± 0.204 85.714 86.416 8.96 ± 0.49
pixz -9 <osie-v0-n=331,c=eef3be9,b=master.tar >/dev/null 128.704 ± 0.757 127.679 130.159 13.40 ± 0.74

I gather that squashing + default pixz is the most beneficial. pixz9 is only slightly better compression but almost 1.5x time to compress.