yanring/setup

Setup a new machine without sudo!

Setup a New Machine without sudo!

Download, compile and install commonly used software to a custom path.

Term of Usage

• Star the repo before clone
• File issue if it does not work
• Unstar if you feel it is unmaintained

Usage

# configure setup path, you can put them into your `.bashrc` or `.zshrc`
export ZZROOT=$HOME/app
export PATH=$ZZROOT/bin:$PATH
export LD_LIBRARY_PATH=$ZZROOT/lib:$ZZROOT/lib64:$LD_LIBRARY_PATH

# e.g. install git
sh zzgit.sh

NOTE: If the download speed of the machine is too slow, you can put the git.tar.gz (using git as an example) into $ZZROOT/downloads before running the script. For the download link, check the DOWNLOADURL variable in its script; for the exact name of the software, check the NAME variable.

NOTE2: If you compile from source, please make sure that they are compiled using the same gcc version!

NOTE3: If you DO have root privilate on your machine/server, then basic tools such as libtool, ccache, git, etc. might be better installed by OS's package manager. Such as apt in Ubuntu. Only install software that are not available on the OS's packaging system.

Supported Softwares

Verified on Ubuntu 18.04

Updated: July 8, 2020

libs

• boost (1.71.0)
  • install script: ./zzboost.sh
  • post install: export BOOST_ROOT=$ZZROOT
• gmp (6.1.2)
  • install script: ./zzgmp.sh
• mpc (1.1.0)
  • depends on: gmp, mpfr
  • install script: ./zzmpc.sh
• mpfr (4.0.2)
  • depends on: gmp
  • install script: ./zzmpfr.sh
• nasm (2.14.02)
  • install script: ./zznasm.sh
• yasm (1.3.0)
  • install script: ./zzyasm.sh
• ncurses (6.1)
  • install script: ./zzncurses.sh
  • post install: export TERMINFO=$ZZROOT/share/terminfo
• freetype (2.10.1)
  • install script: ./zzfreetype.sh
• opencv (4.1.2)
  • note: this config is designed for denseflow. Please modify the cmake flags for your needs.
  • depends on: ccache, ffmpeg
  • install script: ./zzopencv.sh
  • post install: export OpenCV_DIR=$ZZROOT
• openssl (1.1.1d)
  • note: both PREFIX and OPENSSLDIR are set to $ZZROOT/ssl
  • install script: ./zzopenssl.sh
  • post install: for cmake export OPENSSL_ROOT_DIR=$ZZROOT/ssl, and for configure --with-ssl=$ZZROOT/ssl (the flag may be different in different configure scripts)
• libcurl (7.67.0)
  • depends on: autoconf, automake, libtool, m4, nroff, perl, openssl
  • install script: ./zzlibcurl.sh
• libevent (2.1.11)
  • install script: ./zzlibevent.sh
• libvpx (master)
  • install script: ./zzlibvpx.sh
• libx264 (master)
  • install script: ./zzlibx264.sh
• libx265 (3.2.1)
  • install script: ./zzlibx265.sh
• libz (1.2.11)
  • install script: ./zzlibz.sh
• libzstd (1.4.4)
  • install script: ./zzlibzstd.sh
• libzip (1.5.2)
  • install script: ./zzlibzip.sh
• libtool (2.4.6)
  • install script: ./zzlibtool.sh
• libjpeg (6b)
  • depends on: libtool
  • install script: ./zzlibjpeg.sh
• libjpeg-turbo (2.0.3)
  • depends on: nasm, yasm
  • install script: ./zzlibjpeg-turbo.sh
• libpng (1.6.37)
  • install script: ./zzlibpng.sh
• libtiff (4.1.0)
  • install script: ./zzlibtiff.sh
• libwebp (1.0.3)
  • install script: ./zzlibwebp.sh
• giflib (5.2.1)
  • install script: ./zzgiflib.sh
• libopenjp2 (2.3.1)
  • install script: ./zzlibopenjp2.sh

tools

• autoconf (2.69)
  • depends on: m4, perl
  • install script: ./zzautoconf.sh
• ccache (3.7.6)
  • install script: ./zzccache.sh
  • post install: customize cache dir by export CCACHE_DIR=~/.ccache to avoid permission issue
• cmake (3.16.0)
  • install script: ./zzcmake.sh
• anaconda3 (2019.10)
  • note: anaconda is installed default on $HOME/anaconda3
  • install script: ./zzconda.sh
  • post install: export PATH=$HOME/anaconda3/bin:$PATH
• ffmpeg (4.2.1)
  • depends on: nasm, yasm, libx264, libx265, libvpx
  • install script: ./zzffmpeg.sh
• gcc (7.5.0)
  • note: CUDA 10 support up to gcc 7, CUDA 9 support up to gcc 5.
  • depends on: gmp, mpfr, mpc
  • install script: ./zzgcc.sh
• git (2.24.0)
  • depends on: libcurl, libz
  • install script: ./zzgit.sh
• hdf5 (1.10.5)
  • install script: ./zzhdf5.sh
• htop (2.2.0)
  • depends on: ncurses
  • install script: ./zzhtop.sh
• julia (1.3.0)
  • install script: ./zzjulia.sh
• autojump (master)
  • install script: ./zzautojump.sh
  • post install: put lines of configs in .zshrc
• imagemagick (latest)
  • depends on: freetype, libjpeg-turbo, libpng, libtool, perl
  • install script: ./zzimagemagick.sh
• nmon (16j)
  • depends on: ncurses
  • install script: ./zznmon.sh
• zsh (5.7.1)
  • depends on: ncurses
  • install script: ./zzzsh.sh
  • post install: to set zsh as default shell without root, add exec $ZZROOT/bin/zsh -l to .bashrc (example)
• ohmyzsh (latest)
  • depends on: zsh
  • install script: ./zzohmyzsh.sh
• perl (5.30.1)
  • install script: ./zzperl.sh
• ruby (2.7.0)
  • depends on: openssl (with libssl installed)
  • install script: ./zzruby.sh
• tmux (3.0a)
  • depends on: libevent
  • install script: ./zztmux.sh
• unrar (5.8.4)
  • install script: ./zzunrar.sh
• vmtouch (1.3.1)
  • install script: ./zzvmtouch.sh

zz scripts & customized tools

• sshpass (support 2-step verification)
  • depends on: oathtool, autoconf
  • install script: ./zzsshpass.sh
• denseflow (extract frames, optical flow, resize, and more!)
  • depends on: boost, opencv, cmake
  • install script: ./zzdenseflow.sh
• wslssh.bat (use ssh of wsl in win10, no installation, .bat script as is)
  • depends on: sshpass
  • you can set vscode ssh path to this wslssh.bat, work together with sshpass (installed in wsl)
• sshkey.sh (gen an ssh key)
  • run script: ./sshkey.sh
• zshplugin.sh (completions, autosuggestions, syntax-highlighting)
  • depends on: git
  • install script: ./zshplugin.sh
  • post install: put lines of configs in .zshrc
• nordtmux.sh (nord theme for tmux)
  • depends on: git, tmux
  • install script: ./nordtmux.sh
• dircolor.sh (nord theme for zsh)
  • install script: ./dircolor.sh
  • post install: put lines of configs in .zshrc

zz configures

• .tmux.conf (tmux configuration)
• .zshrc (zsh configuration)
• .netrc (lftp password-less configuration)
• .ssh/config (suppress git clone warnings)
• .lftp/rc (suppress lftp ssl warning)

misc

• opengl on remote machine
• useful Julia packages: ArgParse Glob Images FileIO StatsBase Shell UnicodePlots PyCall PyPlot Plots ImageFiltering ImageMagick Interpolations Revise OhMyREPL DataFrames ProgressMeter JLD2 Distances ImageTransformations JuMP HDF5 ImageInTerminal ImageView DataStructures JSON JuliaFormatter
• free JuMP solvers: Cbc Clp COSMO CSDP ECOS GLPK Juniper OSQP ProxSDP SCS SDPA
• good reference: http://www.linuxfromscratch.org/lfs/view/development/index.html and http://www.linuxfromscratch.org/blfs/view/svn/index.html and https://pkgs.org/
• CUDA arch and CUDA gencode list
• Instruction on how to load jpeg faster. https://github.com/innerlee/cvbenchmark#setup
• View image in terminal: viu and tiv
• image lossless optimization: jpegtran optipng
• image lossy optimization: pngquant
• ffmpeg tricks
  • silent: ffmpeg -hide_banner -loglevel panic
  • get resolution ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 input.mp4
  • count frames ffmpeg -i input.mp4 -map 0:v:0 -c copy -f null - 2>&1 | grep frame
  • remove duplicated frames ffmpeg -i input.mp4 -vf mpdecimate -vsync vfr out.mp4

faq

• Q: My boost segfaluts :(
  A: Make sure they the boost are comipiled and linked using the same gcc/g++ version

• Q: Cert verify failed: BADCERT_NOT_TRUSTED :(
  A: Try export SSL_CERT_DIR=/etc/ssl/certs

• Q: warning: templates not found in /workspace/destdir/share/git-core/templates :(
  A: Add the following to $HOME/.gitconfig

[init]
        templatedir = $HOME/app/share/git-core/templates

• Q: fatal: unable to find remote helper for 'https' :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export GIT_EXEC_PATH=$ZZROOT/libexec/git-core
export GIT_SSL_CAINFO=/etc/ssl/certs/ca-bundle.crt

• Q: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications. :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export GIO_EXTRA_MODULES=/usr/lib/x86_64-linux-gnu/gio/modules/

• Q: import pyvips segfault :(
  A: Make sure gstreamer-orc <0.4.30, imagemagick <7.0.9.

conda install -c conda-forge imagemagick=7.0.8
conda install -c conda-forge gstreamer-orc=0.4.29

• Q: GitError(Code:ECERTIFICATE, Class:SSL, the SSL certificate is invalid: 0x08 - The certificate is not correctly signed by the trusted CA) :(
  A: Run the following

julia> import LibGit2
julia> LibGit2.set_ssl_cert_locations("/etc/ssl/certs/ca-certificates.crt")

• Q: dotnet error :(
  A: Add the following to $HOME/.zshrc (or .bashrc of course)

export DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
export MSBuildSDKsPath=$DOTNET_ROOT/sdk/3.1.102/Sdks

• Q: Install OpenCV: CUDA Version wrong:(
  A: Add the path of the correct version CUDA to zzopencv.sh when runing cmake

cmake   ......
        -DCUDA_TOOLKIT_ROOT_DIR=/mnt/lustre/share/cuda-9.0/ \
        ......

• Q: Install Denseflow: OpenCV Version wrong, can't find customized path for opencv:(
  A: Add the path of the correct version opencv and corresponding cuda version to zzdenseflow.sh when runing cmake

cmake -DCMAKE_INSTALL_PREFIX=$ROOTDIR \
      -DOpenCV_DIR=$ROOT_DIR/lib64/cmake/opencv4 \
      -gencode=arch=compute_61,code=sm_61 \
      -DCUDA_TOOLKIT_ROOT_DIR=/mnt/lustre/share/cuda-9.0/ ..

• Q: Unzip filenames looks messy when there are unicodes :(
  A: Specify the encoding of filenames.

$ unzip -h
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
  ...
  -O CHARSET  specify a character encoding for DOS, Windows and OS/2 archives
  -I CHARSET  specify a character encoding for UNIX and other archives
  ...

unzip -O GB18030 gb18030.zip -d target_dir

• Q: Get locale warnings :(
  A: Run

export LANGUAGE=en_US.UTF-8
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8
locale-gen en_US.UTF-8
sudo dpkg-reconfigure locales

• Q: moov atom not found :(
  A: Use https://github.com/anthwlock/untrunc to fix it.

• Q: you must add /ou=globalsign root ca ... :(
  A: Set env var.

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

---

jlinstall

jlinstall is another way to install softwares. It uses pre-built binaries so that no compilation is needed. The default install path of jlinstall is JLROOT=$ZZROOT/jl.

export ZZROOT=$HOME/app
export JLROOT=$ZZROOT/jl
export PATH=$ZZROOT/bin:$JLROOT/bin:$PATH
export LD_LIBRARY_PATH=$ZZROOT/lib:$ZZROOT/lib64:$JLROOT/lib:$JLROOT/lib64:$LD_LIBRARY_PATH