/bashfoo

My personally curated list of bash/command-line commands and snippets that are very useful yet I keep on forgetting

Primary LanguagePython

bashfoo.yaml

https://github.com/dannguyen/bashfoo

Dan Nguyen's personally curated list of bash/command-line commands and snippets that are useful but yet he keeps forgetting

TOC

date add a UTC timestamp in ISO-8601 format to a filename

# Example
# Just the current UTC date using GNU date, i.e. gdate on macOS
printf "mydata_%s.csv\n" $(gdate --utc -I)
# >>>> mydata_2021-08-03.csv

# UTC date with hours+minutes, GNU and BSD compatible
printf "mydata_%s.csv\n" $(date -u +"%Y-%m-%d-%H%M")
# >>>> mydata_2021-08-03-1248.csv

References:

du recursively list directories and their disk space size

# Example
# GNU du (verbose)
du --human-readable --max-depth=2 ./bashfoo

# GNU du (concise) / BSD du (i.e. MacOS)
du -h -d 2 ./bashfoo

Output:

2.4M  ./bashfoo/.git/objects
4.0K  ./bashfoo/.git/info
 44K  ./bashfoo/.git/logs
 48K  ./bashfoo/.git/hooks
 16K  ./bashfoo/.git/refs
2.6M  ./bashfoo/.git
3.6M  ./bashfoo/assets/samples
3.6M  ./bashfoo/assets
6.3M  ./bashfoo

References:

Notes:

More variations and usecases for GNU du:

List only directories of a minimum size

# (concise) $ du -t 3M ./bashfoo
$ du --threshold 3M ./bashfoo
3660  bashfoo/assets/samples
3728  bashfoo/assets
6436  bashfoo/

Specify a size unit, e.g. K, M, G for: kb, mb, gb

# (concise) $ du -B M ./bashfoo
$ du --block-size M ./bashfoo
1M  ./bashfoo/.git/objects/61
...
1M  ./bashfoo/assets/samples/chicago_neighborhoods
4M  ./bashfoo/assets/samples
4M  ./bashfoo/assets
7M  ./bashfoo/

List largest 5 directories using human-numeric-sort

$ du -h ~/Downloads | sort -h -r | head -n 5
2.6G  /Users/me/Downloads
851M  /Users/me/Downloads/ebooks
233M  /Users/me/Downloads/OLD_JUNK
25M   /Users/me/Downloads/names
9.5M  /Users/me/Downloads/bootstrap-4.5.2

ffmpeg convert a video to gif

# Example
# minimalist conversion (infinite looping GIF, i.e. -loop=0, is the default)
ffmpeg -i input.mp4 output.gif

# yes-to-overwrite, 15 frames-per-second
# scaled to 225px wide w/ lanczos scaler, and non-looping
ffmpeg -y -i input.mp4 \
  -vf "fps=15,scale=225:-1:flags=lanczos" -loop -1 \
  output.gif

# generate a palette png file that favors areas with motion
ffmpeg -y -i input.mp4 -vf "palettegen=stats_mode=diff" palette.png

# and then use that palette png
ffmpeg -y -i input.mp4 -i palette.png -lavfi paletteuse output.gif

References:

ffmpeg encode .mkv video to .mp4

# Example
ffmpeg -i input.mkv -acodec aac -vcodec libx264 output.mp4

References:

ffmpeg optimize video and reduce its file size

# Example
ffmpeg -i my_video.mp4 -vcodec libx264 -crf 28 my_video_optimized.mp4

References:

ffmpeg resize a video and preserve its aspect ratio

# Example
# to rescale a video's width to 450 pixels and autoscale its height
ffmpeg -i in_video.mp4 -vf scale=450:-2 out_video.mp4

# to rescale a video's height to 204 pixels and autoscale its width
ffmpeg -i in_video.mp4 -vf scale=-2:204 out_video.mp4

# Note that the given pixel width/height should be an even number
# to avoid getting an error like: height not divisible by 2 (500x201)

References:

find and execute command on each file

# Example
find ./PATTERN -exec FOO BAR {} \;

find PlainText/*.md -exec wc -l {} \;

References:

find and list the top 10 most recently modified subdirectories

# Example
find ~/a -mindepth 1 -maxdepth 2 -type d \
    -not -name '_*' -not -name '.*' \
    -print0
  | xargs -0 -n1 -I{} \
      stat  -f '%Sm %N' -t '%Y-%m-%d %H:%M:%S' {} \
  | sort -rn | head -n10

# GNU variant, e.g. with gnu-coretools on macOS
gfind . -mindepth 1 -maxdepth 2 -type d \
    -not -name '_*' -not -name '.*'  \
    -printf '%T+ %p\n'  \
  | sort -rn \
  | head -n 10

Output:

2020-11-27 14:05:45 /Users/dan/Downloads/r-book
2020-09-25 11:39:45 /Users/dan/Desktop/sf-shelter-data
2020-06-18 22:45:15 /Users/dan/Downloads/journalism-syllabi
2020-08-26 20:16:55 /Users/dan/Desktop/vocal-samples
2020-07-09 14:48:38 /Users/dan/Downloads/svelte-project
2020-07-09 14:29:43 /Users/dan/Desktop/matplotlibguide
2020-05-29 22:00:28 /Users/dan/Downloads/buzzfeed-archives
2020-03-18 14:57:03 /Users/dan/Downloads/transcribe-texts
2020-01-18 05:55:32 /Users/dan/Desktop/oldstuff
2020-01-17 16:43:39 /Users/dan/Desktop/random_images

References:

find and tally total kilobytes of hard disk space for files with given extension(s)

# Example
echo $(find . -type f \
        \( -iname "*.csv" -o -iname '*.xls*' \) \
        -printf "(%k/1024)+" \
        2>/dev/null; \
        echo 0;) | bc

Output:

15732

References:

Notes:

  • Requires the use of gnu-find (gfind on my MacOS)
  • use -printf "%s+" to print size by bytes
  • 2>/dev/null hides error messages
  • -iname is case-insensitive

find directory name recursively

# Example
find START_DIR -type d -name "PATTERN"

References:

find file by name

# Example
find . -name "foo*"

References:

magick convert image to favicon.ico

# Example
magick /tmp/testimage.png -background none -resize 128x128 -density 128x128 favicon.ico

References:

ogr2ogr convert a shapefile into CSV + GeoJSON

# Example
ogr2ogr -f CSV \
    -dialect sqlite \
    -sql 'SELECT *, AsGeoJSON(geometry) AS geom FROM chicago_neighborhoods' \
    chicago_neighborhoods_output.csv  \
    chicago_neighborhoods.shp

References:

Notes:

Chicago neighborhood shapefile comes from the city data portal

Download a copy of the zipped shapefile: chicago_neighborhoods.zip

This is what a shapefile converted into CSV looks like: chicago_neighborhoods_output.csv

pandoc convert a Markdown file into a Word docx

# Example
pandoc README.md -f markdown -t docx -o README.docx

References:

pgrep and get all process info

# Example
# MacOS
pgrep -fil 'rails'

# Linux
pgrep -af 'rails'

Output:


47502 rails master RBENV_VERSION=2.5.1 TERMINAL_FONT=Monaco
47517 rails worker[0] RBENV_VERSION=2.5.1 TERMINAL_FONT=Monaco

References:

pkill using a file pattern

# Example
pkill -fil ipython

Output:

kill -15 90396
kill -15 90523

References:

printf to stderr

# Example
>&2 printf 'Error: %s\n' 'There was a problem' 'And another problem'

Output:

Error: There was a problem
Error: And another problem

References:

pygmentize a code snippet into highlighted rich text that I can paste into GMail

# Example
# Converting plaintext to rich-text format and copying to clipboard
printf "SELECT 1, 2, 3 \nFROM table;" \
  | pygmentize -f rtf -l sql \
  | pbcopy

# Producing a PNG of a code snippet, using the "vim" stylesheet,
# 36-point font (default is 14), 21px line spacing (default is 2px),
# and highlighting lines 2, 4, and 6 in hot pink (line numbers are on by default)
printf 'SELECT *\n\t, name, id\nFROM\n\tdTable AS Texas\nWHERE\n\tid > "Howdy"\nORDER BY\n\tname DESC;' \
  | pygmentize -f png -l sql \
    -O 'style=vim' \
    -O 'font_size=36,line_pad=21' \
    -O 'hl_color=hotpink,hl_lines=2 4 6' \
  > assets/pretty-pyg-sql.png

Example of rich-text format (-f rtf) being pasted into GMail:

pygmentize -f rtf -l

Example of writing pygmentize into PNG file:

pygmentize -f png

References:

Notes:

  • install via pip: pip install --upgrade Pygments
  • Use -L to get your installed version of pygmentize's most complete list of all styles, formatters, and lexers

rsync the contents of one directory into another

# Example
rsync -ahv --progress src_dir/ target_dir

Output:

building file list ... done
created directory /target_dir
./
.gitignore
README.md
static/media/
static/media/demos/
static/media/demos/find.mp4

sent 452477 bytes  received 2378 bytes  50909710.00 bytes/sec
total size is 472016  speedup is 1.00

References:

Notes:

  • only the source directory should have a trailing slash
  • -v makes for verbose output
  • -n does a dry run
  • -z compresses files
  • --progress provides a progress bar
  • -u updates only the destination files older than the source files
  • h human-readable numbers

ssh-add a new SSH key to system ssh-agent (macOS)

# Example
# put this in bash profile
eval "$(ssh-agent -s)"

# note: use -t rsa -b 4096 if system doesn't support Ed25519 algorithm
ssh-keygen -t ed25519 -C "justme@example.com" \
  -f ~/.ssh/myid_as_ed25519

# add key to ssh-agent and store passprhase in keychain
$ ssh-add -K ~/.ssh/myid_as_ed25519

Output:

Agent pid 59566

Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/justme/.ssh/myid_as_ed25519.
Your public key has been saved in /Users/justme/.ssh/myid_as_ed25519.pub.
The key fingerprint is:
SHA256:9zABCDEx9sp+zyxTestTest+LoremIpsum justme@example.com
The key's randomart image is:
+--[ED25519 256]--+
|.  ..     .   .o.|
|     .o.. .o .o .|
|                 |
|     ..o + + = * |
|                 |
|B       o        |
|    o... o . . .+|
|     ...T o   +o.|
|     oo.  .  .== |
+----[SHA256]-----+

References:

Notes:

May need to edit ~/.ssh/config to point to non traditional IdentityFile location, e.g.

Host *
 AddKeysToAgent yes
 UseKeychain yes
 IdentityFile ~/.ssh/id_rsa
 IdentityFile ~/.ssh/myid_as_ed25519

stem a filename, i.e. get filename sans path or extension

# Example
fullname=/tmp/hello/world.txt
newname="a_whole_new_$(basename ${fullname%.*})"
echo $newname

Output:

a_whole_new_world

References:

tar extraction, verbose

# Example
tar xzfv ARCHIVE.TAR.GZ

References:

unzip only an archive's CSV files and pipe to stdout

# Example
unzip -p schools.zip "*.csv" > schools.csv

xargs (BSD) to pipe results into another command, one at a time

# Example
echo Alice Bob Charlie | xargs -I{} -n1 echo 'Hey, {} is a great name!'

Output:

Hey, Alice is a great name!
Hey, Bob is a great name!
Hey, Charlie is a great name!

References:

youtube-dl to download just a video's transcript/subtitle files

# Example
youtube-dl --write-sub --skip-download https://www.youtube.com/watch?v=PMp_-OX15Jc

Output:

[youtube] PMp_-OX15Jc: Downloading webpage
[youtube] PMp_-OX15Jc: Downloading MPD manifest
[info] Writing video subtitles to: What is Public Domain-PMp_-OX15Jc.en.vtt