bash

A collection of bash scripts of varying utility.

Scripts specific to this repository

script	description
`install_packages`	install dependencies
`make_links`	link scripts to `${HOME}/bin`

General utilities

script	description
`add_path`	add arguments to PATH
`bell`
`log`
`untar`	unpack an archive
`update_packages`	update, upgrade, and autoremove installed packages with `apt install` then check for a release upgrade
`whatsup`	print host, display, user, path, and PID information
`xtest`	test X11 functionality

Time

script	description	useage
`date2age`		`ls -tr --color=no \| xargs -n 1 date +%s -r \| xargs -n 1 date2age`
`date2time`
`file_age`		`\ls -tr \| xargs -n 1 file_age`
`print_times`	print different time standards
`sec2elap`	convert an integer to a human-readable interval string

File cleanup

script	description
`clean_mac`
`fix_bad_extensions`
`rmbin`
`unfix_bad_extensions`

bash history

script	description
`dedup_history`
`ps_hist_cp_lnk`
`sort_history`

copying files

script	description
`cp_tar`
`rpath`
`rpull`
`rpush`
`rsync2`

finding files

script	description
`find_matching`
`find_matching_and_move`
`find_missing_and_empty`
`grep_matching`

`grep_matching` Example

Copying data sets

in the xxxxxx data set, for example, there are over 87k files in the xxx/ directory and over 54k files in the xx/ directory; over 141k files in total. Using utilities like ls or find to identify files matching a pattern will take 10s of seconds per search. Instead, use the following procedure.

First, ls for find the directory once. For example, use the command:

find ./ -type f -name "*_f00500_*o.bin" > file_list/find_f00500_obin.txt

to generate a list of files matching a particular pattern within a directory. In this example find_f00500_obin.txt is a list of all xxxxxx files with _f00500 in the file name, corresponding to 500Hz frequency.

Then the ouput file, not the directory can be interrogated very quickly. The list can be further wittled down with grep or sed commands. For example:

grep "_d0010_f00500" find_500_obin.txt | sort -u > find_d0100_f00500_obin.txt

where find_d0010_f00500_obin.txt is a list of all files with _d0010_f00500 in the file name, corresponding to 10m depth and 500Hz frequency.

Once the list has been appropriately reduced, it can be searched using the function grep_matching. Take for example, a list of file patterns, such as

$ head out_grep_2023-07-11-t1255/xxxxxx_bathy_cluster_01.txt
ens0[0-1][0-9]_2020[0-9]\\{6\\}_xxxx_000081_d0010_f00500_[0-9]\\{6\\}o.bin

using the command,

grep_matching out_grep_2023-07-11-t1255/xxxxxx_bathy_cluster_01.txt find_d0010_f00500_obin.txt

will generate a list of found files matching the pattern.

Further, the loop command may be used to find several matches

for n in {1..10}; do grep_matching out_grep_2023-07-11-t1255/xxxxxx_bathy_cluster_$(printf "%02d" $n).txt find_d0010_f00500_obin.txt ; done

other scripts

script	description
`ls2md`
`set_path`
`test_file`
`wiggler`
`write_test_dirs`

Git

see git

Test

see test

jonlighthall/bash