Linux Notes === HELPFUL COMMANDS --- alias l="ls -AlFh" alias la="ls -Al" alias lh="ls -Alh" alias ll="ls -l" # I'm dumb alias rm='rm -i' alias cp='cp -i' alias mv='mv -i' alias :q=exit alias sosh=". \"${HOME}/.$(basename $SHELL)rc\"" function digga() { dig +nocmd "$1" any +multiline +noall +answer } cp_p () { rsync -WavP --human-readable --progress $1 $2 } # Copy w/ progress _haz() { command -v "$1" > /dev/null 2>&1 } # Has command in path alias extip="dig +short myip.opendns.com @resolver1.opendns.com" # Fastest external IP alias 'hist?'="history | grep -i" function psgrep() { ps axuf | grep -v grep | grep "$@" -i --color=auto; } function fname() { find . -iname "*$@*"; } function take() { mkdir $1 && cd $1 } alias tls="tmux list-sessions" alias ta="tmux attach-session -t" alias td="tmux kill-session -t" # Concat a bunch of files with headings like ==> File <== tail -n +1 *.txt # Colored up cat! # Easy Insall & Pygments: sudo apt-get install python-pip && sudo pip install Pygments alias c='pygmentize -O style=monokai -f console256 -g' # Best available less # --- # -F: less exits if entire file can be displayed on one screen # -i: ignore case for searches # -R: display raw control characters for ANSI escape sequences # -S: chop lines longer than screen # -X: don't clear the screen export LESS='-FiRSX' # Checking outbound ports nc -vz portquiz.net -w 1 <port> # Watch a command with a log watch -n1 'uptime | tee uptime.log' # ANSI escape sequences (https://github.com/robertknight/konsole/blob/master/user-doc/README.moreColors) # --- ESC[ ... 38;2;[r];[g];[b] ... m Select RGB foreground color ESC[ ... 48;2;[r];[g];[b] ... m Select RGB background color ESC[ ... 38;5;[i] ... m Select indexed foreground color ESC[ ... 48;5;[i] ... m Select indexed background color [r],[g] and [b] are each values in the range of 0..255 and represent the brightness as usual for the respective color. [i] likely is a value in 0..256, but represents an indexed color assignment composed as follows: 0 .. 15 - System color, these are taken from the schema. 16 .. 231 - Forms a 6x6x6 RGB color cube. 232 .. 255 - A gray scale ramp without black and white. # Create a crypt encrypted password for /etc/passwd python -c 'import crypt; print crypt.crypt("password", "$6$salt$")' MAN PAGE INFO I NEVER REMEMBER --- Systemd compatibility directory debian [can't find man page]: /run/systemd/generator.late/[service].service Linux File System Hierarchy for your flavor man 7 hier History Append history lines entered since the beginning of the current Bash session to the history file: history -a History Append the history lines not already read from the history file to the current history list. history -n Locale Settings [locale(7)]: export LANGUAGE=en_US.UTF-8 export LANG=en_US.UTF-8 export LC_ALL=en_US.UTF-8 Process Tree [pstree(1)]: pstree -p Remove all files in /tmp older than 2 days [find(1) EXPRESSIONS] find /tmp -maxdepth 1 -type f -mtime +2 -exec rm -i "{}" \; Bash Special Variables [bash(1) PARAMETERS, Special Parameters]: - `$*`Expands to the positional parameters, starting from one. Quoted expands to words sparated by 1st char of `$IFS` - `$@``$*` except expands to words separated by space when quoted - `$#` Number of positional params in decimal - `$?`Exit status of most recent foreground pipeline - `$-`Option flags set by the `set` builtin command - `$$`PID of current shell - `$!`PID of most recent async/bg command - `$0`Name of shell or script - `$_`The underscore variable is set at shell startup and contains the absolute file name of the shell or script being executed as passed in the argument list. Subsequently, it expands to the last argument to the previous command, after expansion. Bash test a socket exists [bash(1) CONDITIONAL EXPRESSIONS] test -S [socketname] && echo 'true' Remove user from /etc/passwd manually without userdel [vipw(8)]: EDITOR=vim vipw Remove user from `/etc/shadow` manually without userdel [vipw(8)]: EDITOR=vim vipw -s Remove group from /etc/group manually without groupdel [vigr(8)]: EDITOR=vim vigr Shell Startup Files [bash(1)/zsh(1) INVOCATION]: - Linux/Bash/TTY: `/etc/profile` → first of `~/.bash_profile`, `~/.bash_login`, `~/.profile` that exists - Linux/Bash/XTerm: `/etc/bash.bashrc` → `.bashrc` - Linux/Bash/Scripts (#!/usr/bin/env bash): looks for `$BASH_ENV` var and sources the expansion of that variable - Linux/ZShell/TTY: `/etc/zshenv` → `~/.zshenv` → `/etc/zprofile` → `~/.zprofile` → `/etc/zshrc` → `~/.zshrc` → `/etc/zlogin` → `~/.zslogin` (`/etc/z*` is the default; however `/etc/zsh/z*` seems to be common (at least on Ubuntu)) - Linux/ZShell/XTerm: `/etc/zshrc` → `~/.zshrc` - Linux/ZShell/Scripts (#!/usr/bin/env zsh): `/etc/zshenv` - OSX/Bash/TTY/XTerm: `/etc/profile` → first of `~/.bash_profile`, `~/.bash_login`, `~/.profile` that exists - OSX/Bash/Scripts (/usr/bin/env bash): looks for `$BASH_ENV` var and sources the expansion of that variable - OSX/ZShell/TTY/XTerm: `/etc/zshenv` → `~/.zshenv` → `/etc/zprofile` → `~/.zprofile` → `/etc/zshrc` → `~/.zshrc` → `/etc/zlogin` → `~/.zslogin` - OSX/ZShell/Scripts (/usr/bin/env zsh): `/etc/zshenv` Setting your hostname—Debian and Ubuntu [hostname(1)/hosts(5)]: 1. Edit `/etc/hostname` and add your short hostname, e.g., `echo "parabola" > /etc/hostname` 2. Run `hostname -F /etc/hostname` to update your hostname 3. Edit `/etc/hosts`—order matters: FQDN, HOSTNAME, localhost: 127.0.1.1 parabola.tylercipriani.com parabola localhost Setting Timezone—Debian and Ubuntu: dpkg-reconfigure tzdata Setting Timezone—Arch/CentOS7/systemd: timedatectl list-timezones timedatectl set-timezone America/Mountain Setting Timezone—Old school: ln -sf /usr/share/zoneinfo/US/Mountain /etc/localtime POSIX RegEx Patterns [grep(1)]: [:alnum:] - [a-zA-Z0-9] [:alpha:] - [a-z] [:cntrl:] - [\x00-\x1F\x7F] [:digit:] - [0-9] [:graph:] - Any visible chars except spaces & control [\x21-\x7E] [:lower:] - [a-z] [:print:] - Any visible except control [\x21-\x7E] [:punct:] - [!"#$%&'()*+,\-./:;<=>?@[\\\]^_`{|}~] [:space:] - [ \t\r\n\v\f] [:upper:] - [A-Z] [:xdigit:] - Hexadecimal digits [A-Fa-f0-9] Show linked library dependencies of a binary [ldd(1)] ldd $(command -v <binary>) Dump files in octal [od(1)] od -c [file] Dump file as plain hex [xxd(1)] xxd -p Reassemble plain hexdump [xxd(1)] xxd -p -r Find files in directory with no newline at eof [pcregrep(1)]: pcregrep -LMr '\n$' . Cronjob Time Syntax [crontab(5)]: - m h dom m dow <what_to_do> - m - minute(0–59) - h - hour(0–23) - dom - day-of-month(0–31) - m - month(0-11) - dow - day-of-week(0–6) - <what_to_do> - anycommand Send mail [mail(1)]: echo "Message Body" | mail -s "Mail Subject" user@example.com Send mail to multiple people [mail(1)]: echo "Message Body" | mail -s "Mail Subject" -c "user2@example.com user3@example.com" user@example.com Send mail with attachement [mutt(1)]: echo "Message Body" | mutt -a /path/to/attachment -s "Subject" -- user@example.com Dpkg info about the maintainer of a Debian/Ubuntu package: dpkg-query -W -f='${Maintainer}' coreutils Find your os version information (https://superuser.com/questions/11008/how-do-i-find-out-what-version-of-linux-im-running) cat /etc/*release Create a `.gitkeep` file in all empty directories [find(1)]: find . -type d -empty -print0 | xargs -0 -I{} touch {}/.gitkeep IPTables Block an IPAddress [iptables(8) OPTIONS] iptables -A INPUT -s "$BLOCK_THIS_IP" -j DROP IPTables Insert a rule to allow inbound tcp traffic on port 8000 (puts the rule at the top)[iptables(8) OPTIONS] iptables -I INPUT -i eth0 -p tcp --dport 8000 -j ACCEPT IPTables append a rule to allow inbound tcp traffic on port 8000 (puts the rule at the bottom)[iptables(8) OPTIONS] iptables -I INPUT -i eth0 -p tcp --dport 8000 -j ACCEPT IPTables delete the rule to allow inbound tcp traffic on port 8000 [iptables(8) OPTIONS] iptables -D INPUT -i eth0 -p tcp --dport 8000 -j ACCEPT IPTables save to file [iptables-save(8)] iptables-save > /etc/iptables.up.rules IPTabels survive reboot [https://www.debian-administration.org/article/445/Getting_IPTables_to_survive_a_reboot]: iptables-save > /etc/iptables.up.rules printf "#!/usr/bin/env bash\niptables-restore < /etc/iptables.up.rules\n" \ > /etc/network/ifup.d/iptables chmod +x /etc/network/ifup.d/iptables TCPDump dump packets to/from host [tcpdump(1)] tcpdump -n -vvv -XX host [hostname/ip] TCPDump dump packets to host on ip [tcpdump(1)] tcpdump -n -vvv -XX dst host 192.0.2.1 Nmap ping scan your network [nmap(1)] nmap -sn 192.168.1.0/24 Arp show network neighbors [arp(8)] arp arp -a Create gzipped tar from file or directory [tar(1)] tar -cvzf tarfile.tar.gz directory file otherfile Create bzipped tar file from file or directory [tar(1) OTHER OPTIONS] tar -cvjf tarfile.tar.bz2 directory file otherfile List contents of tar.gz file [tar(1)] tar -tvzf tarfile.tar.gz WGet rip a whole site [wget(1)] wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains <domain-name> --no-parent http://<domain> Upload file via cURL [curl(1)] curl -X 'POST' -H 'Accept: application/json' -F 'file_name=Test File' -F 'file_contents=@/path/to/file.type' www.example.com/file/add Fill login form via cURL [curl(1)] curl -X 'POST' -F 'username=tyler' -F 'password=pass123' www.example.com/login Forward new port in existing SSH session: <newline> ~C ssh> ? SSH Generate a public key given a private key [ssh-keygen(1)] ssh-keygen -y -f [private key] SSH Get fingerprint from public key [ssh-keygen(1)] ssh-keygen -l -f [public key] SSH List key fingerprints in ssh-agent [ssh-add(1)]: ssh-add -l SSH update private keys in new format rather than PEM (use 64 rounds—2 seconds) also add or change passphrase (`-p`)[ssh-keygen(1)] ssh-keygen -o -a 64 -p -f [id_rsa] SSH Skip known host file checking [ssh_config(5)]: ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [server] SSH Remove hostname from known_hosts file [ssh-keygen(1)]: ssh-keygen -R [hostname] >> ~/.ssh/known_hosts SSH automatically add server key to known_hosts file [ssh-keyscan(1)]: ssh-keyscan -H [hostname] >> ~/.ssh/known_hosts List currently established, closed, orphaned and waiting TCP sockets: ss -s `netstat -tlnp` vs `ss -ln` $ time netstat -tlnp netstat -tlnp 0.00s user 0.02s system 97% cpu 0.016 total $ time ss -ln ss -ln 0.00s user 0.00s system 83% cpu 0.005 total Unmount FUSE filesystems [fusermount(1)]: sudo fusermount -u /path/to/fs Check disk I/O: vmstat 1 10 What's using disk I/O [dstat(1)]: dstat --top-io --top-bio Show io per-device (needs sysstat package on debian) [iostat(1)]: iostat -k 2 Show top users of io [iotop(8)] : -b = non interactive, easier to log : -o = only show proc doing IO, not all process : -t = add timestamp : -qqq = don't show Total DISK READ/Actual DISKREAD after every iteration : -n 20 = quit after 20 seconds (default never quits) : tee -a = append to log file as well as show on stdout iotop -botqqq -n 20 | tee -a /var/log/iotop Download missing depedencies automatically on Debian/Ubuntu [apt-get(8)]: sudo apt-get --fix-broken install Search for packages names not descriptions [apt-get(8)]: sudo apt-get --names-only install Insert text at beginning of file without sed: cat [file] | perl -pe 'BEGIN { print "[text]\n" }' > [outputfile] Insert text without abusing cat in perl (with backup file): perl -i.bak -pe 'print "[text]\n" if $. == 1;' [file] Insert text at the end of the file with perl (with backup file): perl -i.bak -ne 'print $_; print "[text]\n" if eof;' [file] Show compile flags for Nginx: nginx -V 2>&1 | tr -- - '\n' | grep _module Linux print: lp -h [cups_server]:[cups_port:-631] -d [destination_printer_name] -o [print_job_options] [filename] Check Shared Memory Segments: ipcs -m Find process attached to shared memory segment lsof | grep [shmid from ipcs] Bash reset IFS (http://redsymbol.net/articles/unofficial-bash-strict-mode/) IFS=$'\n\t' Debug bash scripts, add: set -x Allow traffic via ufw: sudo ufw allow from [ip/any] to [host/any] port [any/port-num] proto [upd/tcp/whatevs] RCS Bulk Commit all files in directory ci -l -t-'[file descriptor]' -m'[commit message]' * Edit last command with $EDITOR fc Check default runlevel for system: grep ^id /etc/inittab List services/runlevels chkconfig: chkconfig --list Add service to startup with chkconfig: chkconfig --level 235 [service] on Remove service from startup with chkconfig: chkconfig --del [service] List service runlevels without chkconfig: ls -l /etc/rc*.d Add service to startup with update-rc.d: update-rc.d [service] defaults Remove service from startup with update-rc.d: update-rc.d -f [service] remove OpenSSL generate 2048bit RSA (genpkey supreceeds gen[alg]) [openssl(1)/genpkey(1)] openssl genpkey -algorithm RSA -out key.pem -pkeyopt rsa_keygen_bits:2048 OpenSSL generate CSR from key, prompts for input, "Common Name" is FQDN [openssl(1)/req(1)] openssl req -new -key privkey.pem -out cert.csr OpenSSL find key length from an rsa private key [openssl(1)/rsa(1)] openssl rsa -in [private_key] -text -noout Umask defaults set in /etc/login.defs [pam_mask(8)] Debian 022 files: 0666 & ~022 = 0644 -rw-r--r-- directorys: 0777 & ~022 = 0755 -rwxr-xr-x CentOS/RHEL 002 files: 0666 & ~002 = 0644 -rw-rw-r-- directorys: 0777 & ~002 = 0755 -rwxrwxr-x GIT --- Git checkout last checked out branch: git checkout - Git stash unstaged commits [git-stash(1)] git stash Git show stash (just changed files) at reference 0 [git-show(1)] git show stash stash@{0} Git show contents of stash at stash ref 1 [git-stash(1)] git stash show -p stash@{1} Git list all stashes [git-stash(1)] git stash list Git list repo contribution authors, order by # of commits [git-shortlog(1)] git shortlog -sn Git create patch file from the last commit[git-format-patch(1)] git format-patch HEAD~ --stdout > <file.patch> Git check what's in a patch file [git-apply(1)] git apply --stat <file.patch> Git test patch file before applying to repo [git-apply(1)] git apply --check <file.patch> Git apply patch file [git-apply(1)] git apply <file.patch> Git show history and diff of a file [git-log(1)] git log -p -- <path/to/file> Git search for deleted file [git-log(1)] # Search for file path git log --diff-filter=D | grep <deleted file name> # Find the delete commit id git log --all -- <deleted/file/path> Git check ignored patter matched [git-check-ignore(1) v.1.8.5+) git check-ignore [path] Git revert merge git revert -m 1 [sha of merge commit] Git diff your branch with upstream: git log HEAD..@{u} Git check status on a directory you're not in GIT_REPO="$HOME/hi" git --work-tree="$GIT_REPO" --git-dir="$GIT_REPO/.git" status Git list remotes: git remote -v Git change remote: git remote set-url [remote-name] [new-url] git remote set-url origin https://something Mongo --- Mongo show dbs with stats show dbs Mongo switch dbs use [dbname] Mongo show collections show collections Mongo show collection stats db.[col].stats() Mongo show collection stats in kilobytes db.[col].stats(1024) Mongo pretty print/limit output db.[col].find().limit(1).pretty() MySQL --- MySQL Find Users SELECT * FROM mysql.user WHERE User like '%whatever%'\G MySQL Add User to db GRANT USAGE ON *.* TO '<user>'@'%' IDENTIFIED BY '<password>'; MySQL Give User Access GRANT SELECT, INSERT, UPDATE, DELETE, EXECUTE ON <database>.* TO '<user>'@'%' MySQL Delete User DROP USER '<user>'@'%'; FLUSH PRIVILEGES; MySQL Find User Privileges SHOW GRANTS for '<username>'; -- or SHOW GRANTS FOR CURRENT_USER; MySQL Revoke User Privileges REVOKE INSERT ON *.* FROM '<username>'@'%'; MySQL Rename User RENAME USER 'jeffrey'@'localhost' TO 'jeff'@'127.0.0.1'; MySQL Set Password SET PASSWORD FOR 'bob'@'%.example.org' = PASSWORD('cleartext password'); MySQL Update User Password UPDATE mysql.user SET Password=PASSWORD('cleartext password') WHERE User='bob' AND Host='%.example.org'; FLUSH PRIVILEGES; MySQLDump a Single database mysqldump --compress -h localhost -u [username] -p --quick --single-transaction [db_name] > [dumpfile] MySQLDump databases -- NOTE TO USE --add-drop-database YOU MUST USE --databases mysqldump -h localhost -u [username] -p --add-drop-database --skip-comments --routines --compress --quick --single-transaction --databases [db_name] > [dumpfile] MySQLDump all DBs like [pattern] mysqldump -h localhost --quick --single-transaction [db_name] `mysql -ND [db_name] -h localhost -e "show tables like '[pattern]'" | awk '{ printf $1" " }'` > dumpfile.sql MySQL Add index: ALTER TABLE `table` ADD INDEX `column_name` (`column_name`) ALTER TABLE `account_agreement_pricebook` ADD INDEX `pricebook_id` (`pricebook_id`); MySQL Show indexes: SHOW INDEX FROM `table`; MySQL Show available engines mysql> show engines\G MySQL Enable engine INSTALL PLUGIN [engine] SONAME 'ha_[engine].so'; The following plugins are installed into the OS but not into MySQL: ha_archive.so - archive ha_blackhole.so - blackhole ha_example.so - example ha_innodb_plugin.so - InnoDB Plugin MySQL Replication things to know: * REPLICATION SETUP: http://plusbryan.com/mysql-replication-without-downtime * Monitor Seconds_Behind_Master * Monitor Exec_Master_Log_Pos * Run SHOW PROCESSLIST;—take note of the SQL thread to see if it is processing long running queries. * Keep an eye on master_db_host:/var/log/mysql/slow.log—this is a log of the longest-running mysql queries—try to optimize 'em * SHOW PROCESSLIST; (or SHOW FULL PROCESSLIST;) on the Slave: * there should be two DB Connections whose user name is `system user` * One of those DB Connections will have the current SQL statement being processed by replication. * As long as a different SQL statement is visible each time you run SHOW PROCESSLIST;, you can trust mysql is still replicating properly. MySQL Replication Last_Error: Duplicate Key Entry stop slave; set global mysql_slave_skip_counter = 1; start slave; -- repeat :) Is MySQL Slave Processing Relay Logs? STOP SLAVE IO_THREAD; SHOW SLAVE STATUS\G -- check Exec_Master_Log_Pos -- SHOW SLAVE STATUS\G -- if it doesn't move—it's working—run: -- START SLAVE IO_THREAD; PostgreSQL show user defined types SELECT n.nspname AS schema, pg_catalog.format_type ( t.oid, NULL ) AS name, t.typname AS internal_name, CASE WHEN t.typrelid != 0 THEN CAST ( 'tuple' AS pg_catalog.text ) WHEN t.typlen < 0 THEN CAST ( 'var' AS pg_catalog.text ) ELSE CAST ( t.typlen AS pg_catalog.text ) END AS size, pg_catalog.array_to_string ( ARRAY( SELECT e.enumlabel FROM pg_catalog.pg_enum e WHERE e.enumtypid = t.oid ORDER BY e.oid ), E'\n' ) AS elements, pg_catalog.obj_description ( t.oid, 'pg_type' ) AS description FROM pg_catalog.pg_type t LEFT JOIN pg_catalog.pg_namespace n ON n.oid = t.typnamespace WHERE ( t.typrelid = 0 OR ( SELECT c.relkind = 'c' FROM pg_catalog.pg_class c WHERE c.oid = t.typrelid ) ) AND NOT EXISTS ( SELECT 1 FROM pg_catalog.pg_type el WHERE el.oid = t.typelem AND el.typarray = t.oid ) AND n.nspname <> 'pg_catalog' AND n.nspname <> 'information_schema' AND pg_catalog.pg_type_is_visible ( t.oid ) ORDER BY 1, 2; PostgreSQL show queries the generate `\d...` commands \set ECHO_HIDDEN PostgreSQL equivalent of MySQL's `\G' <query>\x\g\x PROGRAMMING SPECIFIC --- PHP Find PHP CLI extension directory: php -i | grep 'extension_dir' PHP find pecl config extension directory: pecl config-show | grep 'ext_dir' Bundler show gems with path bundle list --paths NPM ingore SSL problems npm config set strict-ssl false Upgrade Node via NPM sudo npm cache clean -f && sudo npm install -g n && sudo n stable Weechat. --- Weechat add a server /server add [SERVER_NICK_NAME] server.host.name/+6667 -autoconnect Weechat remove a server /server del [SERVER_NICK_NAME] Weechat toggle nicklist /bar toggle nicklist Weechat set nicklist postion /set weechat.bar.nicklist.position top Weechat bind nicklist key /key bind meta-f /bar toggle nicklist Weechat bind newline glitch key /key bind meta-g /set weechat.look.eat_newline_glitch toggle Weechat fix formatting /set weechat.look.prefix_align none /set weechat.look.align_end_of_lines time Weechat disable fix formatting /set weechat.look.prefix_align right /set weechat.look.align_end_of_lines message Weechat split window horizontally /window splith [percentage] Weechat close split window /window merge Weechat jump around splits use F7/F8 Weechat add more nickname colors /set weechat.color.chat_nick_colors red,green,brown,blue,magenta,cyan,white,lightred,lightgreen,yellow,lightblue,lightmagenta,lightcyan DUMB --- - Generate a list of your most used commands— history | sed "s/^[0-9 ]*//" | sed "s/ *| */\n/g" | awk '{print $1}' | sort | uniq -c | sort -rn | head -n 100 > commands.txt - The Useless Use of `cat` Award - http://partmaps.org/era/unix/award.html#cat - Terminal Keynote (Ruby) - https://github.com/fxn/tkn - Boom. Motherfucking Text Snippets on the command line - http://zachholman.com/boom/ - Spark sparklines for your shell - http://zachholman.com/spark/ - Lolcat - https://github.com/busyloop/lolcat - cowsay - https://github.com/schacon/cowsay - Ponysay - https://github.com/erkin/ponysay - FIGlet - http://www.figlet.org/ - Libcaca - http://caca.zoy.org/wiki/libcaca - Toilet - http://caca.zoy.org/wiki/toilet - Boxes - http://boxes.thomasjensen.com/ - Cadubi - http://langworth.com/pub/cadubi/ - img2txt - http://hit9.org/img2txt/ - Nyancat - https://github.com/klange/nyancat - CMatrix - http://www.asty.org/cmatrix/ - janbrennen rice - https://github.com/janbrennen/rice - dotshare.it - http://dotshare.it - linux logo - http://freecode.com/projects/linuxlogo - bb - http://aa-project.sourceforge.net/bb/ - Powerline Font symbols - https://powerline.readthedocs.org/en/latest/fontpatching.html - Stupid Programmer Tricks/Starwars gifs - http://rarlindseysmash.com/posts/stupid-programmer-tricks-and-star-wars-gifs