We are going to attempt to cover the following commands. The examples provided are not exhaustive of what each command can do.
Command | Description |
---|---|
ls | Directory listing |
mkdir | Make a directory |
rmdir | Remove a directory |
rm | Remove a file |
cp | Copy a file |
mv | Rename a file |
cat | Print file contents to screen |
wc | Word count |
grep | Search files matching a pattern |
sed | Editor and text transformer |
cut | Print sections from a line |
awk | Pattern matching and text processing language |
sort | Sort a file |
uniq | Only print unique entries |
tail | Print the last x lines in a file |
head | Print the first x lines in a file |
find | Find a file |
man | Command to view manuals for each command |
chown | Change ownership of a file or directory |
chmod | Change permissions of a file or directory |
touch | Create an empty file or set a timestamp on a file |
The tilde ( ~
) is a short hand for your home directory. If you type the following you will go back to your home directory.
cd ~/
I pulled the 2018 Columbus Clippers player stats to provide the file we are going to work with today.
The file is called findme.txt and it is somewhere in the /tmp directory but we need to find it first. If you are seeing this document and it is outside of the training class I gave, download the findme.txt file and save it to your /tmp directory.
- Finding the file
find /tmp -name 'findme.txt'
- The location of the file is a pain. Let copy it to a better location
cp path/to/findme.txt ~/findme.txt
- Lets rename the file
mv findme.txt data.txt
- Let's look at the file
cat data.txt
- How many lines are in the file
wc -l data.txt
- Show only the first 5 lines of the file
head -5 data.txt
- Show only the last 3 lines of the file
tail -3 data.txt
- Show only right fielders,
RF
grep 'RF' data.txt
- Sort the player list by name alphabetically
sort data.txt
cat data.txt | sort
Redirection allows you to send the output of a command to file. The following redirection will redirect STDOUT (standard out).
The STDOUT redirector uses the >
symbol.
cat FILE > file.txt
- Append to a file
cat FILE > file.txt
cat FILE >> file.txt
More redirection will be covered later.
- Since the header is still part of output, lets remove that
tail -28 data.txt > players.txt
Awk is a scripting language and can do much more than what is shown here.
- Show only player names and home runs
awk -F "," '{ print $1 " " $3 }' players.txt
- Change the previous output to say `PLAYER has scored HR home runs'
awk -F "," '{ print $1 " has scored " $3 " home runs" }' players.txt
Cut can be used the same way awk can be when it comes to printing columnar data.
- Show only player names and home runs
cut -d , -f 1,3 players.txt
- Prints out just the names of all right fielders
grep "RF" players.txt | cut -d , -f 1
Cut allows you to print starting from a chosen column to the end of a the file.
- Prints out the player position and home runs
cut -d , -f 2- players.txt
- Add the text
Player:
in front of each player
sed 's/^/Player: /' players.txt
The `^` means the start of a line.
A `$` means the end of a line.
- Sort the player list, without the header, with the most home runs first
sort -t , -k 3 -n -r players.txt
- Show only the player's first names and how many times that name appears
awk -F "," '{print $1}' players.txt | sort | awk '{print $1}' | uniq -c | sort
- Print the total number of home runs scored by the team
awk -F "," '{total += $3} END {print total}' players.txt
The tail command can also be used to follow the output of a file being written to. Normally, this would be a log file.
The -f
argument means follow
.
tail -f /tmp/current_time.log
The print_time.sh script will output the current time to /tmp/current_time.log which can be following with tail -f
Grep provides some other useful arguments when doing different kinds of searches.
By default when grep is being use to search over multiple files you want the filename where matches were found to be returned.
The -H
argument shows filenames containing matches. On Linux this is a default action.
Sometimes a text string you are looking for is in a binary file but you want to ignore binary files.
The -I
argument ignores binary files.
By default grep performs a case sensitive search. What if you know the word you are looking for but you do not know th case?
The -i
argument provies case insensitive searches.
grep -i NAME /tmp/name*
If you know a directory tree of files contains something you are looking for you can perform a recursive search to look in all sub directories of the directory you perform the grep command.
The -r
argument will perform a recursive search.
Note: If you want to follow symbolic links use the -R
argument.
If you are dealing with files which contain many lines, 1000s, it is helpful for grep to return the line number of the match.
The -n
argument will return the line number.
Sometimes instead of searching for something you want to not search for something. A good example are looking at a large file while removing comments. Use the -v
argument to invert your match
grep -v '^#' /tmp/comments.txt
Sometimes you encounter files that have lot of empty lines and you desire to give rid of them so you only see the relevan text.
Recall earlier it was stated that ^
represents the start of a line and that $
represents the end of a line. In sed the d
modifier means delete
.
sed '/^$/d' /tmp/comments_newlines.txt
To extend the previous grep match inversion, we can remove the all lines that start with a # and then pipe the output to sed to remove new lines.
grep -v '^#' /tmp/comments_newlines.txt | sed '/^$/d'
When it comes to redirection there are three file descriptors that are primarily used.
Value | File Descriptor Name | Human Readable |
---|---|---|
0 | STDIN | Stanard In |
1 | STDOUT | Standard Out |
2 | STDERR | Standard Error |
In a large number of cases, you will really only ever deal with STDOUT. As we talked about earlier we know that >
redirects STDOUT. There is a also a concept of STDERR (standard error).
Normally, STDOUT and STDERR will both print out to the screen.
$ ./std_out_err.sh
I am writing to STDOUT
I am writing to STDERR
If you only eve want to see one of the outputs, redirect the other to /dev/null or to a file.
$ ./std_out_err.sh 1> /dev/null
I am writing to STDERR
or
$ ./std_out_err.sh 2> stderr
or
$ ./std_out_err.sh 2> /dev/null
I am writing to STDOUT
or
$ ./std_out_err.sh 1> stdout
or
./std_out_err.sh 1> stdout 2> stderr
Sometimes you want both STDERR and STDOUT to go to the same file.
./std_out_err.sh > both_stdout_stderr 2>&1
- Changing ownership of a file or directory
chown USER:GROUP [file|directory]
- Change owner to
root
of fileroot_owner
touch root_owner
chown root: root_owner
drwxr-xr-x 3 mike users 4096 Jul 11 21:33 Pictures
-rw-r--r-- 1 mike users 205 Sep 12 2017 rsync.exclude
-rwx------ 1 mike users 644 Sep 12 2017 rsync_me.sh
- --- --- ---
| | | |- other ---- rwx
| | |----- group ---- rwx
| | -------- user ----- rwx
|------------ type ----- d,c,l,s
r w x
| | |- 1
| |--- 2
|----- 4
chmod UGO [file|directory]
U - user
G - group
O - other
Create a read only file - owner only
touch readonly
chmod 400 readonly
ls -l readonly
-r-------- 1 mike users 0 Jul 28 15:22 readonly
Create a write only file - owner only
touch writeonly
chmod 200 writeonly
ls -l writeonly
--w------- 1 mike users 0 Jul 28 15:21 writeonly
Create an executable file only - owner only
touch execonly
chmod 100 execonly
ls -l execonly
---x------ 1 mike users 0 Jul 28 15:20 execonly
Create a read and write file only - owner only
touch readwriteonly
chmod 600 readwriteonly
ls -l readwriteonly
-rw------- 1 mike users 0 Jul 28 15:24 readwriteonly
-
What is the command for a read/write/executable file?
-
What is the command for a read only file for user, group, other?
This method is more powerful because it provides more options and flexibility but it can be more verbose when you want to set permissions.
chmod [ugoa]([-+=]([rwxXst] [file|directory]
Create a read only file - owner only
touch readonly
chmod u=r,g-rwx,o-rwx readonly
ls -l readonly
-r-------- 1 mike users 0 Jul 28 15:22 readonly
Create a write only file - owner only
touch writeonly
chmod u=w,g-rwx,o-rwx writeonly
ls -l writeonly
--w------- 1 mike users 0 Jul 28 15:21 writeonly
Create an executable file only - owner only
touch execonly
chmod u=x,g-rwx,o-rwx execeonly
ls -l execonly
---x------ 1 mike users 0 Jul 28 15:20 execonly
Create a read and write file only - owner only
touch readwriteonly
chmod u=rw,g-rwx,o-rwx readwriteonly
ls -l readwriteonly
-rw------- 1 mike users 0 Jul 28 15:24 readwriteonly
When you only want to change a certain permission on a set of files and/or directories, the text method is the best way. For example, if you want to only add write permissions to group ownership you can do something like this.
chmod g=w <files>
-
What is the command for a read/write/executable file?
-
What is the command for a read only file for user, group, other (everyone)?
Why are the following permissions bad?
-rwxrwxrwx 1 mike users 0 Aug 3 16:14 test
-
man pages
- man
command
- man
-
Julia Evans
She works on the infrastructure team at Stripe. She has a wonderful ability of distilling down concepts to their simplest parts,
https://www.tldp.org/guides.html
https://www.computerworld.com/article/2598082/linux/linux-linux-command-line-cheat-sheet.html
https://www.tecmint.com/linux-commands-cheat-sheet/
https://learncodethehardway.org/unix/bash_cheat_sheet.pdf
https://www.linode.com/docs/tools-reference/tools/
https://wiki.archlinux.org/index.php/File_permissions_and_attributes
https://en.wikipedia.org/wiki/File_descriptor
https://linuxacademy.com/ - This is a paid Linux training website
https://www.commandlinefu.com/commands/browse - Lots of different command line examples.