In this lecture, I'll be talking about pipes, which are one of the most useful command line features 1
in Linux. Pipes are used in countless ways and are just about everywhere. 2
The Unix and the Linux design philosophy is to have many small utilities that perform very well, just 3
one dedicated function without the needless functionality. In Linux 4
you write programs to do one thing and to do it well; and then you make them work together. 5
Let's take a look at this image. Every command or program we run has three data streams connected to 6
it: STDIN (0) or Standard input which represents data fed into the program or the data 7
the program is working with; STDOUT (1) or Standard output which is data printed by the program, by 8
default to the terminal and STDERR (2) or Standard error for error messages and also defaults 9
to the terminal. 10
Using the pipe symbol, which is a vertical bar, we can connect two our more commands at the time. 11
This supports the initial Unix philosophy of chaining programs together to complete complex tasks. 12
So a pipe lets us send the output of one command as input to another one. Piping and redirection are 13
the means by which we may connect these streams, input output and error among programs and files 14
to direct data in interesting and useful ways. 15
Let's take a look at this mechanism. 16
I'm running: ls -lSh /etc 17
The command has listed all files in the etc directory sorted by size in human readable. There are lots of 18
files, and I finished up seeing only the last ones, the smallest files; the largest ones are at the 19
beginning and I have to scroll to see them. 20
What I could do is connect the ls command with the head command. Using a pipe I'll send the output of 21
the first command, which is the ls as input to the second command, which is head like this: 22
| head 23
And we are seeing only the head or the first 10 files of the output and the those are the largest ones. 24
It's common and recommended to add the white space before and after the pipe symbol. 25
But that is not, however, mandatory, this isn't an error. 26
We can pipe together as many commands as we wish. In the next example, I'll filter and display only 27
the 20th file of the output. 28
It's just an example to see how it works. 29
So head -n 20 | tail -n 1 30
This is the 20th line of the output. 31
Let's take a closer look: at the output of the ls command becomes the input for head, which will display 32
the first 20 lines. 33
These first 20 lines, which are the output, are fed as input into the next command, which is tail 34
and the tail command will display only the last one, which is in fact the 20th line of the output. 35
So the pipe takes the output from one command and uses it as input for another one. 36
And the last example! I want to find out how many authentication failures took place. There isn't 37
such a command in Linux, so I have to build my own using pipes. 38
First, using cat, I'm displaying the entire file cat /var/ log/ auth.log, this is the file where 39
authentications are saved. 40
This is the file! I can also add a -n to see the line numbers. 41
Then the output of the cat command will be given as input to another comment called grep. Grep is a command 42
line utility that searches for a string inside a file or a text and displays the lines containing the 43
string. 44
So | grep. 45
-ai and the spring I'm searching for " authentication, failure" 46
And there are some matches; there are more lines that contain the string. 47
These are all authentication failures. 48
And finally, I want to see how many lines were found. The output of the grep command will be given 49
to another command called wc, which counts how many lines were given. 50
So | wc -l 51
And it returned 6, so there were 6 authentication failures saved in the log file. 52
Let me tell you a few words about wc, which is short for word count. 53
It prints newline, word, and byte counts for each FILE. 54
wc /etc/passwd 55
There are 48 lines. 56
83 words and 2825 characters in that file. This command 57
is used to get different statistics. 58
I've just found out that there are 48 accounts on the system. 59
If you only want to see the line counts, then use option -l: wc -l /etc/group 60
There are 76 groups on the system. 61
wc-w will display only the word counts, and -c only the character counts. 62
So wc -w /usr/share/dict/american-english 63
This is a standard dictionary that you'll find on Ubuntu. 64
And there's this number of words in this American English Dictionary file. 65
Let's see how many characters are in the file. 66
And there are almost one million characters in the file. 67
You can also use it with a pipe, like see how many users are logged in. 68
Who is displaying the logged in users, so who | wc -l 69
There's only one user logged in. 70
One of the main purposes of piping is filtering. 71
You use piping to filter the contents of a large file to find the particular string or a word 72
for example. Note that when you use a pipe, you are not running a single command, but each command 73
in the pipe starts. 74
So in this example, in this one, I started three processes, each command starting a process: cat, grep 75
and wc. 76
Take your time to exercise and deeply understand how pipes work because this is a key concept in Linux 77
and you'll find it everywhere. 78
The pipe is one of the most used shell features.