Managing Input and Output Streams in the Linux Shell
In this Linux tutorial, I teach how to manage input and output streams in the command line through the use of redirection and piping.
Introduction
Redirecting the standard input and output streams is very useful. So, what are the standard streams? They are:
- standard input
- standard output
- standard error
Then, we will pipe output from one command to the input of another. After that, we will go over sending output to standout output and a file.
Standard Streams
Bash, the default Linux shell, receives input and sends its output as streams of characters. We usually type the input. Output is the stream of characters you see in your window. This stream of characters is often the result of a command.
“Stdin” is the standard input stream. The primary purpose is to provide input to commands. “Stdout” is the standard output stream and displays the output from commands. “Stderr” displays error output from commands.
Input streams provide input to programs. We do this through the use of the terminal. Output streams show us the command output. If there are errors, the standard error stream will display instead.
An important distinction to I need to make here is this is file redirection. Information is taken from a file or sent to a file. This is what redirection is for.
File Descriptors
Each type of input and output will have a file descriptor associated with it. These are:
- stdin = 0
- stdout = 1
- stderr = 2
Redirection is about file descriptors. When Bash opens, the file descriptors for stdin, stdout, and stderr, are opened too. File descriptors are interesting. You can make more of them, read from them, write to them, and close them. You use them in front of the redirection operator to say what you want to do. If you do not use them, the default is the “1” option, or output.
Redirect Output
Let’s start with redirecting output to a file. I do this a lot and is incredibly useful. The first thing we need to talk about is the output redirection operator, > or >>. They do different things, so don’t get confused.
If I redirect output from one command with >, the target file is created or overwritten. You must have proper permissions to write to the file. This action will destroy any data in the file, if it was already there.
ls -l 1> contents.txt
The file, contents.txt is created. The output of the previous command is added to the file.
cat contents.txt
Now, lets use the >> operator. If the output file does not exist, it will be created. If the file exists, the new data will be added on to the end.
pwd 2>> contents.txt
cat contents.txt
Since we used the >> operators, the information from “pwd” was added to the end of the file.
Another useful process, is to redirect the stderr to a file, so you can examine it later. You do it like this:
ls x 2>> error.txt
This saves the output error of the previous command to the file.
Another nice trick is to redirect the output and error information to a file at once.
ls x &>> error.txt
As you can see, the information is there.
Redirect Input
This is capturing an output and delivering it as the input to something else. Capturing a file’s contents is often done. We can then process the contents. Something I have had to do in the past is sort a file’s contents alphabetically and display it. We can do this by using:
sort 0< pokemon.txt
This text file has a list of Pokemon in it, and I want to display the list alphabetically. This command will do that. The 0 denotes the file descriptor for standard input. By the way, this is not the work example I was talking about earlier. I just made this list up for this example.
So, with input indirection, you can process the contents of a file with any number of commands. Your imagination is the limit.
You can also redirect this information again. For example, if I wanted to send the sorted results to a new file, it is easy to do.
sort 0< pokemon.txt 1> sorted-pokemon.txt
This command takes the contents of the original file and sorts it alphabetically. Then, it creates a new file with the sorted information. If you have to do this often, this is an excellent technique to know.
Piping Output
Piping is like redirection. It has some differences, though. It is most often used to take the output of one command and present it as input to another command. You can chain together several commands and pipes. The commands can use any parameters that apply. This allows nice flexibility in accomplishing tasks.
The original input can also come from a file. You can use the typical input redirection for this. Then, you just setup a chain of commands using pipes, depending on what you want to accomplish.
The main purpose of using pipes is to process information from commands and to send it to other commands. Any command that accepts stream input can be used.
- sort
- grep
- ls
These above commands are very popular to use with pipes. Like I mentioned above, many commands can use pipes.
ifconfig | grep 10
You can also do something like this:
ls | wc -l
This will give you a list of files in current directory.
There are many possibilities with pipes. Experimentation will help a lot and get you used to the command line too.
Executing with Xargs
The xargs command reads from your standard input. It can build and execute subsequent commands. It will use this input as parameters. If there is no additional command specified to use with xargs, it will use echo by default.
The "xargs" command takes multiple line input and provides it in one line.
xargs < pokemon.txt
Lists the contents of this file and it does it on one line, separated by spaces.
If you have several files in a directory, you can find the wordcount and other information easily.
ls *.txt | xargs wc
This will list all the text files in the current directory and give the wordcount information for all at once. There is a lot more you can do with "xargs", it is a helpful command.
Splitting Your Output
Splitting your output is useful in some situations. For example, when you want to display output on screen and send it to a file at the same time. There are multiple ways of doing this but using the “tee” command is probably the best.
You use the “tee” command with a pipeline. A filename is usually the argument. Let’s use pur Pokemon example from above.
sort pokemon.txt | tee sorted-pokemon.txt
The sort command does it work and then we pipe it to tee. Then, we see the sorted results on screen and that same information is saved to our sorted-pokemon.txt file.
Let us use the previous wordcount command for another example. We can get this information, display it on screen, and send it to another text file at the same time.
wc -l pokemon.txt | tee pokemon-count.txt
Conclusion
So, in this section we have learned all about file redirection and piping. These are both great techniques. With the above examples, you should do those examples so you can see how they work. The first time you are asked to sort a large text file alphabetically, I hope you will remember these examples and how Linux makes such tasks trivial.
Thanks For Reading
Thank you for reading this, I really appreciate it.
If you would like to join my newsletter, you can do so here:
If you need a suggestion on what to read next, then try these Linux articles: