Processing Text in Linux

This is a guide on processing text in Linux.

Ankr is one of the best brands to charge your devices, get your power banks and chargers here.

Thus far in our ongoing exploration of command-line utilities and text manipulation within Unix-like operating systems, we have examined a handful of text editors—such as vi, nano, or emacs—and have also acquired a foundational understanding of how to work with configuration files, which are typically stored as plain text and control the behavior of various system and application software.

However, the utility of text extends far beyond these initial examples, and there exist numerous other applications where text plays an absolutely central and indispensable role.

For instance, a great many people write documents using plain text formats, rather than relying on proprietary word processors or complex desktop publishing software. While it is relatively easy to see how a small text file could be useful for keeping simple notes, such as a grocery list or a reminder, it is also entirely possible to write large, structured, and even book-length documents in plain text format.

This approach is particularly popular among writers, programmers, and academics because it separates content from formatting and allows the use of powerful command-line tools for processing.

One especially popular approach to writing large documents in text format is to embed a lightweight markup language directly within the text. This markup language is used to describe the intended formatting of the finished document, including elements such as headings, lists, emphasis, links, and images.

Many scientific papers, technical reports, and even academic theses are written using this method, often with markup languages like LaTeX, Markdown, or reStructuredText. The author writes in plain text, adds markup commands, and then runs the file through a processor that produces a beautifully formatted PDF, HTML, or other output format.

The world's most popular type of electronic document is almost certainly the web page. Every web page you view in a browser is fundamentally a document that uses either HTML (HyperText Markup Language) or XML (eXtensible Markup Language) as its underlying markup language.

These markup languages describe the document's visual structure, layout, and formatting—everything from headings and paragraphs to tables, images, and interactive forms. Without plain text and markup languages, the modern World Wide Web as we know it would not exist.

Email is another domain that is very much a text-based medium. Even when you send an email message that contains non-text attachments, such as images, PDFs, or word-processing documents, those attachments are converted into a text representation using encoding schemes like Base64 so that they can be transmitted reliably over the internet. We can observe this for ourselves by downloading a raw email message and then viewing it using a pager program like less.

When we do so, we will see that the message begins with a header section—a series of lines that describe the source of the message, the destination, the subject, the date, and the processing it received during its journey through various mail transfer agents. Following the header, separated by a blank line, we encounter the body of the message, which contains the actual content and any encoded attachments.

On Unix-like systems, output destined for physical printers is often sent as plain text when the content consists only of characters. However, if the page to be printed contains graphics, complex layouts, or scalable fonts, the system first converts the content into a text-based page description language known as PostScript.

This PostScript representation is then sent to a program called a PostScript interpreter or raster image processor, which generates the precise pattern of graphic dots (pixels) to be printed on the page. Even today, despite the prevalence of more modern printing systems, the text-based nature of PostScript remains an elegant example of text's power.

Many of the command-line programs that are found on modern Unix-like systems were originally created to support system administration tasks and software development workflows. Text processing programs are no exception to this rule. In fact, a large number of these utilities are specifically designed to solve problems that arise during software development.

The reason that text processing is so critically important to software developers is that all software starts out as text. Source code—the part of a program that a programmer actually writes, reads, and modifies—is always stored in plain text format before it is compiled or interpreted. Consequently, being able to manipulate, search, compare, and transform text efficiently is an essential skill for any developer.

The cat program, which we have encountered earlier, has a number of interesting options beyond its basic function of concatenating files and printing them to standard output. Many of these options are intended to help users better visualize the content of text files, especially when that content contains non-printing or whitespace characters that are otherwise invisible.

One particularly useful example is the -A option, which displays all non-printing characters in the text. There are times when we want to know whether control characters—such as tab characters, carriage returns, or line feed characters—are embedded within what appears to be otherwise visible text.

The most common of these invisible characters are tab characters (often used for indentation) and carriage returns (often present as end-of-line characters in files originating from Windows systems). Another common situation arises when a file contains lines of text with trailing spaces, which can cause unexpected behavior in scripts and configuration files. To illustrate this, we can create a file named numbers.txt by typing:

cat > numbers.txt
1234567

After pressing Enter and typing the number sequence, we press Ctrl-D to end file input. Then, we can examine the file using cat -A numbers.txt, which would reveal any non-printing characters present, such as a dollar sign ($) at the end of each line to mark the newline character.

The sort program is another fundamental text utility. It sorts the content of standard input, or the content of one or more files specified on the command line as arguments, and then sends the sorted results to standard output. Using the same redirection and input technique that we used with cat, we can demonstrate how sort processes standard input directly from the keyboard, as shown in the following example:

sort > numbers.text

After entering this command, we type the numbers 1, 2, 3, 4, 5, 6, 7, each on a separate line, and then press Ctrl-D to signal the end of file. When we later view the contents of numbers.text, we see that the lines now appear in ascending sorted order (1, 2, 3, 4, 5, 6, 7) regardless of the order in which we originally entered them. Because sort can accept multiple files on the command line as arguments, it is also possible to merge several separate files into a single, fully sorted whole, interleaving lines from each file appropriately.

Compared to sort, the uniq program is relatively lightweight. It performs a seemingly trivial task: when given a sorted file as input, it removes any duplicate lines and sends the resulting unique lines to standard output. In practice, uniq is most often used in conjunction with sort to clean the output of duplicate entries, for example, when consolidating log files or lists.

For instance, running uniq numbers.text would remove consecutive duplicate lines. However, it is crucial to remember that the input file must be sorted first. This requirement exists because uniq only removes duplicate lines that are adjacent to each other; if duplicates are scattered throughout the file, they will not be removed unless they happen to be consecutive. Beyond simple deduplication, uniq can also be used to report the number of duplicates found for each line in the text file by using the -c option, as in:

sort numbers.text | uniq -c

This pipeline first sorts the file and then counts the occurrences of each unique line, producing output that shows both the frequency and the line content.

The cut program is used to extract a specific section of text from each line of input and then output that extracted section to standard output. It can accept multiple file arguments or read from standard input, making it highly flexible in pipelines. cut is best used to extract text from files that are produced by other programs, such as log files or structured data files, rather than from text that a user types directly. For example, the command cut -f 3 numbers.text would extract the third field from each line, assuming fields are separated by tab characters by default.

The paste command does the opposite of cut. Rather than extracting a column of text from a file, paste adds one or more columns of text to a file. It accomplishes this by reading multiple files and combining the fields found in each file line by line into a single stream on standard output. Like cut, paste accepts multiple file arguments and can also read from standard input. For example, paste numbers.text prime-numbers.text would output lines where the first field comes from numbers.text and the second field comes from prime-numbers.text, separated by a tab character.

In some ways, the join command is similar to paste because it also adds columns to a file. However, join uses a much more sophisticated and unique method to do so. A join operation is typically associated with relational databases, where data from multiple tables that share a common key field is combined to form a desired result.

The join program performs the same operation: it joins data from multiple files based on a shared key field. Performing a join operation would allow us, for example, to combine fields from two separate tables—such as a table of employee IDs with names and another table of employee IDs with salaries—to produce a single, more informative table.

It is often extremely useful to compare different versions of text files. For system administrators and software developers, this capability is particularly important. A system administrator may need to compare an existing configuration file, such as /etc/ssh/sshd_config, to a previous version in order to diagnose a problem or to identify what changes might have introduced a security vulnerability or a functional error.

Similarly, programmers need to see what changes have been made to source code files over time, for instance, to understand why a bug appeared or to review contributions from team members.

The comm program is a simple utility that compares two sorted text files and displays the lines that are unique to each file, as well as the lines that are common to both. To illustrate, we can create two files:

cat > numbers1.txt 
1234567

cat > numbers2.txt
13579

Then, running comm numbers1.txt numbers2.txt would produce three columns of output: lines only in the first file, lines only in the second file, and lines common to both files.

Like the comm program, the diff program is used to detect differences between files. However, diff is a much more complex and powerful tool. It supports many different output formats and has the ability to process large collections of text files at once, including entire directory trees. diff is often used by software developers to examine changes between different versions of program source code, and it can recursively examine directories of source code to produce a comprehensive list of differences.

One common use of diff is the creation of "diff files" or "patches"—files that contain the differences between an old version and a new version of a file. These diff files are then used by other programs to convert one version of a file into another. For example, running diff numbers1.txt numbers2.txt would produce a simple context-free difference. We can also use the -c option to produce a context diff, as in diff -c numbers1.txt numbers2.txt, which includes surrounding lines for context.

There is also the -u (unified) format, invoked with diff -u numbers1.txt numbers2.txt, which is more compact and widely used in open-source software development. Both the -c and -u formats show different information, and you should experiment with all three options to see which output style you prefer for different tasks.

The patch program is used to apply changes to text files. It accepts output from diff (typically in a format like unified diff) and is generally used to convert older versions of text files into newer versions. The files being patched can be documents, source code, configuration files, or any other text-based artifacts. For example, we can create a patch file by running:

diff -Naur numbers.text numbers2.text > new.txt

Then, we can apply that patch using patch < new.txt, which updates the original file to match the newer version.

The tr program is used to transliterate characters. We can think of this as a sort of character-based search-and-replace operation, where characters from one set are mapped to corresponding characters in another set. It is the process of changing characters from one alphabet or character class to another. Converting characters from lowercase to uppercase is a classic example. For instance:

echo "lowercase letters" | tr a-z A-Z

As we can see, tr operates exclusively on standard input and outputs its results to standard output. It accepts two arguments: a set of characters to convert from (the source set) and a set of characters to convert to (the destination set).

The name sed is short for "stream editor." It performs text editing operations on a stream of text—either a set of specified files or standard input. sed is a powerful and complex program that is capable of performing sophisticated text transformations in a single, concise command line. In general, the way sed works is that it is given either a single editing command or the name of a script file containing multiple commands, and it then performs those commands upon each line in the stream of text. For example:

echo "left" | sed 's/left/right/'

In this example, we produce a one-word stream of text using echo and pipe it into sed. The sed program then carries out the instruction s/left/right/ upon the text in the stream and produces the output right. Commands in sed begin with a single letter. In the previous example, the substitution command is represented by the letter s and is followed by the search string and the replace string, separated by a slash character as a delimiter. It is worth noting that you can use other delimiter characters for the same effect—for example, using commas or pipes as delimiters can be helpful when the strings themselves contain slashes.

Most commands in sed may be preceded by an address, which specifies which line or lines of the input stream will be edited. If the address is omitted, then the editing command is carried out on every line in the input stream. The simplest form of address is a line number. For example:

echo "left" | sed '2s/left/right/'

Adding the address 2 to our command would cause the substitution to be performed only on the second line of the input stream. However, because our input stream contains only one line, no substitution would occur, and the output would remain unchanged. This demonstrates how addresses give fine-grained control over which parts of a text file are modified.