Stream Editor sed part 1

1. Stream Editor : sed

sed (Stream Editor) is a robust command-line tool utilized in Linux and Unix-like operating systems for the purpose of parsing and modifying text.

It operates as a non-interactive text editor, which indicates that it automatically processes text according to a predefined set of editing commands or scripts, without the need for user interaction during the editing process.

sed reads input text line by line, applies specified operations, and outputs the modified text.

Its primary use is to perform various text transformations, including:

• Searching and Replacing: Replacing occurrences of a pattern with another string * Deleting Lines: Removing lines that match a specific pattern or within a specified range
• Inserting and Appending Text: Adding new lines of text before or after existing lines.
• Printing Specific Lines: Displaying only the lines that match a pattern or a specific range.

sed was developed from 1973 to 1974 by Lee E. McMahon which linguistic insight fostered the characteristic text-processing—as distinct from text-formatting—capabilities of the Unix V7, a major release of the Unix operating system in 1979. sed was based on the interactive editor ed and the earlier qed.

2. Basic syntax

sed can be used in two ways :

In-line commands inside single quotes :

sed [OPTIONS] 'COMMANDS' inputfile(s)

Commands included in a script file , the same way as in awk seen previously :

sed [OPTIONS] -f scriptfile inputfile(s)

Some of the most frequently used options :

3. The substitute command s

sed has various commands that we will see later. The s command (substitute) is the most commonly used, it replaces matching regex with replacement portion, the syntax is :

s/pattern/replacement/flags

• s stands for the substitute command
• / is an idiomatic delimiter character to separate various portions of the command
• pattern stands for regular expression
• replacement refers to the replacement string
• flags are options to change the default behavior of the command

Example : we'll use 3 files for the various examples throughout this page, a log file called books.log and the same files as in awk examples (students.txt and bank.csv ) :

and the books.log :

3.1 Replace the first occurrence

Replace "success" with "achieved". By default, the sed command replaces the first occurrence of the pattern in each line and it won't replace the second, third...occurrence in the line :

sed 's/success/achieved/' students.txt

3.2 Replace the nth occurrence

To replace only the nth occurrence of a word in a line (here replace the second occurrence of "book" with "Preston" ):

sed 's/book/Preston/2' books.log

3.3 Replace all the occurrences

The substitute flag /g (global replacement) specifies the sed command to replace all the occurrences of the string in the line.

sed 's/books/Preston/g' books.log

3.4 Replace from the nth occurrence until the end

We can combine n and g to replace all the patterns from the nth occurrence of a pattern in a line :

sed 's/books/Preston/2g' books.log

3.5 Replace the string on a specific line number

Replace "500" with "1000" on the 11th line only

sed '11 s/500/1000/' books.log

3.6 Replacing String on a Range of Lines

Replace "success" with "achieved" from line 2 to line 8 :

sed '2,8 s/success/achieved/' students.txt

3.7 Replacing from nth line to the last line of file

Replacing String from line 2 to the last line of students.txt :

sed '2,$ s/success/achieved/' students.txt

3.8 Substitute a word at the beginning of a line only

sed '11 s/^book/livre/' books.log

3.9 Substitute the final word in a line

sed 's/success$/pass/g' students.txt

3.10 Add a comma after each number

The & command is a special character which we'll see later in further detail.

sed 's/[0-9]/&,/' students.txt

3.11 Edit file in place

By default, sed outputs the results of its alterations in the terminal (stdout), without modifying the original file. To directly apply the changes to the file, employ the -i (in-place) option. The syntax is : -i[SUFFIX], --in-place[=SUFFIX] If suffix is provided, it takes a backup of the original file, otherwise the original file overwritten.

sed -i 's/success$/pass/g' students.txt

Create a backup before making changes, the file sudents.txt.bak is created automatically :

sed -i.bak 's/success$/pass/g' students.txt

4. Basic commands : delete, write, read, execute, append, insert, change

4.1 delete

4.1.1 delete line number n :

sed '11d' books.log

4.1.2 delete several chosen lines :

Delete all 19th century books

sed '1d;2d;3d' books.log

4.1.3 delete the last line :

sed '$d' books.log

4.1.4 delete lines from number n to m

Delete all contemporary books

sed '4,10d' books.log

4.1.5 delete from line number n to last line :

In our case almost the same previous command

sed '4,$d' books.txt

4.1.6 delete pattern matching line :

The syntax is : sed '/pattern/d' books.log

delete, for example, Éric-Emmanuel Schmitt's books

sed '/Éric-Emmanuel Schmitt/d' books.log

4.1.7 Delete the line whose pattern does not match :

delete all lines except those containing Éric-Emmanuel Schmitt

sed '/Éric-Emmanuel Schmitt/!d' books.log

4.1.8 delete empty lines :

Delete all blank (empty) lines from the file

sed '/^$/d' students.txt

4.1.9 delete between two patterns

Delete all lines between two patterns, inclusive :

sed '/153/,/70/d' students.txt

4.2 write

The write command saves the pattern buffer contents into a file. SED creates a file with the given name and overwrites any existing file with the same name, so we have to be careful with name and extension.

sed -n 'w books.bak' books.log 

The file books.bak is saved with exactly the same content as books.log. This command is the same as the cp command but saving the pattern buffer allows us to do much more, for example save a file with contemporary books removed :

sed -n -e '4,$d' -e 'w 19thONLY.log' books.log 

Another example is pattern matching, so let's save in a separate file failing students :

sed -n '/fail/ w failing.txt' students.txt

or by author Éric-Emmanuel Schmitt's books :

sed -n '/Éric-Emmanuel Schmitt/ w Éric-Emmanuel-Schmitt.log' books.log

4.3 read

sed can read the content of a given file and display it when a pattern is matched. For instance, let's insert the contents of 19.log after the first to the 3rd lines.

echo "19th-century literature" > 19.log

sed '1, 3 r 19.log' books.log 

another example with matching pattern :

echo "failure: below the pass mark" > fail.txt

sed '/fail/ r fail.txt' students.txt 

4.4 execute ( works only in GNU sed )

The execute command, which is less common, runs a shell command from inside sed.

It's syntax : sed 's/REGEX/COMMAND/e' file

When there is pattern matching, the command is executed first and then the contents of the pattern buffer is displayed.

The whoami command :

 sed '/Éric-Emmanuel Schmitt/ e whoami' books.log 

Another example where the date is inserted before the first line :

 sed '1 e date' books.log

4.5 append after a line

The below command appends text after the specified 6th line in bank.csv :

sed '6 a payment,25-10-13,300,USD,3600' bank.csv

We can also use a matching pattern , so the text will be appended after the first occurrence of "USD" :

sed '/USD/ a payment,25-10-13,300,USD,3600' bank.csv

In this example "USD" appears once, but if there are multiple occurrences the text is appended after each match.

4.6 insert text before a line

In contrast to append, the insert command operates similarly but places a line before a particular position.

The below command inserts the latest operation before the 2nd line in bank.csv :

sed '2 i payment,25-10-13,300,USD,3600' bank.csv

With a matching pattern :

sed '/USD/ i currency : USD ' bank.csv

4.7 change

The change command allows to replace an entire line with a new one. It also works with a line range.

sed '3 c purchase,15-08-07,100,USD,2500' bank.csv

The same way as for the previous commands we use here a matching pattern :

sed '/Victor Hugo / c 1. Bug-Jargal | Victor Hugo | Format: Poche | Pages: 272 ' books.log

Below, we will replace several lines ( from 4 to 11) with a single line :

sed '4, 11 c 1. Bug-Jargal | Victor Hugo | Format: Poche | Pages: 272 ' books.log 

5. Active pattern space (pattern buffer) and Auxiliary hold space (hold buffer)

Two central buffers are used by Sed for text manipulation; they are identified as the pattern buffer (pattern space) and the hold buffer (hold space).

Pattern buffer :

Sed reads an input file or inline data line by line. The line that has been currently read is inserted into the pattern buffer which is the primary workspace where sed operates on the current line of input. Most used Sed commands, such as substitution (s), deletion (d), and printing (p), directly operate on the content of the pattern buffer. By default, the content of the pattern buffer is printed to standard output at the end of each processing cycle (unless the -n option is used), and then the pattern buffer is cleared to receive the next input line.

Hold buffer :

The hold space is a secondary buffer used for temporary storage of data that needs to be preserved across multiple lines or for more complex manipulations. Unlike the pattern buffer, the hold space, as the name denotes, maintains its content throughout cycles unless we intentionally modify it, and moreover it does not automatically print its contents.

Data cannot be directly manipulated within the hold space; instead, its contents must be transferred to the pattern buffer for processing. To interact with the hold space we can use the following commands :

To put it plainly and resume, the pattern buffer can be seen as a scratchpad where we actively work ( on the current line), and the hold buffer can be seen as a separate storage area where we can temporarily save information for later use just like a clipboard.

Next, we'll briefly review some elementary examples involving prior commands.

5.1 Append the content of the hold space to pattern buffer: G

The following command adds a space between lines :

 sed 'G' books.log

In the above example :

1. Sed processes a line and stores it in the pattern buffer.
2. G command appends the hold space to the pattern space preceded by a newline \n. Since the hold buffer is initially empty a newline will be appended with the pattern buffer content.

To insert two blank lines :

sed 'G;G' books.log

To use matching patterns :

The above command

5.2 Exchange the content of the pattern space and hold space : x

 sed '/Éric-Emmanuel Schmitt/{x;p;x;}' books.log

The above command inserts a blank line before each line that contains Éric-Emmanuel Schmitt's book.

The three commands {x;p;x;} are executed when the pattern matches :

1. x

   • exchange pattern ↔ hold , after that :
   • pattern buffer = empty (what was in the hold buffer)
   • hold buffer = "4. Oscar et la dame rose | Éric-Emmanuel Schmitt | Format: Poche | Pages: 112"

2. p

   • prints the current contents of the pattern buffer
   • Since the pattern buffer contained (after exchange) what the hold buffer carried, it is often an empty line or the content of a previous line.
   • In this context, as the hold buffer was most likely ‘free’ or empty, p will print a blank line (or just a line break).

3. x

   • we swap again pattern ↔ hold
   • after that :
     • pattern buffer = "4. Oscar et la dame rose | Éric-Emmanuel Schmitt | Format: Poche | Pages: 112"
     • hold buffer = what the pattern buffer had before the p (often empty)

Then sed continues, eventually the line containing "Éric-Emmanuel Schmitt" is printed (either by sed's default behaviour or by the next instruction in the script), so we will have:

 ....
 <blank line inserted>
 4. Oscar et la dame rose | Éric-Emmanuel Schmitt | Format: Poche | Pages: 112
 ...

Thus, a blank line is inserted before the line containing "Éric-Emmanuel Schmitt".

This behavior is repeated for all lines where /Éric-Emmanuel Schmitt/ matches.

Simplified visual diagram

Suppose that the hold buffer begins empty, and consider the passage on line 4:

• x → Hold = "4. Oscar et …", Pattern = ""
• p → prints the (new) pattern, which is empty → we get an empty line in the stdout
• x → Hold = "", Pattern = "4. Oscar et …"
• Then the line "4. Oscar et …" is printed normally.

Note :

If "G" is used directly on the matching /Éric-Emmanuel Schmitt/ lines, this inserts a blank line after these lines.

5.3 Copy the contents of the pattern space into the hold buffer: h

The following example will work the same as the tac command which prints the input file in reverse order with the last line first.

sed -n '1!G;h;$p' books.log

The command '1!G;h;$p' contains three parts :

1. 1!G : executes G on all lines except the first (1! means ‘all except line 1’). if it is not the first → append the contents of the hold space to the pattern space
2. h: executes h on each line. Copies the pattern space into the hold space
3. $p: executes p only on the last line . if it is the last one → print the pattern

Detailed steps :

5.4 Append the contents of the pattern space to the hold buffer : H

The following example will append lines containing "Amélie" or "Agnès" to the hold space and then print the accumulated content.

sed -n '/Amélie/H; /Agnès/H; ${g;p}' books.log

• -n: Suppresses automatic printing of pattern space.
• /Amélie/H: If a line contains "Amélie", append it to the hold space.
• /Agnès/H: If a line contains "Agnès", append it to the hold space.
• ${g;p}: On the last line ($), get the contents of the hold space into the pattern space (g), and then print the pattern space (p).

5.5 Replace the pattern space with the contents of the hold buffer : g

We'll print the line immediately before a pattern match, in this case "Camus" :

sed -n '/Camus/{g;1!p};h' books.log

The above command contains 2 mains parts :

1. Block {g;1!p} is executed once the current line contains /Camus/ .

• g : Copy the hold buffer to the pattern buffer. This places the contents of the “previous line” (if it has been stored) into the current line.
• 1!p : “if it is not the first line (1!) then p (print)”. We do not want to print anything if the line in question is the first line, because there is no line preceding the first line. So this block prints the contents of the hold buffer (the previous line) only when we are on a line that contains Camus, and that line is not line 1.

2. h

After the block, for each line (whether it matches or not), the h command is executed. h copies the current line (the pattern buffer for that line) to the hold buffer. Thus, at the end of processing a line, the hold buffer contains the current line, making it the “previous” line for the next iteration.

In summary:

Each line is stored in the hold buffer via h.

When a line containing “Camus” is encountered, the current line (pattern space) is replaced with the contents of the hold buffer (previous line) via g, then this content is printed (if it is not the first line) via 1!p.

The line containing “Camus” itself is not printed by this block (it could be printed by default if -n is not used).

The overall effect is to locate “Camus” and move up one line to print it.

6. The print command p

Displaying the contents of a file is one of the fundamental tasks we carry out. The print command prints the contents of the pattern buffer, given that all input lines are printed by default.

The p command is often paired with the -n option, which stops default printing, allowing us to display only the filtered lines.

6.1 Selective printing

6.1.1 default printing behavior

In case the -n option is not used the print command will cause redondant line, each line displayed twice : one from the default behavior of sed which prints the pattern buffer and the other due to p.

sed 'p' books.log

To display the file normally, similar to cat command, the right syntax should be :

sed -n  'p' books.log
or 
sed '' books.log

6.1.2 Print a specific line

Print the second line only :

sed -n  '2p' books.log

6.1.3 Print a range of lines

Print the 19th century books only, from the first to third line :

sed -n  '1, 3 p' books.log

Print from the fourth line to last line :

sed -n  '4, $ p' books.log

6.1.4 Print a range of lines using operator +

Print 6 lines more starting from line 4 (included) :

sed -n  '4, +6 p' books.log

6.1.5 Print a range of lines using operator ~

Beginning at line 1, sed processes every 2 lines; in this case, we print only the odd-numbered lines:

sed -n '1~2 p' books.log

To print the even lines only :

sed -n '2~2 p' books.log

6.1.6 Print using matching pattern

same as: grep 'Camus' books.log :

sed -n '/Camus/p' books.log

find the first occurrence of the pattern "120" and print the remaining lines :

sed -n '/140/,$ p' students.txt

print all the lines between the patterns 4 and 10 :

sed -n '/^4./, /^10./ p' books.log

4. Oscar et la dame rose | Éric-Emmanuel Schmitt | Format: Poche | Pages: 112
5. Monsieur Ibrahim et les fleurs du Coran | Éric-Emmanuel Schmitt | Format: Poche | Pages: 96
6. L'Étranger | Albert Camus | Format: Poche | Pages: 186
7. Stupeur et tremblements | Amélie Nothomb | Format: Poche | Pages: 160
8. Vernon Subutex 1 | Virginie Despentes | Format: Poche | Pages: 352
9. Couleur du temps | Leïla Slimani | Format: Poche | Pages: 224
10. La vie est facile, ne t'inquiète pas | Agnès Martin-Lugand | Format: Poche | Pages: 288

Matching pattern and + operator :

sed -n '/^2./, +8 p' books.log

6.2 p as a flag for the substitute command s

p can be a flag for the substitute command which in this case only prints the processed line if the substitution command succeed.

sed -n ' s/book/livre/p' books.log

Perform multiple substitutions :

sed -n -e '2,8 s/success/achieved/p' -e '1,8 s/grade/mark/p' students.txt

ID name forename mark result
120 weslati firas 85 achieved
140 dridi mohamed 80 achieved
145 yacoubi salah 95 achieved
147 benrejeb wissal 90 achieved
148 mekki ryan 75 achieved
152 mbanebe philippe 80 achieved

filter + substitute

sed -n '/^14[0-9]/ s/success/PASS/gp' students.txt

140 dridi mohamed 80 PASS
145 yacoubi salah 95 PASS
147 benrejeb wissal 90 PASS
148 mekki ryan 75 PASS

The other way to delete command :

-n option paired with !p flag is equivalent to delete command :

sed -n '/^14[0-9]/!p' students.txt

ID name forename grade result
120 weslati firas 85 success
152 mbanebe philippe 80 success
153 sako aminata 70 fail