UNIX Bourne Shell Programming

Please contact or Mt Xia for assistance with all your shell programming needs.
Go to the bottom, first,previous,next, last section, table of contents.


6.5  sed: Edit a File to Standard Output

UNIX provides a method of editing streams of data. It is the sed
utility.  The name of this utility is derived from Stream EDitor.
This is not the same as the vi editor. The vi editor edits text in
a file. The sed utility edits text in a stream. In order to edit
a character stream two things are required.  First, the line to
edit must be identified (regular expressions) and second, how to
edit the line.

The formal form for the sed utility is as follows:


                                                                
    Command Format: sed [-n] [-e script] [-f sfile] [files]    
                                                               
    Details in on-line man pages                               
                                                               


The sed utility copies the named files (standard input default) to
the standard output, edited according to a set (script) of
commands. The -f options cause the script to be taken from file
"sfile".

The general form is:

     $sed /address/instruction

Note:     If no address is specified, all lines are chosen to edit.


'sed' addresses can be line numbers or regular expressions.

Example:
 
     line numbers        2,4
                         2,$  ($ represents the last line)

     textual address     /regular-expression/



Note:     Forward slashes enclose textual addresses

The sed instructions indicate what editing function to perform.
Here some useful sed instructions:

     s         substitute

     d         delete

Note: Most sed command lines contain spaces or metacharacters and
they should be quoted to protect them from the shell. There are
many more editing commands provided by sed. Here is a sample sed
command to edit the records in the database file that we are
already familiar with, namely, phone.lis:

Sample session:


    $sed /s/Smith/Smythe/ phone.lis                            
    Smythe, Joan        7-7989                                 
    Adams, Fran         2-3876                                 
    StClair, Fred       4-6122                                 
    Jones, Ted          1-3745                                 
    Stair, Rich         5-5972                                 
    Benson, Sam         4-5587                                 
    $                                                          

                                                                 
sed is an editor . It simply copies the standard input to the
standard output, editing the lines that match the indicated
address. The original file is not changed. 

Here's another example of a sed command.

Sample session:


    $sed '2,4 s/2$/3/' phone.lis                               
    Smith, Joan         7-7989                                 
    Adams, Fran         2-3876                                 
    StClair, Fred       4-6123                                 
    Jones, Ted          1-3745                                 
    Stair, Rich         5-5972                                 
    Benson, Sam         4-5587                                 
    $                                                          


What does this sed command do?  If you read command in English it
reads like this: On lines 2 through 4 substitute the 2 at the end
of the line with a 3.  Notice that the phone number for
StClair,Fred changed from 4-6122 to 4-6123.  The number for
Stair,Rich didn't change because it was outside the range.

The sed utility can also be use to delete parts of a line of data.
This is done by substituting nothing for the parts you want to
delete. It looks like this:

Sample session:


    $sed 's/^.*, //' phone.lis                                 
    Joan               7-7989                                  
    Fran               2-3876                                  
    Fred               4-6122                                  
    Ted                1-3745                                  
    Rich               5-5972                                  
    Sam                4-5587                                  
    $                                                          


Reading this command it means:                                   

Substitute from the beginning of the line followed by any number
of characters followed by a comma with the null string (nothing).
This has the effect of removing the text.               

Here's a delete command and how it's used.

Sample session:


    $sed d phone.lis                                           
    $                                                          


Why is there no output? Well, it read standard input and did the
editing function on all the selected lines. Since no lines were
specified all lines were selected to be edited. The editing was to
delete the line.

 Question: Has the original file been destroyed?

Multiple commands are allowed in sed. Each instruction is applied
to each input line.

Sample session:


    $sed '/Stair/d                                             
    >/Adams/d' phone.lis                                       
    Smith, Joan         7-7989                                 
    StClair, Fred       4-6122                                 
    Jones, Ted          2-1136                                 
    Benson, Sam         4-5587                                 
    $                                                          


The records for Adams and Stair have both been removed from the
database.

Note:     The > character is the BourneShell secondary prompt.

6.6  awk: A Pattern Matching Programming Language

Suppose you wanted to change the format of the database phone.lis
to be the first name followed by the last name. There is no easy
way to do this with sed. Fortunately, UNIX not only provides a
stream editor (sed) but it also has a formatting tool. The
formatting tool in UNIX is called awk. This tool is named after
authors who wrote it  Alfred V. Aho, Peter J. Weinberger, and Brian
W. Kerninghan so it really doesn't have any meaning.

The awk utility is a pattern scanning and processing language. It
will search one or more files for a specified pattern and then
performs an action, such as writing to standard output or
incrementing a counter when it finds a match. You can use awk to
generate reports or filter text. It works equally well with numbers
or text. The authors designed it to be easy to use and sacrificed
execution speed toward this end.

While the sed utility allows us to change the text in a stream,
awk allows us to rearrange, add, or delete text in a stream,
easily.

The awk takes advantage of many constructs from the C programming
language. It has the following features:


     flexible format
     conditional execution
     looping statements
     numeric variables
     string variables
     regular expressions
     C's printf


The awk will take it's input from the files you specify on the
command line or from standard input. The following is the format
for awk:


                                                                
    Command format: awk [-Fc] [prog] [files]                   
                                                               


The awk will scan each line of file for lines that match a set of
patterns specified by prog. With each pattern in prog there can be
an associated action to be performed when the line is found. The
set of patterns may appear literally as prog, or in a file
specified as -f file. The prog string should be enclosed in single
quotes to protect it from the Shell.

Files are read in order and if there are none specified the
standard input is read. Each line is matched against the pattern
portion of every pattern-action statement.  The associated action
is performed for each matched pattern.  An input line is made up
fields separated by white space.  $1, $2.. define the fields. $0
refers to the whole line.

A pattern-action statement has the form:


     pattern {action}


A missing action means print the line; a missing pattern always
makes a match. a statement can be one of the following:


     if (conditional) statement [else statement]
     while (conditional) statement
     for (expression;conditional;expression) statement
     break
     continue
     {[statement]...}
     variable=expression
     print [expression-list] [>expression]
     printf format [,expression-list][>expression]
     next # skip remaining pattern on this input line
     exit # skip the rest of the input


Statements are terminated by semi-colons, new-lines (CR), or right
braces.

Let's look at the syntax for awk in a little simpler manner.


     awk 'commands' [filename]


An awk program (commands) consists of a optional pattern to match
and an action to perform if a match is found on the current line.
This syntax looks like this:


     awk '/pattern/{action}' [filename]
The pattern used is a regular expression enclosed in forward
slashes. If no pattern is listed, the action will be performed for
every line. An action can contain several commands.  There can be
multiple patterns and actions.

     awk '/pattern1/{action1}
      /pattern2/{action2}' [filename}


One of awks commands is print. It puts the current line on standard
output.

Sample session:


                                                              
   $awk '{print}' phone.lis                                    
   Smith, Joan         7-7989                                  
   Adams, Fran         2-3876                                  
   StClair, Fred       4-6122                                  
   Jones, Ted          1-3745                                  
   Stair, Rich         5-5972                                  
   Benson, Sam         4-5587                                  
   $                                                           


The awk splits every input line at whitespace and keeps track of
the number of fields on each line and counts the number of lines
read. Each field is identified by its field number and a $.

     $1   Identifies the first field

     $2   Identifies the second field

     .

     $0   Identifies the entire line

     NF   Identifies the number of fields on the line

     NR   Identifies the number of lines that have been read
Sample session:


                                                               
    $awk '{print NR,$1}' phone.lis                             
    1 Smith,                                                   
    2 Adams,                                                   
    3 StClair,                                                 
    4 Jones,                                                   
    5 Stair,                                                   
    6 Benson,                                                  
    $                                                          


To change the order of the names in phone.lis use awk. The comma
in the print command tells awk to separate each field with a space.
Without the comma, the output would have no spacing.

Sample session:

    $awk '{print $2, $1 ""$3}' phone.lis                  
    Joan Smith,        7-7989                                  
    Fran Adams,        2-3876                                  
    Fred StClair,      4-6122                                  
    Ted Jones,         1-3745                                  
    Rich Stair,        5-5972                                  
    Sam Benson,        4-5587                                  
    $
Go to the top, first,previous,next, last section, table of contents.