sed & awk

sed & awkSearch this book
Previous: 8.4 ArraysChapter 8
Conditionals, Loops, and Arrays
Next: 8.6 System Variables That Are Arrays
 

8.5 An Acronym Processor

Now let's look at a program that scans a file for acronyms. Each acronym is replaced with a full text description, and the acronym in parentheses. If a line refers to "BASIC," we'd like to replace it with the description "Beginner's All-Purpose Symbolic Instruction Code" and put the acronym in parentheses afterwards. (This is probably not a useful program in and of itself, but the techniques used in the program are general and have many such uses.)

We can design this program for use as a filter that prints all lines, regardless of whether a change has been made. We'll call it awkro.

awk '# awkro - expand acronyms 
# load acronyms file into array "acro"
FILENAME == "acronyms" {
	split($0, entry, "\t")
	acro[entry[1]] = entry[2]
	next
} 

# process any input line containing caps 
/[A-Z][A-Z]+/ {
	# see if any field is an acronym
	for (i = 1; i <= NF; i++)
		if ( $i in acro ) {
			# if it matches, add description 
			$i = acro[$i] " (" $i ")"
		}
}

{
	# print all lines
	print $0
}' acronyms  $*

Let's first see it in action. Here's a sample input file.

$ cat sample
The USGCRP is a comprehensive 
research effort that includes applied 
as well as basic research.
The NASA program Mission to Planet Earth 
represents the principal space-based component
of the USGCRP and includes new initiatives
such as EOS and Earthprobes.

And here is the file acronyms:

$ cat acronyms
USGCRP	U.S. Global Change Research Program
NASA	National Aeronautic and Space Administration
EOS	Earth Observing System

Now we run the program on the sample file.

$ awkro sample
The U.S. Global Change Research Program (USGCRP) is a comprehensive
research effort that includes applied
as well as basic research.
The National Aeronautic and Space Administration (NASA) program
Mission to Planet Earth
represents the principal space-based component
of the U.S. Global Change Research Program (USGCRP) and includes new
initiatives
such as Earth Observing System (EOS) and Earthprobes.

We'll look at this program in two parts. The first part reads records from the acronyms file.

# load acronyms file into array "acro"
FILENAME == "acronyms" {
	split($0, entry, "\t")
	acro[entry[1]] = entry[2]
	next
}

The two fields from these records are loaded into an array using the first field as the subscript and assigning the second field to an element of the array. In other words, the acronym itself is the index to its description.

Note that we did not change the field separator, but instead used the split() function to create the array entry. This array is then used in creating an array named acro.

Here is the second half of the program:

# process any input line containing caps 
/[A-Z][A-Z]+/ {
	# see if any field is an acronym
	for (i = 1; i <= NF; i++)
		if ( $i in acro ) {
			# if it matches, add description 
			$i = acro[$i] " (" $i ")"
		}
}

{
	# print all lines
	print $0
}

Only lines that contain more than one consecutive capital letter are processed by the first of the two actions shown here. This action loops through each field of the record. At the heart of this section is the conditional statement that tests if the current field ($i) is a subscript of the array (acro). If the field is a subscript, we replace the original value of the field with the array element and the original value in parentheses. (Fields can be assigned new values, just like regular variables.) Note that the insertion of the description of the acronym results in lines that may be too long. See the next chapter for a discussion of the length() function, which can be used to determine the length of a string so you can divide it up if it is too long.

Now we're going to change the program so it makes a replacement only the first time an acronym appears. After we've found it, we don't want to search for that acronym any more. This is easy to do; we simply delete that acronym from the array.

if ( $i in acro ) {
	# if it matches, add description 
	$i = acro[$i] " (" $i ")"
	# only expand the acronym once
	delete acro[acronym]
}

There are other changes that would be good to make. In running the awkro program, we soon discovered that it failed to match the acronym if it was followed by a punctuation mark. Our initial solution was not to handle it in awk at all. Instead, we used two sed scripts, one before processing:

sed 's/\([^.,;:!][^.,;:!]*\)\([.,;:!]\)/\1 @@@\2/g'

and one after:

sed 's/ @@@\([.,;:!]\)/\1/g'

A sed script, run prior to invoking awk, could simply insert a space before any punctuation mark, causing it to be interpreted as a separate field. A string of garbage characters (@@@) was also added so we'd be able to easily identify and restore the punctuation mark. (The complicated expression used in the first sed command makes sure that we catch the case of more than one punctuation mark on a line.)

This kind of solution, using another tool in the UNIX toolbox, demonstrates that not everything needs to be done as an awk procedure. Awk is all the more valuable because it is situated in the UNIX environment.

However, with POSIX awk, we can implement a different solution, one that uses a regular expression to match the acronym. Such a solution can be implemented with the match() and sub() functions described in the next chapter.

8.5.1 Multidimensional Arrays

Awk supports linear arrays in which the index to each element of the array is a single subscript. If you imagine a linear array as a row of numbers, a two-dimensional array represents rows and columns of numbers. You might refer to the element in the second column of the third row as "array[3, 2]." Two- and three-dimensional arrays are examples of multidimensional arrays. Awk does not support multidimensional arrays but instead offers a syntax for subscripts that simulate a reference to a multidimensional array. For instance, you could write the following expression:

file_array[NR, i] = $i

where each field of an input record is indexed by its record number and field number. Thus, the following reference:

file_array[2, 4]

would produce the value of the fourth field of the second record.

This syntax does not create a multidimensional array. It is converted into a string that uniquely identifies the element in a linear array. The components of a multidimensional subscript are interpreted as individual strings ("2" and "4," for instance) and concatenated together separated by the value of the system variable SUBSEP. The subscript-component separator is defined as "\034" by default, an unprintable character rarely found in ASCII text. Thus, awk maintains a one-dimensional array and the subscript for our previous example would actually be "2\0344" (the concatenation of "2," the value of SUBSEP, and "4"). The main consequence of this simulation of multidimensional arrays is that the larger the array, the slower it is to access individual elements. However, you should time this, using your own application, with different awk implementations (see Chapter 11, A Flock of awks).

Here is a sample awk script named bitmap.awk that shows how to load and output the elements of a multidimensional array. This array represents a two-dimensional bitmap that is 12 characters in width and height.

BEGIN { FS = ","   # comma-separated fields
	# assign width and height of bitmap
	WIDTH = 12
	HEIGHT = 12
	# loop to load entire array with "O"
	for (i = 1; i <= WIDTH; ++i)
		for (j = 1; j <= HEIGHT; ++j)
			bitmap[i, j] = "O"
}
# read input of the form x,y. 
{
	# assign "X" to that element of array 
	bitmap[$1, $2] = "X"
}
# at end output multidimensional array
END {
	for (i = 1; i <= WIDTH; ++i){
		for (j = 1; j <= HEIGHT; ++j)
			printf("%s", bitmap[i, j] )
		# after each row, print newline
		printf("\n")	
	}
}

Before any input is read, the bitmap array is loaded with O's. This array has 144 elements. The input to this program is a series of coordinates, one per line.

$ cat bitmap.test
1,1
2,2
3,3
4,4
5,5
6,6
7,7
8,8
9,9
10,10
11,11
12,12
1,12
2,11
3,10
4,9
5,8
6,7
7,6
8,5
9,4
10,3
11,2
12,1

For each coordinate, the program will put an "X" in place of an "O" as that element of the array. At the end of the script, the same kind of loop that loaded the array, now outputs it. The following example reads the input from the file bitmap.test.

$ awk -f bitmap.awk bitmap.test
XOOOOOOOOOOX
OXOOOOOOOOXO
OOXOOOOOOXOO
OOOXOOOOXOOO
OOOOXOOXOOOO
OOOOOXXOOOOO
OOOOOXXOOOOO
OOOOXOOXOOOO
OOOXOOOOXOOO
OOXOOOOOOXOO
OXOOOOOOOOXO
XOOOOOOOOOOX

The multidimensional array syntax is also supported in testing for array membership. The subscripts must be placed inside parentheses.

if ((i, j) in array)

This tests whether the subscript i,j (actually, i SUBSEP j) exists in the specified array.

Looping over a multidimensional array is the same as with one-dimensional arrays.

for (item in array)

You must use the split() function to access individual subscript components. Thus:

split(item, subscr, SUBSEP)

creates the array subscr from the subscript item.

Note that we needed to use the loop-within-a-loop to output the two-dimensional bitmap array in the previous example because we needed to maintain rows and columns.


Previous: 8.4 Arrayssed & awkNext: 8.6 System Variables That Are Arrays
8.4 ArraysBook Index8.6 System Variables That Are Arrays

The UNIX CD Bookshelf NavigationThe UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System