Getting into awkward situations

Intro

Processes

/etc/passwd

Performance

CSV Files

Conclusion

Intro

Awk is mostly a text processing language. It is lighter than Perl, and very often will do exactly what you need for day-to-day system administration tasks. I'll break down a few of these cases for you now...

Killing off a lot of processes in a hurry

If we run ps -ef, we see output that looks like this:

$ ps -ef

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Sep06 ?        00:00:22 init [2] 
root         2     1  0 Sep06 ?        00:01:34 [kswapd]
root         3     1  0 Sep06 ?        00:00:00 [kreclaimd]
root         4     1  0 Sep06 ?        00:02:16 [kflushd]
root         5     1  0 Sep06 ?        00:00:33 [kupdate]
root       364     1  0 Sep06 ?        00:02:13 /sbin/syslogd
root       372     1  0 Sep06 ?        00:00:00 /sbin/klogd
root       376     1  0 Sep06 ?        00:04:29 /usr/sbin/named
root       387     1  0 Sep06 ?        00:00:01 /usr/sbin/inetd
root       390     1  0 Sep06 ?        00:00:00 /usr/sbin/ippl
root       393   390  0 Sep06 ?        00:00:11 /usr/sbin/ippl
root       402     1  0 Sep06 ?        00:00:00 /bin/sh /usr/bin/safe_mysqld
mysql      437   402  0 Sep06 ?        00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking
mysql      442   437  0 Sep06 ?        00:00:12 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking
mysql      443   442  0 Sep06 ?        00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking
mysql      446   442  0 Sep06 ?        00:00:00 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking
root       646     1  0 Sep06 ?        00:00:06 /usr/sbin/sshd
root       649     1  0 Sep06 ?        00:00:42 /usr/sbin/ntpd
root       653     1  0 Sep06 ?        00:00:16 /usr/sbin/cron
root       662     1  0 Sep06 ?        00:01:20 sendmail: MTA: accepting connections      
root       668     1  0 Sep06 tty2     00:00:00 /sbin/getty 38400 tty2
root       669     1  0 Sep06 tty3     00:00:00 /sbin/getty 38400 tty3
root       670     1  0 Sep06 tty4     00:00:00 /sbin/getty 38400 tty4
root       671     1  0 Sep06 tty5     00:00:00 /sbin/getty 38400 tty5
root       672     1  0 Sep06 tty6     00:00:00 /sbin/getty 38400 tty6
root       714     1  0 Sep06 tty1     00:00:00 /sbin/getty 38400 tty1
root      5557     1  0 Sep06 ?        00:00:49 /usr/sbin/apache
gabby     8410   387  0 Sep06 ?        00:00:36 imapd
root     10741   646  0 Sep07 ?        00:00:00 /usr/sbin/sshd
dhansen  10743 10741  0 Sep07 ?        00:00:00 /usr/sbin/sshd
...

You will notice that the output is layer out in different columns (or "fields").

By using a pipe, we can ask awk to print for us only the specified fields:

$ ps -ef | awk '{print $1, $8}'

UID CMD
root init
root [kswapd]
root [kreclaimd]
root [kflushd]
root [kupdate]
root /sbin/syslogd
root /sbin/klogd
root /usr/sbin/named
root /usr/sbin/inetd
root /usr/sbin/ippl
root /usr/sbin/ippl
root /bin/sh
mysql /usr/sbin/mysqld
mysql /usr/sbin/mysqld
mysql /usr/sbin/mysqld
mysql /usr/sbin/mysqld
root /usr/sbin/sshd
root /usr/sbin/ntpd
root /usr/sbin/cron
root sendmail:
root /sbin/getty
root /sbin/getty
root /sbin/getty
root /sbin/getty
root /sbin/getty
root /sbin/getty
root /usr/sbin/apache
gabby imapd
root /usr/sbin/sshd
dhansen /usr/sbin/sshd

In this case, we asked awk to print the first ($1) and eighth ($8) fields. By default, awk will split fields whenever it encounters white-space.

How does this help us kill-off a lot of processes at once? Simply. Notice that in the first example, that field 2 ($2) contains the PID of the running programs.

Suppose that we wanted to kill-off all imapd instances. We might start by doing this:

$ ps -ef | grep imapd

gabby     8410   387  0 Sep06 ?        00:00:36 imapd
smiss    27005   387  0 Sep11 ?        00:00:08 imapd
knox     31839   387  0 07:37 ?        00:00:01 imapd
knox     31968   387  0 07:42 ?        00:00:01 imapd
jbusby    1182   387  1 11:27 ?        00:00:03 imapd
steelmi1  1214  1117  0 11:31 pts/0    00:00:00 grep imapd

Ok, good. $2 contains the PID for all of the instances we want to kill. So, we refine it a step:

$ ps -ef | grep imapd | awk '{print $2}'

8410
27005
31839
31968
1182
1223

Excellent. We now have a list of PIDs by themselves.

Now, kill, like most UNIX commands will accept input parameters in the form of a list, meaning for instance that we could do this:

kill -9 8410 27005 31839 ...

However, we don't want to risk typing the wrong PID and risk killing some important process. So, using back-ticks, we tell the shell to substitute the output of our awk statement into the input of kill:

$ sudo kill -9 `ps -ef | grep imapd | awk '{print $2}'`

bash: kill: (1223) - No such process

$

You will probably get back something like the above line telling you "No such process." This is because when we do our ps -ef | grep imapd, your grep shows up in the list too. It is annoying, but we can fix it like such:

kill -9 `ps -ef | grep imapd | grep -v grep | awk '{print $2}'`

We simply add an extra grep to filter out occurrences of the string 'grep' in the output of the ps.

You can also kill-off processes based on any criteria that you can uniquely grep for. For instance, if you wanted to kill all processes owned by a specific user, or all processes that have the string "Aug" in the fourth column, etc...

Reporting on /etc/password and/or /etc/shadow

Until this point, we have used awk with the default field separator (any white space). But, what if we need to process a stream that is split into fields, but uses something different to separate them? Awk is up to the task. We simply use the -F parameter to specify it.

For example, /etc/password is split into well known fields, separated by ":"

The output of

$ man 5 passwd 

tells us:

NAME
       passwd - The password file

DESCRIPTION
       passwd contains various pieces of information for each user account.  Included is

            Login name

            Optional encrypted password

            Numerical user ID

            Numerical group ID

            User name or comment field

            User home directory

            User command interpreter

So, $1 will be the login name, $2 the optional encrypted password, and so-on.

Let's say we'd like to see just the login name and comment field for all users on the system. We'd need $1 and $5:

$ awk -F: '{print $1, $5}' /etc/passwd

root root
daemon daemon
bin bin
sys sys
sync sync
games games
man man
lp lp
mail mail
news news
uucp uucp
proxy proxy
mailman Mailman
majordom Majordomo
postgres postgres
www-data www-data
...

Hmm, we notice here that we get a lot of built-in system accounts that we really didn't care about... However, if we look in /etc/passwd, we notice that our actual users have UIDs (in $3) that are numbered greater than 999. No problem, awk is a grown-up programming language, and supports conditionals. I'll break it into lines to make it more readable... Just hit enter at the end of each line:

$ awk -F: '{
if ($3 > 999)
	print $1, $5
}' /etc/passwd

nobody nobody
steelmi1 Michael Steele
jbusby Johnny E Busby II
smiss Stacy Busby
bwood William Wood
mcmillan Barry McMillan
rroth Ryan B. Roth
jmurrah Josh Murrah
paul Paul Watson
knox The Knoxster
dhansen David Hansen
mcoker Bonehead,,,
amerus Nicolai Pavlov
jjardina Jason Jardina
davidm David McMillan
dspisak Dana Spisak
jberg Jonathan Bergmann
webguru SALUG WWW Guru

...

Better. But it might be nice to have the list sorted by login name instead of by UID:

$ awk -F: '{ 
if ($3 > 999)
    print $1, $5
}' /etc/passwd | sort

aearon Chris Shiver
amerus Nicolai Pavlov
bbrenner 
billman Billy Beaty
bishop Rod Steiner
bmiddlet Brent Middleton
bobbyrd 
bwood William Wood
byost Barry Yost
cbgamb Charles Gambrell
cfleming Chris Fleming
chioke Chioke Jaffree
clegros Clint LeGros

...

Nice.

Now, let's put that comment field into "" marks, and add a "-->" between the fields in the output, so it is easier to read:

$ awk -F: '{ 
if ($3 > 999)
    print $1, "--> \"" $5 "\""
}' /etc/passwd | sort

aearon --> "Chris Shiver"
amerus --> "Nicolai Pavlov"
bbrenner --> ""
billman --> "Billy Beaty"
bishop --> "Rod Steiner"
bmiddlet --> "Brent Middleton"
bobbyrd --> ""
bwood --> "William Wood"
byost --> "Barry Yost"
cbgamb --> "Charles Gambrell"
cfleming --> "Chris Fleming"
chioke --> "Chioke Jaffree"
clegros --> "Clint LeGros"
coil --> "COIL Quake Clan"
davidm --> "David McMillan"
dbaxter --> "Dave Baxter"
dhandjr --> ""
dhansen --> "David Hansen"

As you can see, the print statement becomes a little more complex, as we have to use a \ to escape the " marks that we want printed to the output stream. And you'll also notice that the string-literals you want to output have to be enclosed in quotes.

What if we needed to find out in a hurry how many non-system accounts are on our system?

Easy:

$ awk -F: '{
if ($3 > 999)
print $1, $5
}' /etc/passwd | wc -l

     66

Also, there is no rule that says you have to print the fields in order. You can just as easily '{print $5, $1}' as you can the other way around.

If you look at the man page for shadow(5), you will see that /etc/shadow has well-defined fields too. In this case though, $5 is the length of time before the user has to change his password. Let's say you want to find out the user names that do not have to change their passwords for more than 90 days. You won't be able to do this on a machine unless you are root:

# awk -F: '{
if ($5 > 90)
	print $1
}' /etc/shadow | sort

Performance Stats

We use the sar program at work to get a general idea of system performance metrics. In Linux, sar is part of the sysstat package, but this example was done on Solaris 9. Your mileage may vary.

Running sar without any parameters produces CPU utilization output in the following columns:

Time %usr %sys %wio %idle

Here is a sample:

$ sar

SunOS spiderman 5.9 Generic_118558-09 sun4u    09/12/2005

00:00:00    %usr    %sys    %wio   %idle
00:01:00       0      11       1      88
00:02:01       1      10       0      89
00:03:00       0      10       0      89
00:04:00       0      10       0      90
00:05:00       1      10       0      89
00:06:00       0      10       0      90
00:07:01       1      10       0      90
00:08:00       0      10       0      90

...

Average        1       1       0      98

The %idle column == 100% - (%usr + %sys + %wio). So if %idle == 15 for one sample, then the system was %85 busy, and if %idle == 88, then we know that the system was %12 busy. It might be nice to turn that around, so we could graph just the sample time and %busy -- which isn't a column in our output -- without having to do arithmetic in our heads.

Before we fire up awk, we think ahead just a little bit, and notice that we have some header/footer information that is going to junk-up our output. With some planning, we can grep our way out of trouble. We want to ignore:

1. Lines that start with the string SunOS
2. Lines that have the column headers in them
3. Blank lines
4. Lines that start with the string Average

Here is our command:

$ sar | egrep -v '^SunOS|usr|^$|^Average' | awk '{ print $1, (100 - $NF) }'

00:01:00 12
00:02:01 11
00:03:00 11
00:04:00 10
00:05:00 11
00:06:00 10
00:07:01 10
00:08:00 10

You'll notice several new bits of syntax here.

First, we've switched from grep to egrep. This gives us more flexibility with regular expressions than the normal version of grep on some systems.

Next, we're using the -v option to tell grep to reverse its search. Instead of showing us the lines that contain the patterns listed, show us the lines that DON'T contain the patterns listed.

Then, we have the patterns enclosed in single quotes, and separated by the "or" comparison operator. So 'pattern1|pattern2' means the line matches if it contains "pattern1" or if it contains "pattern2" etc...

The patterns individually are:

1. ^SunOS --> The line starts with (that's what the ^ symbol means) the string SunOS
2. usr --> The line contains the string 'usr' which is one of the column headers
3. ^$ --> The line starts with an end of line (that is what the $ symbol means), which means the line is blank.
4. ^Average --> The line starts with the string "Average".

Finally, we use awk as we normally would, except that we use an arithmetic expression of 100 subtracted from the value of a field called $NF that we've never seen before. Awk keeps several variables hanging around and automatically defined to help make your life easier. One of these variables is NF which stands for "Number of Fields." Since awk starts numbering fields at 1, this means that $NF just refers to "the value of the last field, whichever that is."

Since that is the case, you can also do things like '{print $(NF - 1)}' which says to print 1 field before the last field.

CSV Files

One final topic of discussion is files that contain rows (records) of Comma Separated Values. In reality, you'll probably have to deal with these quite often, usually when a user needs some data moved between UNIX and an Excel spreadsheet.

There's nothing to panic over... Awk is quite good at this, provided that your records don't have commas inside the columns. For instance this is a row:

"Field one has no commas in it.","Field two however, does.","Field three is fine again."

If we paste this text into a file called infile.txt, then tell awk to print fields 1, 2, and 3, each on their own line, this is what we get:

$ awk -F, '{print $1 "\n" $2 "\n" $3}' infile.txt

"Field one has no commas in it."
"Field two however
 does."

Eww. In reality, that happens quite a bit too.

Different versions of Awk handle this with varying degrees of success. I typically just work around it, by making " my field separator, and incrementing each column as needed to get the right result:

$ awk -F\" '{print $2 "\n" $4 "\n" $6}' infile.txt

Field one has no commas in it.
Field two however, does.
Field three is fine again. 

This method isn't fool-proof, and you may end up wasting a lot of time trying to get the right result out of it.

With GNU Awk (shipped with most Linux distributions), I was able to get away with using the two characters ", together (properly escaped, of course) as a field separator. This yielded some success:

$ awk -F\", '{print $1 "\n" $2 "\n" $3}' infile.txt

"Field one has no commas in it.
"Field two however, does.
"Field three is fine again."

Mac OS X (10.4.2) worked this same way, as expected.

But it didn't give the correct result on Solaris 9's default awk. However, there is a "legacy" version of Awk that looks like a relic from Solaris 2.4 in /usr/xpg4/bin/awk that worked fine. GNU Awk on Solaris would probably also yield the correct result. This is just another example of the subtle variations between different UNIXes (and even different iterations of the same UNIX!). Part of being a good system administrator is learning how to deal with these little problems that end up taking a long time to sort out because "That should work!" or "It works just fine on <OS>!" Being flexible often means knowing more than one way to work around a problem.

Also notice, that all lines except the last one have their ending " removed. In this case, you might clean it up a little with sed, if you don't care about losing all of the " marks:

$ awk -F\", '{print $1 "\n" $2 "\n" $3}' infile.txt | sed 's/\"//g'

Field one has no commas in it.
Field two however, does.
Field three is fine again.

Alternatively, you could get a little more complex, and use sed to put the ending quotes back, depending on the situation:

$ awk -F\", '{print $1 "\n" $2 "\n" $3}' infile.txt | sed 's/\.$/\.\"/'

"Field one has no commas in it."
"Field two however, does."
"Field three is fine again."

You might have some luck with a more complex expression, using Awk's split() function to do a better job, a Google search yielded several nasty example. Past a certain complexity I prefer to up-shift into Perl.

Conclusion

We've really only scratched the surface of Awk. It can do a whole lot more, but we're just looking to pick the low-hanging fruit. I've shown you how to solve some real-world problems quickly, without having to write tons of Perl or resorting to pasting the data into Excel for manipulation. Once you get awk into your repertories, you will be able to do some amazing things. You'll be able to get answers for your boss quicker, you'll be able to rearrange data for customers or coworkers on-the-fly.

Comments

jbusby wrote in to remind me that in our passwd example, we can specify an Output Field Separator with awk's OFS option to make things a little bit neater. So this:

$ awk -F: '{ 
if ($3 > 999)
    print $1, "--> \"" $5 "\""
}' /etc/passwd | sort

becomes this:

$ awk -F: '{
OFS=" --> "
if ($3 > 999)
    print $1,"\"" $5 "\""
}' /etc/passwd | sort

Top