Redirection and Pipes

There are lots of strange and interesting ways to connect utilities together. Most of these you have probably already seen.

The standard redirect to file; 


ls > /tmp/listing
and piping output from one command to another 

ls | wc -l
But bourne-shell derivatives give you even more power than that.

Most properly written programs output in one of two ways.

  1. Progress messages go to stdout, error messages go to stderr
  2. Data goes to stdout, error AND progress messsages go to stderr
If you know which of the categories your utilities fall into, you can do interesting things.

Redirection

An uncommon program to use for this example is the "fuser" program under solaris. it gives you a long listing of what processes are using a particular file. For example: 


$ fuser /bin/sh
/bin/sh:    13067tm   21262tm
If you wanted to see just the processes using that file, you might initially groan and wonder how best to parse it with awk or something. However, fuser actually splits up the data for you already. It puts the stuff you may not care about on stderr, and the meaty 'data' on stdout. So if you throw away stderr, with the '2>' special redirect, you get 

$ fuser /bin/sh  2>/dev/null
    13067   21262
which is then trivially usable.

Unfortunately, not all programs are that straightforward :-) However, it is good to be aware of these things, and also of status returns. The 'grep' command actually returns a status based on whether it found a line. The status of the last command is stored in the '$?' variable. So if all you care about is, "is 'biggles' in /etc/hosts?" you can do the following: 


grep biggles /etc/hosts >/dev/null
if [[ $? -eq 0 ]] ; then
	echo YES
else
	echo NO
fi
As usual, there are lots of other ways to accomplish this task, even using the same 'grep' command. However, this method has the advantage that it does not waste OS cycles with a temp file, nor does it waste memory with a potentially very long variable.
(If you were looking for something that could potentially match hundreds of lines, then var=`grep something /file/name` could get very long)

Inline redirection

You have seen redirection TO a file. But you can also redirect input, from a file. For programs that can take data in stdin, this is useful. The 'wc' can take a filename as an argument, or use stdin. So all the following are roughly equivalent in result, although internally, different things happen: 
wc -l /etc/hosts
wc -l < /etc/hosts
cat /etc/hosts | wc -l

Additionally, if there are a some fixed lines you want to use, and you do not want to bother making a temporary file, you can pretend part of your script is a separate file!. This is done with the special '<<' redirect operator. 

command << EOF
means, "run 'command', but make its stdin come from this file right here, until you see the string 'EOF'"

EOF is the traditional string. But you can actually use any unique string you want. Additionally, you can use variable expansion in this section! 

DATE=`date`
HOST=`uname -n`
mailx -s 'long warning' root << EOF
Something went horribly wrong with system $HOST
at $DATE
EOF
if you do NOT want to use variable expansion, then use "EOF" rather than EOF.

Pipes

In case you missed it before, pipes take the output of one command, and put it on the input of another command. You can actually string these together, as seen here; 
grep hostspec /etc/hosts| awk '{print $1}' | fgrep '^10.1.' | wc -l
This is a fairly easy way to find what entries in /etc/hosts both match a particular pattern in their name, AND have a particular IP address ranage.

The "disadvantage" to this, is that it is very wasteful. Whenever you use more than one pipe at a time, you should wonder if there is a better way to do it. And indeed for this case, there most certainly IS a better way: 


grep '^10\.1\..*hostspec' /etc/hosts | wc -l
There is actually a way to do this with a single awk command. But this is not a lesson on how to use AWK!

Combining pipes and redirection

An interesting example of pipes with stdin/err and redirection is the "tar" command. If you use "tar cvf file.tar dirname", it will create a tar file, and print out all the names of the files in dirname it is putting in the tarfile. It is also possible to take the same 'tar' data, and dump it to stdout. This is useful if you want to compress at the same time you are archiving: 
tar cf - dirname | compress > file.tar.Z
But it is important to note that pipes by default only take the data on stdout! So it is possible to get an interactive view of the process, by using 
tar cvf - dirname | compress > file.tar.Z
stdout has been redirected to the pipe, but stderr is still being displayed to your terminal, so you will get a file-by-file progress report. Or of course, you could redirect it somewhere else, with 
tar cvf - dirname 2>/tmp/tarfile.list | compress > file.tar.Z

Indirect redirection (Inline files)

Additionally, there is a special type of pipes+redirection. This only works on systems with /dev/fd/X support. You can automatically generate a "fake" file as the result of a command that does not normally generate a file. The name of the fake files will be /dev/fd/{somenumberhere}

Here's an example that doesnt do anything useful 

wc -l <(echo one line) <(echo another line)
wc will report that it saw two files, "/dev/fd/4", and "/dev/fd/5", and each "file" had 1 line each. From its own perspective, wc was called simply as 
wc -l /dev/fd/4 /dev/fd/5

There are two useful components to this:

  1. You can handle MULTIPLE commands' output at once
  2. It's a quick-n-dirty way to create a pipeline out of a command that "requires" a filename (as long as it only processes its input in a single continuous stream).

TOP of tutorial
Next: Interesting stuff Prev: Built-in functions
This material is copyrighted by Philip Brown