Thursday, January 05, 2006

Two Cool Bash Tricks

I'll start off this blog with two super-cool bash shell tricks. You'll be the envy of your friends with these up your sleeve.

  1. The "/dev/tcp" trick

    Bash does some super-cool magic when you access /dev/tcp/hostname/port. It will create a TCP socket that is connected to the named host on the given port. This lets you easily use network sockets with regular shell IO redirection. For example, the following simply prints the time from NIST:

    cat < /dev/tcp/

    And YES, you can read AND write to these sockets. Here's how. The following example fetches the source for the homepage.

    exec 5<> /dev/tcp/
    printf "GET / HTTP/1.0\n\n" >&5
    cat <&5
    exec 5>&-

    Check out the bash(1) manpage for more details.

  2. Process Substitution

    This trick allows you to use a process *almost* anywhere you can use a file. To illustrate, let's consider the diff command. Most versions of diff require you to pass exactly two file names as arguments. But what if we want to diff something, like the contents of a directory, that doesn't necessarily exist in a file? This is where we can use process substitution. For example, to diff the contents of two directories, you could use:

    diff <(find dir1) <(find dir2)

    The syntax <(command) creates a named pipe, and attaches command's STDOUT to the pipe. So, anything that reads from the pipe will actually be reading the output of command. To prove this to yourself, try the following:

    $ echo <(/bin/true)
    $ ls -l <(/bin/true)
    lr-x------ 1 jgm eng 64 Jul 13 21:50 /dev/fd/63 -> pipe:[31340331]
    $ file <(/bin/true)
    /dev/fd/63: broken symbolic link to pipe:[31340360]

    Similarly, you can use the syntax >(command) to have the command read from the pipe. An example is:

    tar cvf >(gzip -c > dir.tar.gz) dir

    Obviously, there are better ways to accomplish taring and compressing, but the point was to use process substitution.


Naaman said...

Cool, hope to see many more posts :-)

Anonymous said...

One question for you,

Could "Process Substitution" read a ssh password ?

How could we do:

diff <(ssh host1 "find /lib/modules") <(ssh host2 "find /lib/modules")

Is it possible (answering every password with keyboard)?

Thank you !

Greg said...

Yes, you can. I mention it briefly in the post at, but it I didn't mention the SSH part of it. Basically, that trick relies on SSH being configured to allow password-less logins using something like RSA based authentication. If you just man ssh, or google it, you'll find details on setting up RSA authentication with SSH.

Anonymous said...

Thanks greg, it is ok, i know about ssh with RSA authentication and that it will work fine inside Process Substitution.

But I'd like to know if can I get the password from keyboard (I don't like RSA authentication because I want to type password when I do something with remote hosts)

I suppose it is not possible but it will be a nice bash feature for me.

Thank you again!

Greg said...

Hmm, I'm not sure if you can get the sub shells started by the process substitution to read from the keyboard. But there's potentially an alternative -- the dreaded expect. You could try wrapping the ssh command in a small expect command (or script). For example, something like this would use expect to log you in via SSH and a password:

expect -c 'spawn ssh cat /etc/passwd' -c 'expect password:' -c 'send P4$$w0Rd\n' -c 'expect eof'

Then you could wrap that whole mess in the process substitution. Something like:

diff <(expect -c 'spawn ssh cat /etc/passwd' -c 'expect password:' -c 'send P4$$w0Rd\n' -c 'expect eof') <(cat /etc/passwd)

That mess worked on my machine. *shrug*

Anonymous said...

Thank grep

You can use sshpass

it is more simple that expect

Chris said...

True unix junkies here! You can tell from the muscle memory of the anonymous poster when he typed grep instead of greg.

Cocker68 said...

The solution for Your task with one remote-host is:

ssh host1 "find /lib/modules" | diff - $(find /lib/modules)

Password is prompted for host1.
host2 has to be localhost or be password-freely reachable.

ssh host1 "find /lib/modules" | diff - $(ssh host2 "find /lib/modules")

- Cocker ;wq