Thursday, August 17, 2006

Old (but useful) Shell Tricks

I used to have a somewhat long list of somewhat interesting Unix/Shell tips-n-tricks on an old version of my website. A few folks asked me where it was, and as it's no longer available, so I figured I'd repost a handful of the tips:

  1. From within an executing script, how do I find the directory where the script lives?

    A simple pwd doesn't work because it gives the Present Working Directory, and we want to know the directory where the script actually resides on disk. I can't remember the previous solution I came up with, but here's another solution that should work:
    dir=$(dirname $(echo $0 | sed -e "s,^\([^/]\),$(pwd)/\1,"))
    echo I live in $dir

    Here how it works. $0 is the name of the script as it was executed, so this may be, ./, /tmp/, etc.. This gets sent to sed, which then uses a basic regular expression that says "If the first character of $0 was NOT a forward slash, then prepend $0 with my present working directory ($(pwd)) followed by whatever that first character was (\1), finally, take the dirname of this value, then assign it to dir". If the first character in $0 is a forward slash, then we were invoked via an absolute path and so we don't want to change anything.

  2. How do I copy a directory structure from one machine to another?

    $ tar cf - some_directory | ssh kramer "( cd /path/to/destination; tar xf - )"

    tar cf - some_directory creates a tar file of some_directory but the dash (-) tells tar to write to STDOUT instead of writing to an actual file on disk. The STDOUT from the first tar command is piped to STDIN of the next command. The right hand side of the pipe says to log into the host kramer using ssh and run the commands cd and tar xf -. The trick is with the commands "( cd /path/to/destination; tar xf - )". The parens create a subshell, in which the current directory is changed to /path/to/destination, and tar xf - reads from STDIN and extracts the tar file. This STDIN is the same STDIN that was sent to us over the pipe from STDOUT of the first tar command. Thus the directory structure on jerry gets tar'd up, transfered to kramer, then extracted all in one fell swoop.

  3. How do I diff two files on different machines (using Bash)?

    $ diff <(ssh -n george cat /etc/passwd) <(ssh -n kramer cat /etc/passwd)

    This is just the Bash Process Substitution trick.

  4. How can I run a shell script on a remote host without copying the script out?

    One way would be:
    jerry:~$ cat | ssh kramer /bin/sh

    This one is pretty simple how it works, but it is often overlooked as an option for getting things done. The shell script is written to STDOUT. /bin/sh is executed on the remote server, and it reads on STDIN, thus executing the local copy of the script. This is way convient for some things. The only problem I see with this one is that you can't pass command line arguments to the script.

  5. How can I run long pipe lines of commands on a remote host via SSH without escaping all the meta characters?

    jerry:~$ ssh kramer <<EOF
    ps -ef | grep http | awk '{print \$NF}'

    The only tricks here are the use of a bash here document, and the fact that the command line is typed directly into ssh's STDIN so there's no need to escape things like pipes and semi-colons, etc. However, notice that you do still need to escape dollar signs because they'll still be interpreted as shell variables.

  6. How do I change every occurance of a string in multiple files?

    Why, use perl pie!
    perl -p -i -e 's/jerry/george/g' *.txt

    See perl -h for a description of the flags.


Anonymous said...

Rather than using sed to manipulate file paths I prefer dirname and basename - they're more obvious and more likely to work if you have unusual names or character sets involved.

SandyDee said...

Great tips ;)
Hi Greg,
My name is Sandra and the only way I found "to contact you" is your blog. I found the link thanks to the mac google blog. I'm French and I'm finishing my MS degree in Computer Science in Paris at EPITA ( and I have deposed my resume on Google for an internship as a "product manager associate". My boyfriend has also done his internship in Google Moutain View as a SE.
And I'm writing to you in order to have some more information about PMA.. maybe you could help me with that ^^ I read everything about it Google/Jobs but I want to have more details. How is the job ? How is the work ? and so on :) I am very curious :)
Could you help me with that if you have some free time ?

I hope I'm not boring you.
Thanks a lot,

Anonymous said...

very useful!

Trent said...

Re finding the script directory; the following works for me - it also gives you the real path when the script is referenced via a symlink

dir=$(dirname $(readlink -f $0))

Anonymous said...

RE: How do I change every occurance of a string in multiple files?

> Why, use perl pie!
> perl -p -i -e 's/jerry/george/g' *.txt
> See perl -h for a description of the flags.

How would I run this script, then instead of writing the chnages out to the files I woul like to send the files to a tar archive? What would this look like?