Today I’m going to explain the use of features like history, brace and file expansion, and other tricks by example and give you references for later.

Master your history

Those who forget history are doomed to repeat it. This is arguably one of the best productivity enhancing features of any shell.

You can check your history with the history command, which prints your last 500 commands (or so) by default. Alternatively, you can filter the list:

# Print last 10 entries
history 10

# searches history for cmd
history | grep cmd

Each entry has a number, which you can then execute with !<number>

Now suppose I want to copy a file to a directory and then change to that directory. The quick way to do that with history is:

cp myfile.txt my/directory/path
cd !$  # cd my/directory/path

or if I forget to run a command as super-user:

vi /etc/fstab  # oops!
sudo !!  # sudo vi /etc/fstab

to execute the last command starting with "mount", since I don’t want to type it all out:

# Previously...
mount 192.168.0.100:/my/path/to/music /media/music

# Later...
!mount
# Repeats last mount command

Note that I often (but not always) prefer Ctrl-R, which will search history as you type. As an added bonus you can view the command before executing it.

Other examples:

eric@sawyer:~$ echo foo -a bar baz
foo -a bar baz
eric@sawyer:~$ echo !:3-4
bar baz
eric@sawyer:~$ !-2 #2nd-to-last command
foo -a bar baz
eric@sawyer:~$ ^ba^ya #replace 1st "ba" with "ya"
foo -a yar baz
eric@sawyer:~$ !^:p #MUCH cooler than "echo ..." ;)
foo
eric@sawyer:~$ !?bar #Last command containing "bar"
foo -a bar baz
eric@sawyer:~$ !:gs/ba/ya #replace all "ba" with "ya"
foo -a yar yaz

Quick reference

Download as PDF

Helpful .bashrc entries for history

Copy and paste these into ~/.bashrc

# Don't put duplicate lines in the history
export HISTCONTROL=ignoredups

# Store a lot history entries in a file for grep-age
shopt -s histappend
export HISTFILE=~/long_history
export HISTFILESIZE=50000

# No reason not to save a bunch in history
# Takes up several more MBs of RAM now, oOOOooh
export HISTSIZE=9999

# Ignore dupe commands and other ones you don't care about
export HISTIGNORE="&:[ ]*:exit"

Another neat trick with .inputrc

If you are still particularly fond of the up and down arrows, copy and paste the following into a ~/.inputrc file. This will allow you to start typing a command and then hit the up-arrow to search backwards through your history for commands starting with what you typed. I prefer other methods usually but this is pretty cool, huh?

"\eOA": history-search-backward
"\e[A": history-search-backward
"\eOB": history-search-forward
"\e[B": history-search-forward
"\eOC": forward-char
"\e[C": forward-char
"\eOD": backward-char
"\e[D": backward-char

Further reading

Peteris Krumins has an excellent write-up called The Definitive Guide to Bash Command Line History which goes in-depth on many of the above topics, should you crave more bash history goodness.

Brace expansions

No shorthand list would be complete without the (in)famous brace expansions. Basically, they allow you to specify part(s) of an command to repeat substituting different values of a set within braces. Let me show you what I mean:

# Quickly make a backup
cp file.txt{,.bak}
# Equivalent to 'cp file.txt file.txt.back'

This is obviously very useful to prevent having to repeat parts of files or directory paths.

Suppose I wanted to make a template folder structure, I could make most of the directories I need with:

mkdir -p {src,test}/com/eriwen/{data,view}

This will expand every combination so I end up with src/com/eriwen/data, src/com/eriwen/view, test/com/eriwen/data, and so forth. Being able to create a template directory structure with one line is a big time saver!

Better filename expansion

I'm sure you often use the * operator to match files beginning or ending with something, but bash goes far beyond that. It should be noted, though, at some point a good find | grep is more powerful and useful. See Find is a beautiful tool for more information.

In addition to wildcards with *, you can use ? to match any single character. You can also limit matching to certain characters with []. For example:

ls
# prints "myfile netbeans.conf netbeans-6.5rc2 netbeans-6.5 netbeans-6.7 src"
ls netbeans-6.?
# matches "netbeans-6.5 netbeans-6.7"
ls netbeans-6.[1-5]*
# matches "netbeans-6.5rc2 netbeans-6.5"

.bashrc entries for better filename expansion

# Include dot (.) files in the results of expansion
shopt -s dotglob
# Case-insensitive matching for filename expansion
shopt -s nocaseglob
# Enable extended pattern matching
shopt -s extglob

cd shorthand

There are a couple quick tricks to change to oft-used directories. For example:

# Lame way to go home
cd ~

# The cool way
cd

You can also switch to the previous directory with cd - like so:

pwd  # prints /home/eriwen/src
cd /my/webserver/directory

# Do something...

cd -
# Now I'm back in /home/eriwen/src

To get more advanced with this, try mastering your directory stack with pushd and popd.

Conclusion

Effective use of history, brace expansions, and other shortcuts will to save you a lot of time. However, nothing is more productive than no typing. Automating things is best where possible. If you can't automate, use smart aliases.

I know I did not cover tilde expansions, shell parameter expansions or bash key commands. You will want to check those out as well, but I find them less useful than what I've covered here.

Have any quick shortcuts you love? Share them in the comments!

Related posts:

1. grep is a beautiful tool
2. Using aliases and command-line functions for speed
3. awk is a beautiful tool

What is the directory stack?

Most Linux environments have a way for you to put paths on a stack (push) and then take them off in reverse order (pop). This is useful when you have more than one directory that you need to switch between frequently. Let’s take a look at how to do this.

The pushd command

Suppose you need to switch between your project: ~/src/myproject, your web-server: /opt/webserver7/logs and some code examples: ~/examples/othercode often.

You add things to the directory stack with pushd like so:

# switch from project to web server and put on the stack
pushd /opt/webserver7/logs
# Stack is now '/opt/webserver7/logs ~/src/myproject'

# show latest errors
tail -40 errors.log

Now your in your server logs and you want to go to the examples:

# switch from project to web server and put on the stack
pushd ~/examples/othercode
# Stack is now '~/examples/othercode /opt/webserver7/logs ~/src/myproject'

# bring the 3rd directory on the stack to the front (0-based) and rotating the stack
pushd +2
# Stack is now '~/src/myproject ~/examples/othercode /opt/webserver7/logs'

# copy Java example to myproject
cp MyExample.java `dirs | awk '{print $1}'`

The popd command

Now we want to go back to my project to deploy it to our web server. Since we have our directory stack working we can do this quickly:

# go back to myproject
popd
# Stack is now '~/examples/othercode /opt/webserver7/logs'

# deploy stuff
ant dist

For the purposes of this example, let’s pretend we don’t care to go back to the examples. Let’s remove it from the stack:

# remove examples from the directory stack
popd +0
# Stack is now '/opt/webserver7/logs'

One final pop will get me back to my server logs once I test it out in my browser. Of course, we can keep using the stack continuously.

Alternatives and nifty tips

You might not always want to use the directory stack. You can use these really fast shortcuts to navigate to home or back quickly:

# change to home directory quickly in bash
cd

# change to last directory in bash
cd -

Now for some final examples that may help out just a little more:

# print directories on the stack
dirs

# hey just type less
alias p='pushd'
alias o='popd'

Conclusion

Using pushd and popd effectively can help you get around your command-line environment quickly. See the MAN pages for more information.

Take a bit of time to get the hang of it now and it’ll pay off later. Share your other examples or questions!

Related posts:

1. Using aliases and command-line functions for speed
2. Effective bash shorthand
3. Start using crontab for automation

Fire up your terminal or Cygwin now.

crontab tutorial

Suppose I want to copy my personal wiki to my website every other hour between 8:30 and 18:30 on weekdays only. This only takes a couple minutes to setup with a bit of cron-fu.

I’m going to go ahead and use FTP to put my wiki where I want, so I wrote a quick bash script (backup_wiki.sh) for this purpose:

#!/bin/bash
# File: backup_wiki.sh
HOST='mysite.com'
USER='myuser'
PASS='mypassword'

ftp -n ${HOST} <<END_SCRIPT
quote USER ${USER}
quote PASS ${PASS}
put path/to/my/wikifile.html wikifile.html
bye
END_SCRIPT
exit 0

Sweet, so now we can just use backup_wiki.sh

Let’s edit (or create) our new crontab file:

crontab -e

This brings up vi (by default) with a file that may have a comment or may be empty. I don’t feel like using vi right now, so I’ll change it to jEdit by adding the following to my .bashrc file:

export EDITOR="[/path/to/jedit.bat (windows) or 'jedit' (*nix)]"

Ah, that’s better. Now that we can open it up in our fav. text editor, let’s learn how to create an entry in our crontab file.

crontab file structure

Lets break down a sample command that we’ll be putting into our crontab file

#min hour dom month dow command
30 8-18/2 * * 1-5 ./path/to/backup_wiki.sh

This command will run our backup_wiki.sh script at 8:30, 10:30, … 18:30 every Monday(1) through Friday(5). The crontab file basically has one command per line in the following format with the following separated by a space:

• Minutes [0-59]
• Hour [0-23]
• Day of Month [1-31]
• Month [1-12] – January is 1, obviously
• Day of Week [0-6] – Sunday is 0
• Command to run (can have spaces)

An asterisk (*) means all possible values, so in our example above we mean all days of all months. You can specify a range by using a dash (-). Also, using a slash and then a number (like /2) after a command means only run on increments divisible by 2, so 8-19/2 means 8, 10, 12, 14, 16, 18.

Now suppose we just want to

# view our crontab entries without editing them
crontab -l

# remove your crontab file and start fresh
crontab -r

# on *some* systems (not Cygwin), view the last edit time of crontab
crontab -v

example crontab entries

You’d be surprised what you can automate. Here are some simple examples to give you ideas:

# Run backup at 3am every night
0 3 * * * ./nightly_backup.sh >> /logs/dir/backups.log

# Mail me feedburner subscribers every weekday
0 6 * * 1-5 mail -s "Daily Subscriber Report" myemail@gmail.com < `get_subscribers.sh`

# Download meeting agenda wiki page every Tues/Thurs in Dec. 1st-24th
30 8 1-24 12 2,4 ./path/to/curl_get_agenda.sh

Friend Mark Sanborn has another post about cron that has other useful bits I may not have covered here.

I hope you found this intro. to crontab useful. Please share your ideas for automating with it in the comments!

Related posts:

1. Using aliases and command-line functions for speed
2. Find is a beautiful tool
3. Get sed savvy – part 2

Stop typing so much!

One great way to enhance your physical productivity is, obviously, to type less. You know what you want to do and it should take no time for you to tell your computer to do it. On the command-line, we can do this most easily with smart aliases, and perhaps creating functions that streamline our tasks for us so that we can accomplish this.

I’m going to share with you a few techniques I use to cut the number of keystrokes I type significantly while boosting my efficiency, so break out Cygwin or your favorite terminal and let’s get started.

Find out what commands you use most frequently

Here is a quick command you can use to figure out the 10 most used commands in your history (learned from lifehacker’s post):

# Get most used commands from your history
history|awk '{print $2}'|awk 'BEGIN {FS="|"} {print $1}'|sort|uniq -c|sort -r

My top list included cd .., ls, grep, and ant just to name a few. Your list will differ. I probably use each at least a dozen to a hundred times per day. If we do some stupid averaging of time spent with those extra characters, we could make some completely unscientific guess that we can save about 20 minutes a day if we only had to type two or three characters (including enter) for each of these commands.

Create aliases smartly

Now we can take this list and create some aliases to shorten those commands most frequently used. I use bash so I’ll edit my ~/.bashrc file (create in your home directory, if it doesn’t exist), and you can edit the file pertaining to your shell adding something like the following:

# Simple Commands
alias ..='cd ..'
alias ...='cd ../..'
alias a='ant'
alias c='cd'
alias f='find'
alias g='grep'
alias h='history'
alias l='ls -l'
alias o='popd'
alias p='pushd'

# Alias directories for quick access - links often work better but these are helpful
alias myproj='/path/to/myproject/'
alias otherproj='/other/path/directory/'

# More complex commands
alias mybox='ssh myusername@mybox.blah.com'
alias vpn='sudo vpnstuff -l connectstr && sudo morevpnstuff'

You get the idea. There are obviously a lot more, but it’d get boring if I shared ALL of mine. I recommend aliasing many commands to just one character. Over time, it will become second nature and your command-line will burst in flames from your speed (I still owe Casey a new monitor… sorry buddy… forgive me?)

When an alias just won’t do

Now there are obviously times where you really don’t want to put it all in one command, you might not want cd && blah && stuff || mkdir -p foo/bar as an alias. Unfortunately, I see so so few people use command-line functions to their advantage, but they really help you automate tasks.

Suppose you want to grab a certain file off of FTP, but you’ll need to do it often as the file changes. Setup a function so you don’t have to do it physically:

# Create variables
HOST=myhostname
PASSWD=mypass
FILE=myfile.out

function getmyfile() {
  ftp -n $HOST <<END_SCRIPT
  user ${USER} ${PASSWD}
  cd ~/the/file/path
  get $FILE
  quit
  END_SCRIPT
}

# To invoke on the command-line:
getmyfile

You can put whatever sequence of commands you want in a function to automate your task. This can get much more complex if you like, using variables and output from other commands, stay tuned. The more you automate, the better you get at it.

Conclusion

The point here is to express as much as you can with the least amount of effort. Keep building on your automation in this way and you’ll find yourself with much more time for more important tasks. Now go melt some faces!

Please share your favorite alias or function in the comments!

Related posts:

1. Crush images on the command-line with Groovy
2. Use the table-layout CSS property to speed up table rendering
3. Effective bash shorthand

Tutorial

To best introduce awk I’d like to start with a practical example. Most of the applications for awk that I’ve dealt with involve formatting some output or data into something cleaner and more usable. This is certainly not the limit of awk, it is a full fledged language with all the power and responsibility to go with it.

Awk operates on one “record” at a time, which is each line by default. Each “field” in a record is separated by a space (by default) or another defined separator (using the -F option).

We’re going to print the file names, line-numbers, and function names of all duplicate functions so they’re super easy to find and remove. Suppose we have some output from grep with file names and line numbers using a command like this (pulled from my grep tutorial):

# Prints javascript functions like this - <file>:<line-num>:<line-content>
grep -EnH "^\s*function \w+" *.js | sort

That’s all good and well, but this doesn’t quite give us what we want in a clear manner, it prints all functions and information in a kinda cludgy fashion. We can clean this up with a bit of awk. First let’s learn a basic awk command:

awk '/'^$'/ { print "blank line" }' myfile

This snippet will print “blank line” for line that matches the regex: ^$ (a blank line). The pattern (between the two slashes, inclusive) is optional and we’ll see in the next example:

awk 'BEGIN { print "Hello, awk!" } { print $2, $1 } END { print "Goodbye, awk!"}' myfile

How this reads is: Before processing (BEGIN), print "Hello, awk!" and then print the second field ($2) and then the first ($1) for each line of myfile, then after processing (END) print "Goodbye, awk!". The BEGIN and END clauses are optional.

As explained above, a field is a sequence of non-whitespace characters. So if a line contained "foo bar other stuff", awk would print "bar foo".

One more thing before continuing: awk scripts can get ugly so it is useful to know you can read a file for awk commands:

#My awk file: foo.awk
BEGIN { print "Hello, awk!" }
{
    print $2, $1
    # This is an awk comment
}
END { print "Goodbye, awk!" }

#Invoke foo.awk
awk -f foo.awk myfile

OK, let’s add some power to that old grep command. We can use the -F option to specify our delimiter so let’s add to our grep command like so:

grep -Eoni "^\s*function \w+" *.js | awk -F ':' '{print $3," ",$1,$2}' | sort

Now we have a sorted list of function names followed by their filename and line number by spaces. We needed to print $3 first so that we could easily sort by the function name and look for duplicates, but we’re not going to do that manually… oh noooooo way.

Let’s make it a bit prettier and uniq-ify our list of functions by function:

# New part of our command
awk '{print $3,"line",$4,$2}' | uniq -f 3 -D

# Full command
grep -Eoni "^\s*function \w+" *.js | awk -F ':' '{print $3," ",$1,$2}' | sort | awk '{print $3,"line",$4,$2}' | uniq -f 3 -D

Here we pipe our last command back into awk printing <filename> line <line-num> <function-name> and then use uniq on the 3rd field (-f 3) showing only the duplicates (-D).

Other awk examples

#Backup all JavaScript files with a .bak extension -- replace 'bash' with your shell
 ls *.js | awk '{print "cp "$0" "$0".bak"}' | bash

#Print the number of lines that contain "function"
awk '/'function'/ {i = i + 1} END {print i}' myfile.js

Conclusion

AWK is admittedly a ten ton gorilla of a tool, so there is no way I could cover everything that it can do in one post. If there is enough demand I can write about some of the more advanced features like conditionals, variables, and formatting. Stay tuned!

Further Reading

To see how far this rabbit hole goes, you should check out a couple of my favorite references:

• awk, nawk, and gawk cheat sheet
• AWK – Wikipedia
• Hartigan’s AWK reference

Hope you found this informative and want to know more. I know you all probably have some awk gems to share, let’s see them!

Related posts:

1. grep is a beautiful tool
2. Find is a beautiful tool
3. Vim is a beautiful tool

Soon we’ll be looking at awk and other tools to continue the quest for command-line fluency. If you haven’t already, install Cygwin and check out part 1 and part 2.

Tutorial

One of the best ways to crank out code quickly is by using templates. Using the Stream EDitor, you can streamline the use of templates.

Suppose we have a template HTML file that we want to reuse often. Maybe it looks like this:

<html>
    <head>
        <title>template.html</title>
     </head>
    <body>
        <div id="nav">Navigation here</div>
        <div id="content">
%%CONTENT%%
        </div>
    </body>
</html>

Now we want to replace "%%CONTENT%%" with the contents of an HTML Fragment file. The syntax is simple: ‘/<pattern>/r’:

sed '/^%%CONTENT%%/r fragment.htmlf' template.html

The above script will append the contents of fragment.htmlf immediately after "%%CONTENT%%". So we can use the delete command to fix that:

sed -e '/^%%CONTENT%%/r fragment.htmlf' -e '/^%%CONTENT%%/d' template.html > whole.html

This might seem slightly useless, but the power here is in the simplicity. Many times I’m generating bits of Wiki code or HTML, and this has been invaluable.

OK, now for one more command: write (w). Suppose we have a CSV file that we want to split into several files depending on the value in the last cell. We could do this with grep, or awk (coming soon), but with sed we can do it with more efficiently:

#sedscript file
/,[0-9]+$/w numbers.csv
/,[A-Za-z]+/w letters.csv
/,[^A-Za-z0-9]+/w symbols.csv

sed -r -f sedscript original.csv

Now numbers.csv will contain all rows that the last cell containing numbers, and so on for letters.csv and symbols.csv. A neat application for this might be sorting your giant contacts list into different files based on some criteria. This is a simple example, but you can probably think of a more useful scenario where you’d want to filter and split a file.

Other Examples

#Print everything outside the <html> tag (check DOCTYPES)
sed '/<html>/,/</html>/d' myfile.html

#Convert rn (DOS) to UNIX n
sed 's/$//' myfile        #Windows
sed 's/.$//' myfile       #Linux/UNIX

Conclusion

You have now learned several sed commands and patterns you can use to make editing files and some searching tasks much more efficient, and even better, scriptable so they can be automated in a process. One cool application would be to get comments from a set of files and post them to a wiki. It would sure make collaboration slick, right?

Obviously, you can bookmark this stuff but you’ll really get good at it only if you just try it out. Keep sharing your experiences and command lists, they’re great!

Related posts:

1. Get sed savvy – part 2
2. Get sed savvy – part 1
3. awk is a beautiful tool

Suppose we want to let someone else know what kinds of functions are in a given Javascript file. Think of it as a simple sort of Javadoc for CSS or Javascript. The way we are going to do this is look at all of the files modified in the last day and then extract the comments out of them and put them somewhere (on a wiki perhaps?). Doing this kind of automation will increase team communication and productivity immensely if done correctly.

Tutorial

Download and install Cygwin if you’re on Windows to follow along.

# Single-line comments - grep's better but we can use sed
sed -n '///p' blah.js > /tmp/comments.out

# Multi-line comments
sed -n '//*/,/*//p' blah.js >> /tmp/comments.out

Now, the sed commands above are tricky so here is how you can understand them: The -n option tells sed not to print anything unless you tell it specifically what to print. The comma [,] in between the two patterns tells sed to match everything between the two patterns, in this case everything between multi-line comments /* and */ and then the p-command prints whole lines that match the pattern space.

We can combine these two commands to streamline a killer process.

# sed script file
////p
//*/,/*//p

# Use the sed script to print all comments
sed -n -f sedscr blah.js > /tmp/comments.out

Now we have a nice little summary of our Javascript files we can post to a wiki or diff with another version to see what was added. Note that the sed print command prints the whole line, so if you have comments at the end of a line you will get the beginning of that line also. Not a perfect solution, but something quick and easy!

Other Examples

# Print lines longer than 80 characters
sed -n '/^.{81}/p' myfile

# Delete blank lines
sed '/^$/d' myfile

# Substitution optimized for speed
sed '/Yahoo/ s//Not Microhoo/g' myfile

Conclusion

You should now be getting pretty proficient with sed. Use it along with find and grep and you will find yourself feeling much more comfortable on the command-line.

I encourage you to experiment a bit and use this even in circumstances where you know it’s not necessary, just to get the hang of it. In the long run you’ll end up increasing your productivity by using these most powerful tools.

Related posts:

1. Get sed savvy – part 3
2. Get sed savvy – part 1
3. awk is a beautiful tool

The bread and butter of sed is its search-and-replace functionality. Let’s start with that and then throw in some other fun commands.

Tutorial

As with the previous posts, if you are on Windows you’ll want to install Cygwin or one of the various other tools suggested in the previous comments. sed also uses regular expressions so you’ll want to keep your regex reference handy. From Wikipedia:

[sed] reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

sed 's/#FF0000/#0000FF/g' main.css

We can read this like so: search [s/] for red [#FF0000/] and replace it with blue [#0000FF], globally [/g] in main.css. Two notes here: 1) This does not actually modify the file, but outputs what the file would look like if it did the replace and 2) If we left off the "g" at the end it would only replace the first occurrence. So let’s modify the file this time.

sed -i -r 's/#(FF0000|F00)b/#0F0/g' main.css

This is an example from the find tutorial that replaces all instances of red with green in our CSS file. The -r option here gives us extra regex functionality. As Sheila mentioned in the find post, -i does not work on Solaris and she suggests something like perl -e s/foo/bar/g -i instead.

Suppose we want to change a whole color scheme though, the best way is to use a sed script file like so:

# sedscript - one command per line
s/#00CC00/#9900CC/g
s/#990099/#000000/g
s/#0000FF/#00FF00/g
...

# use sedscript with -f
sed -i -f sedscript *.css

sedscript is obviously a new file we have created. Note that we don’t quote the commands in the file. Now we have successfully changed our color scheme in our CSS files.

Other Examples

# Trim whitespace from beginning and end of line
# You *might* have to type a tab instead of t here depending on your version of sed
sed -r 's/^[ t]*//;s/[ t]*$//g'

# Delete all occurances of foo
sed 's/foo//g'

Conclusion

You should start seeing how you can make a lot of changes with simple one-liners with sed. Using it effectively can really increase your efficiency with some tasks.

Here are some good references you should bookmark (including this page of course ;)

• http://sed.sourceforge.net/sed1line.txt – Eric Pement
• http://www.catonmat.net/blog/sed-stream-editor-cheat-sheet/ – Peteris Krumins

I’d say that 90% of the time I use sed for search-and-replace, so you’ve got a good start here. As I mentioned earlier, there is a LOT more to sed. Later, I’ll show you how to make deletions, add line numbers to files, print specific lines by line number, and much more. Stay Tuned, and share your favorite one-liners in the comments!

Related posts:

1. Get sed savvy – part 3
2. Get sed savvy – part 2
3. Find is a beautiful tool