Documentation

When you’re writing code that other developers have to use or maintain, you ought to provide some amount of documentation. Your code is simply not self-documenting enough.

My favorite doc tool right now is jsduck developed by Sencha Labs for their Ext JS 4 docs. It basically consumes JSDoc-style comments (with some extras for namespaces, etc.) and generates beautiful documentation. Super easy to install and use:

gem install jsduck
jsduck src/js --output target/docs

Use the Gradle wrapper

Using the Gradle wrapper allows users to build your project without installing Gradle, which is beautiful because it removes a very significant dependency for building and running your project.

Here’s how to use it. Add this to your build.gradle file:

task wrapper(type: Wrapper) {
    // version of Gradle you want
    gradleVersion = '1.0-milestone-6'
}

Then you just have to run in your shell:

gradle wrapper

and Gradle will add a few files to the project, most importantly gradlew and gradlew.bat (for Windows). These are binaries that your can run in lieu of gradle, and it wraps (obviously) the execution of Gradle such that it will download the correct version of Gradle and execute it locally.

I recommend that you switch to the Gradle wrapper when you have consumers of your project that may not have Gradle. I’m going to be doing this for all of my OSS projects from now on.

Introducing the Gradle Web Suite

Gradle really has building JVM-centric source down (for most languages), but I felt like it was cumbersome to operate on my web files with it. So I’ve written a collection of plugins that focus on making tasks with JavaScript, CSS, and other client-side tech dead simple.

buildscript {
	repositories {
		mavenCentral()
	}
	dependencies {
		classpath 'com.eriwen:gradle-css-plugin:0.2'
		classpath 'com.eriwen:gradle-js-plugin:0.2'
	}
}

apply plugin: 'css'
apply plugin: 'js'

And now we can use the tasks provided by these plugins:

// Combine, minify and GZip CSS to teenytiny.css
css {
    input = fileTree(dir: "${projectDir}/css", include: ["file1.css", "file2.css"])
    output = file("${buildDir}/teenytiny.css")
}

// Same thing here, and look, file-globs :)
js {
    input = fileTree(dir: "${projectDir}/js", include: "**/*.js")
    output = file("${buildDir}/combinedMinifiedAndGzipped.js")
}

// ... and there's support for other tools like CSS Lint!
csslint {
	inputs.files fileTree(dir: "${projectDir}/css", include: "**/*.css")
	outputs.file file("${buildDir}/csslint.xml")
	options = ["--rules=adjoining-classes,box-model", '--format=lint-xml']
}

Gradle CSS Plugin | Gradle JS Plugin

JsTestDriver

One tool I hadn’t paid enough attention to until recently was JsTestDriver. The awesome thing about it is that it has adapters for other JS testing frameworks like Jasmine. This is great because I don’t have to convert my tests (mostly) to JsTestDriver’s test API and I get crazy speed and reporting I wouldn’t otherwise get.

In a nutshell, JSTD is a testing tool developed by Google whereby you start as server and then “capture” browsers so that JSTD uploads tests to them, the browser runs tests and then sends back test results. This is awesome because it’s super fast and you can have it run every major browser simultaneously!

Here’s how you can use it. First, we need to download JsTestDriver and the coverage plugin JARs. Then we need a configuration file for JsTestDriver:

server: http://localhost:4224

load:
  - lib/jasmine.js
  - lib/sinon-1.2.0.js
  - lib/jasmine-sinon.js
  - lib/JasmineAdapter.js
  - ../js/main.js

test:
  - spec/*.js

plugin:
  - name: "coverage"
    jar: "lib/plugins/coverage.jar"
    module: "com.google.jstestdriver.coverage.CoverageModule"
    args: useCoberturaFormat

timeout: 30

then we need to start the JSTD server:

java -jar path/to/JsTestDriver.jar --port 4224

Now we want to capture a browser by navigating it to http://localhost:4224/capture. Finally, run our tests with all captured browsers:

java -jar path/to/JsTestDriver.jar --tests all

and you should see something like this:

Chrome: Reset
.................
Total 17 tests (Passed: 17; Fails: 0; Errors: 0) (13.00 ms)
  Chrome 17.0.942.0 Mac OS: Run 17 tests (Passed: 17; Fails: 0; Errors 0) (13.00 ms)
/Users/eric/src/eriwen.com/test/spec/PageSpec.js: 95.789474% covered
/Users/eric/src/eriwen.com/test/lib/jasmine.js: 51.640926% covered
/Users/eric/src/eriwen.com/test/lib/sinon-1.2.0.js: 22.916668% covered
/Users/eric/src/eriwen.com/test/lib/jasmine-sinon.js: 73.333336% covered
/Users/eric/src/eriwen.com/test/lib/JasmineAdapter.js: 64.83517% covered
/Users/eric/src/eriwen.com/js/main.js: 60.15037% covered

And here’s how you can run it with Gradle:

task jstd(type: Exec, dependsOn: 'init', description: 'runs JS tests through JsTestDriver') {
    // Default to MacOS and check for other environments
    def firefoxPath = '/Applications/Firefox.app/Contents/MacOS/firefox'
    if ("uname".execute().text.trim() != 'Darwin') {
        firefoxPath = "which firefox".execute().text
    }

    commandLine = ['/usr/bin/env', 'DISPLAY=:1', 'java', '-jar', "${projectDir}/path/to/JsTestDriver.jar", '--config', "${projectDir}/path/to/jsTestDriver.conf", '--port', '4224', '--browser', firefoxPath, '--tests', 'all', '--testOutput', buildDir]
}

Notice the coverage numbers? JSTD has generated a report in lcov format, but this isn’t very readable to humans.

Code Coverage!

I couldn’t find any working scripts on the interwebs to convert the coverage file to Cobertura XML for reporting, so I wrote an lcov-to-cobertura-xml converter so we can report code coverage with Jenkins!

And again with Gradle:

task jsCoverage(type: Exec, dependsOn: 'jstd', description: 'JS code coverage with cobertura') {
	commandLine = ['python', "${projectDir}/path/to/lcov-to-cobertura-xml.py", '-b', "${projectDir}/", '-e', 'test.spec', '-e', 'test.lib', '-o', "${buildDir}/coverage.xml", "${buildDir}/jsTestDriver.conf-coverage.dat"]
}

I talked about all of this at the Rich Web Experience and I have some slides from my talk that might be helpful if you want more detail about this stuff.

What about PhantomJS?

PhantomJS is still very necessary because JsTestDriver does NOT allow page or DOM interaction like Phantom does. I’m really excited about CasperJS as it looks like a really nice way to do more functional testing with Phantom. I recommend you check it out.

Conclusion

You can see all of this in action in the build for this very site.

So that’s all I have for now. Let me know your experiences with these things or if there is something I missed, comment it up! Cheers!

Related posts:

1. Why every programmer should have a Tiddlywiki
2. Continuous Integration for Javascript
3. Guest Post: Save Text Size Preference Using MooTools and PHP

Jenkins is a CI tool that is often used for Running tests and code analysis for Java and .NET projects. There are a lot of benefits that we as a community are not taking advantage of for our web (CSS, JS, etc) code. In this article I’m going to walk you through setting up automated building and testing for a JavaScript project.

NOTE: The steps outlined are generally Linux/Mac centric, I don’t go into depth on Windows setup, but it shouldn’t be much different using Cygwin.

Why use CI?

Aside from the traditional benefits you see from your compiled code, there are some very compelling reasons:

1. Automate versioning, combining, minifying, and gzipping files
2. Run automated tests and get reports, keeping the codebase maintainable
3. Run static analysis tools like the closure compiler or jshint
4. Auto-deploy files (to S3, say) if our build passes
5. Tag and other special stuff for release builds
6. And that’s not all!^TM We can also hook in Selenium tests, CSS Lint, and more

Not convinced? Tell me why in the comments.

Jenkins setup

Downloading and running Jenkins is incredibly easy. Just download Jenkins and run:

wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war
java -jar jenkins.war

There are also native installers for most environments.

Now point your browser to http://localhost:8080 to see it running.

Nailed it! Now we have a CI server!

Prerequisites

If we’re going to be using git and gradle, we’ll need to install them (Windows Git installer).

sudo yum install git-core  # CentOS/RedHat/etc
sudo apt-get install git   # Debian/Ubuntu/etc
sudo port install git-core # MacPorts
sudo brew install git      # Homebrew

wget http://edub.me/ngMAeR # Gradle 1.0m3 ZIP
sudo mv gradle-1.0* /usr/local
sudo unzip /usr/local/gradle-1.0-milestone-3-bin.zip
sudo ln -s /usr/local/gradle-1.0-milestone-3 /usr/local/gradle

To tell Jenkins where to find Gradle, we go to Manage Jenkins > Configure System from the top page. Add a name and enter your GRADLE_HOME (/usr/local/gradle if you followed the instructions above). It should look like this:

Setup a CI job

As an example, I’m going to use my stacktrace.js project on GitHub. Even though it’s a small JS library, almost all of the setup can be used for any type of project.

First, we want to install a few plugins that will help us out. We’ll need Git to pull down the code and Gradle to build it. You’ll want the Git, Gradle, and Violations plugins installed. It’s pretty easy to figure out, but hit up the Jenkins wiki if you get stuck.

We want to create a new job that runs analysis on our JS and runs our tests. Click New Job on the Jenkins sidebar and you’ll see the setup form.

Checking out and building a repo

Under Source Code Management choose Git and enter git://github.com/eriwen/javascript-stacktrace.git for the repo location and master for the branch. It’ll look something like this:

For the Build section of the setup, we want to run the minify build target from Gradle. You’ll also want to enter the location of the build file as build/build.gradle. Don’t worry about the contents of our build script right now, I have a slew of blog posts in-progress that explain it. Stay tuned.

Now would be a good time to click Save at the bottom and then click Build Now on the sidebar for the job. You should see Jenkins checkout, pull down the latest Closure Compiler, and use it to minify stacktrace.js.

Running JS unit tests with PhantomJS

The days are past when you have to open a browser to see if your JS is generally working. PhantomJS is a headless WebKit browser that lets us interact with web pages (click links, pull from localStorage, etc.) without the browser window! This will let Jenkins run a browser with our tests and report the result. To follow along with the example, download and install PhantomJS in /usr/local/bin (on Mac OS I just sudo ln -s /Applications/phantomjs.app/Contents/MacOS/phantomjs /usr/local/bin/phantomjs). Note that if you’re running Jenkins on a headless server, you’ll want to have xvfb.

We also need to update the Build part of the configuration by telling gradle to run the test target.

Gradle is now setup to run the QUnit tests in the project, but we need to tell Jenkins where to find the test reports. Easy! I encourage you to check out the source of build.gradle if you want to know how this is setup. It’s really quite interesting.

Save and try that build again to bask in automated JavaScript testing awesomeness.

Finding potential bugs with JSHint

I’m a big fan of JSHint and I think it’s worthwhile to have it run on every git push. Here’s how to add that. To follow the example, you’ll have to install NodeJS and the jshint module from npm:

sudo brew install node      # Homebrew
sudo port install node      # MacPorts
sudo apt-get install nodejs # Debian-based

curl http://npmjs.org/install.sh | sudo sh # Install NPM
sudo npm install -g jshint

Now we just need to tell Jenkins to run jshint as well as our tests and where to find the JSHint report. The gradle build puts it in target/jshint.xml. It should look like this:

One more Build Now and we have a fully functioning JavaScript build. You’ll also see graphs with test and jshint results on the job page and can drill down into details.

I have really loved my CI setup for my web projects as well as my JVM-based ones. Now you have the power to go build a cool CI system for your projects. You can read more of my thoughts on this from my notes from my pursuit of the perfect front-end build. Enjoy!

Related posts:

1. Javascript Stacktrace update
2. How to highlight search results with JavaScript and CSS
3. Javascript: Measure those “em”s for your layout

With that in mind, I am introducing an application that I think will help you with that: Cheqlist. A desktop application powered by Remember The Milk that does 2 things very well: make managing tasks very efficient and look sweet on your desktop.

What makes Cheqlist awesome

I wanted to build an application that I, myself, would use everyday. There are a few key features that would make that happen:

• Adding tasks quickly using RTM’s new Smart Add feature
• Visual appeal and lots of room for customization. Special thanks to Rakesh Menon for allowing me to use his JavaFX color picker
• Easily search tasks and use RTM’s custom search keywords
• Must work on all the OSes I use: Windows, Mac OS X, and Linux

On top of that, I think there are some things you’ll come to appreciate:

• Easy-update: Cheqlist checks for updates on startup, and if you want to upgrade, it takes just seconds!
• Free and open-source
• Lots of nice “easter egg” features for those of you who take time to find them ;)

A quick look

I’ve created a short (3 minutes) introductory video to show you the main features. UPDATE: Apparently screenr videos don’t show up in RSS properly. View it here.

Here are a few screenshots:

You can find more on the Cheqlist Wiki.

Try it out!

What you need:

• A marginally recent (6u11 or newer) version of Java
• Remember The Milk account
• Launch Cheqlist

Other resources

Something missing? Request a feature at the issue tracker.

Having trouble? Let me know directly.

For more information, you can hit the project page and Wiki on GitHub.

Enjoy!

Related posts:

1. Introducing GroovyRTM: A Groovier way to Remember The Milk
2. What and why: the tech behind Cheqlist
3. How I use multiple desktops to get things done

Why would a personal wiki make you productive?

In a phrase: to keep a your web of knowledge accessible from one place. You need a place to put your meeting agenda, important project/server links, and even a to-do or waiting-for list. Not only can you store a lot, you can tag it, search it and otherwise customize the crap out of Tiddlywiki.

You are not going to be able to remember everything you need, keeping links and information in email folders just doesn’t cut it. You can easily keep track of your meeting notes, DB schema diagrams, bug lists, blah blah blah…

In other news, great programmers Venkat Subramaniam and Mark Richards called Tiddlywiki their favorite technical tool. Favorite. As in more valuable than their IDE.

Roll your own… now!

It’s really stinkin’ easy! Just grab my favorite flavor of Tiddlywiki, d-cubed, and save it somewhere you can access easily. The first time you use it your browser will ask you to confirm that you can edit your HTML file through it (standard security check).

You will probably want to theme it so it fits you, and I found a bunch of nice themes at TiddlyThemes. It’s just CSS and JavaScript anyway, dowhatyouwant! whatyouwant!!

Once you have customized your wiki’s look-and-feel etc., it’s time to grab some slick plugins. Here are a few I use and recommend:

• Checkbox plugin – So you can manage your tasks with checkbox style
• Calendar plugin and Date plugin – Manage your calendar in your wiki!
• Your Search plugin – Really, really sweet search upgrade. A must-grab!

To install a plugin, just make a new Tiddler (wiki page) with the same title, content, and tags. So easy!

Next steps

I recommend starting by creating tiddlers for all your projects and just putting links to other resources in there. After awhile, copy bits of emails you need to remember to relevant places so you can search it anytime. Baby steps, but stick to it. The setup cost will be returned in cases (as in the beer others will buy you for rocking)!

You’re now well on your way to wiki-fying yourself. Share your experiences and have fun!

Oh, hai Hacker News! You should note that this article is almost 3 years old but nevertheless I still use Tiddlywiki! I sync it with my Dropbox account, which syncs it across all of my environments as well as keeps the last few versions. I recommend that, too.

Related posts:

1. Find is a beautiful tool
2. Get sed savvy – part 2
3. 8 steps to my personal Firefox setup for productivity

When it comes to productivity I can’t think of any application that has saved more time and frustration than Vim. Vim is the ultimate productivity tool for programmers. Most users of Vim that I know that have spent the time to learn it absolutely love it and couldn’t imagine a world without it. My only regret with Vim is not learning it sooner.

So what is Vim?

Vim is a modal text editor that is highly configurable, efficient, and lightweight. Some people refer to it as the programmer’s text editor because it has a plethora of features that make programming easier, although it could be used for other things like writing email and editing config files.

What makes Vim especially great is the shortcuts, commands, and the blistering speed at which you can edit text files since Vim has all the common editing tasks mapped to single keys on or near the home row on your keyboard.

Sounds great… but there is a catch:

Vim isn’t an editor designed to hold its users’ hands. It is a tool, the use of which must be learned.

Easing the learning curve

Vim is known for having a steep learning curve. I think Vim has this reputation because people tend to want to learn all of the features right away and quickly become overwhelmed.

Vim is a lot like Adobe Photoshop, you can learn just a few simple techniques in Photoshop that can make a world of a difference in the images you can create and ways you can manipulate them. Yet it may take you a lifetime to master all the functions Photoshop has to offer. Vim is the same way. You can learn only a few shortcuts and commands that will completely change the way you do text editing but may take years to learn them all (not that you would need to).

The great thing about Vim is everyone uses it differently. I have my favorite shortcuts that I use every day. Someone else might have a different set they use depending on what types of files they generally edit. Once you learn a few basics and get comfortable with the shortcuts and commands that you use every day you can learn another to increase your productivity.

Getting Started

I suggest going through vimtutor once right now and again after you have played around with Vim for a while. You can access vimtutor by typing in the command, ‘vimtutor’ for Linux users and by running vimtutor.bat in your Vim folder for Windows users.

Vimtutor has all the commands and how to use them as well as practice sections for all your basic editing needs. Don’t worry if you don’t remember all the commands when going through the vimtutor for the first time; however, it is important to do the exercises at the end of each section so you can get used to the way Vim works. It should only take about a half hour to make it through your first run through the tutorial.

You might consider bookmarking this article for your reference and checking out vimtutor now.

Making Vim Work for You

After using Vim for a while now I have a good idea which commands I use most often when doing web development, programming, and configuration editing. We will use this experience to look at different ways Vim can improve your programming and make it faster while trying not to overwhelm you with too many new commands. Grab a few of these, use them, and add more when you feel comfortable.

Vim modes

In Vim there are three major modes:

• Insert mode is like the Windows ‘notepad’. Simply type and text will output to the screen.
• Command mode is where Vim really shines. This mode is usually called "normal mode" since you will be using this mode a lot. To get into Command mode just hit ‘Esc’. When in command mode almost all letters numbers and keys on the keyboard are now shortcuts. The keys, ‘hjkl’ in command mode are now your "arrow keys".
• Visual mode is where you can highlight text.

If you ever made a mistake and hit the wrong key you can use ‘Esc‘ to get back to normal mode.

Commands by function

Note: As with other *nix tools, these commands are case sensitive.
You can use these commands in conjunction with another command or repeated x number of times by adding a digit to the beginning. For example, ‘4dd‘ would delete 4 lines. ‘dw‘ would delete a word.

Here are some of the most common vim commands and their meanings:

Open and save:

‘:e path/to/filename‘ – With Vim running, this will open the file you specify. This command can also be used to open remote files over SSH.
‘:w‘ – Will save the current file
‘:q‘ – Exits Vim

You can also use these commands together:

‘:wq‘ – Writes to the file and exits Vim

If you want to quit without saving your changes you can do

‘:q!‘ – Quit without saving

Moving Around:

To move your cursor around with vim you will use:

h – left
j – down
k – up
l – right

These movement keys are all located on the home row. It will take a little getting used to but you will thank yourself for not using arrows or the mouse when you realize much time is wasted moving your hands. I even find myself trying to use these keys in other programs and get frustrated when they don’t work. Sometimes I am pleasantly surprised when programs can be navigated with these keys.

Undo and Redo:

Of course you can’t have an editor without undo and redo.

u – undo the last change
U – undo all changes on that line. This is very useful when programming.
Ctrl + r – redo the last change

Copy and Paste:

You will probably find yourself wanting to copy and paste at some point. I will first explain how to copy and paste lines of text in the way that you are used to and then describe a few alternative ways.

To highlight the text you want to copy you will need to put Vim into visual mode. You can do this different ways.

v – go into visual mode and highlight with the Vim movement keys.
V – go into visual mode and highlight lines of text

Once you have the text highlighted you will then "yank" the text with the ‘y’ key.

y – yank (copy) text.

Once you have the text yanked (copied) you can "put" (paste) it with the ‘p’ key.

p – put/paste text.

So far Vim has failed to show any real improvements over the way you are probably used to copying and pasting text. y/p probably isn’t much faster than ctrl+c ctrl+v other than the fact that you don’t have to stretch your fingers to do it. Here are some improvements.

yy – copy current line. This is incredibly useful and fast when doing programming or any type of editing. When using this command in conjunction with, ‘p’ it will automatically add a line to make room. For example you can go to the desired line, hit ‘yy’ then ‘p’ and it will copy the line paste it below and move your cursor to the newly created line.

My most commonly used commands

Like I have said earlier Vim has more shortcuts than you would ever even want to know about but here are the golden ones. These are the commands that I use daily.

• ‘h,j,k,l‘ – The movement keys are essential
• ‘w,b‘ – move forward ‘w’ and backward ‘b’ one whole word.
• ‘dw‘ – delete a word
• ‘u,ctrl+r‘ – Undo and Redo are a must
• ‘dd‘ – This deletes the current line. I am constantly using this. It also puts the text into a buffer so you can put/paste it.
• ‘0,$‘ – The ’0′ will put you at the beginning of the line and ‘$’ will put you at the end.
• ‘i,a‘ – The ‘i’ will put you in insert mode at the cursor. ‘a’ will put you in insert mode after the cursor. Think of it like append.
• ‘:52‘ – This will put you at line 52. When you have errors in your program on line X use this command.
• ‘o,O‘ – The ‘o’ will create a line below and put you in insert. ‘O’ does the opposite. It creates a line above. This doesn’t really seem all that useful because you can always hit the enter key if you are at the end of the line but you are not always in insert mode or at the end of the line when you decide to make another line.
• ‘f,F‘ – This is one of my favorite commands. ‘f’ will search forward in the line for the next character you type and ‘F’ will search backwards.

For example, lets say you have the following line of code:

print LOG "StateProvinceCode: $StateProvinceCode\n";

You could hit the ‘f‘ key then the ‘\‘ key and the cursor would be positioned right near the ‘n’ so I could change this to a tab character instead of a newline character.

This doesn’t mean it will keep on searching, there is a different key for that.

‘/‘ – Search. This is the key to initiate a search. You can use regular expressions or do a simple word search.

Special commands for programmers

Syntax highlighting — If for some reason Vim isn’t highlighting code by default you can turn it on by typing, ‘:syntax on‘ or add it to your .vimrc config file.
‘=%‘ – Will auto tab nested ifs/loops/functions. Although Vim automatically tabs everything out for you, sometimes you will run into old code or poorly tabbed code where you want to fix it up real quick. Just go to the first ‘{‘ in the code and hit this command and everything will be auto tabbed.
‘gd‘ – Jump to variable definition. The g stands for go. There are multiple ways you can use ‘g‘, ‘gg‘ will go to the beginning of the file and ‘G’ will go to the end. In this case we are using ‘gd’ to go to the variable definition. For example, lets say you are about a mile down your code and you have a function that calls the variable, $someOldVariable. To find out what is inside this variable or where it was first created you would move your cursor to that variable and hit, ‘gd’ this would auto-magically move you to where that variable was first seen, AKA the variable definition.
‘%‘ – Go to matching () [] {}. Sometimes you will have a very long loop and a bunch of nested functions. If your tabbing isn’t the best it is sometimes tricky to find out which bracket belongs to which function. The ‘%‘ will move your cursor to the start or end of the matching bracket.
‘>> and >‘ – The double arrow, ‘>>‘ will shift the current line over one tab. If you highlight a region of text you can shift the entire text over with ‘>‘.

Conclusion

Although there are literally hundreds of shortcuts and commands for Vim you will be far ahead of the regular text editor with just these few. If you simply learned the commands in this article alone you would program/edit much faster and be more productive.

Remember: learn just the basics first. Open a text file or project source file and move around and learn how to save it. Learn how to undo-redo and basic editing in insert modes. To really get the hang of Vim you will need to spend about 5 hours messing around in the editor.

Related posts:

1. awk is a beautiful tool
2. grep is a beautiful tool
3. Find is a beautiful tool

Tutorial

To best introduce awk I’d like to start with a practical example. Most of the applications for awk that I’ve dealt with involve formatting some output or data into something cleaner and more usable. This is certainly not the limit of awk, it is a full fledged language with all the power and responsibility to go with it.

Awk operates on one “record” at a time, which is each line by default. Each “field” in a record is separated by a space (by default) or another defined separator (using the -F option).

We’re going to print the file names, line-numbers, and function names of all duplicate functions so they’re super easy to find and remove. Suppose we have some output from grep with file names and line numbers using a command like this (pulled from my grep tutorial):

# Prints javascript functions like this - <file>:<line-num>:<line-content>
grep -EnH "^\s*function \w+" *.js | sort

That’s all good and well, but this doesn’t quite give us what we want in a clear manner, it prints all functions and information in a kinda cludgy fashion. We can clean this up with a bit of awk. First let’s learn a basic awk command:

awk '/'^$'/ { print "blank line" }' myfile

This snippet will print “blank line” for line that matches the regex: ^$ (a blank line). The pattern (between the two slashes, inclusive) is optional and we’ll see in the next example:

awk 'BEGIN { print "Hello, awk!" } { print $2, $1 } END { print "Goodbye, awk!"}' myfile

How this reads is: Before processing (BEGIN), print "Hello, awk!" and then print the second field ($2) and then the first ($1) for each line of myfile, then after processing (END) print "Goodbye, awk!". The BEGIN and END clauses are optional.

As explained above, a field is a sequence of non-whitespace characters. So if a line contained "foo bar other stuff", awk would print "bar foo".

One more thing before continuing: awk scripts can get ugly so it is useful to know you can read a file for awk commands:

#My awk file: foo.awk
BEGIN { print "Hello, awk!" }
{
    print $2, $1
    # This is an awk comment
}
END { print "Goodbye, awk!" }

#Invoke foo.awk
awk -f foo.awk myfile

OK, let’s add some power to that old grep command. We can use the -F option to specify our delimiter so let’s add to our grep command like so:

grep -Eoni "^\s*function \w+" *.js | awk -F ':' '{print $3," ",$1,$2}' | sort

Now we have a sorted list of function names followed by their filename and line number by spaces. We needed to print $3 first so that we could easily sort by the function name and look for duplicates, but we’re not going to do that manually… oh noooooo way.

Let’s make it a bit prettier and uniq-ify our list of functions by function:

# New part of our command
awk '{print $3,"line",$4,$2}' | uniq -f 3 -D

# Full command
grep -Eoni "^\s*function \w+" *.js | awk -F ':' '{print $3," ",$1,$2}' | sort | awk '{print $3,"line",$4,$2}' | uniq -f 3 -D

Here we pipe our last command back into awk printing <filename> line <line-num> <function-name> and then use uniq on the 3rd field (-f 3) showing only the duplicates (-D).

Other awk examples

#Backup all JavaScript files with a .bak extension -- replace 'bash' with your shell
 ls *.js | awk '{print "cp "$0" "$0".bak"}' | bash

#Print the number of lines that contain "function"
awk '/'function'/ {i = i + 1} END {print i}' myfile.js

Conclusion

AWK is admittedly a ten ton gorilla of a tool, so there is no way I could cover everything that it can do in one post. If there is enough demand I can write about some of the more advanced features like conditionals, variables, and formatting. Stay tuned!

Further Reading

To see how far this rabbit hole goes, you should check out a couple of my favorite references:

• awk, nawk, and gawk cheat sheet
• AWK – Wikipedia
• Hartigan’s AWK reference

Hope you found this informative and want to know more. I know you all probably have some awk gems to share, let’s see them!

Related posts:

1. grep is a beautiful tool
2. Find is a beautiful tool
3. Vim is a beautiful tool

Soon we’ll be looking at awk and other tools to continue the quest for command-line fluency. If you haven’t already, install Cygwin and check out part 1 and part 2.

Tutorial

One of the best ways to crank out code quickly is by using templates. Using the Stream EDitor, you can streamline the use of templates.

Suppose we have a template HTML file that we want to reuse often. Maybe it looks like this:

<html>
    <head>
        <title>template.html</title>
     </head>
    <body>
        <div id="nav">Navigation here</div>
        <div id="content">
%%CONTENT%%
        </div>
    </body>
</html>

Now we want to replace "%%CONTENT%%" with the contents of an HTML Fragment file. The syntax is simple: ‘/<pattern>/r’:

sed '/^%%CONTENT%%/r fragment.htmlf' template.html

The above script will append the contents of fragment.htmlf immediately after "%%CONTENT%%". So we can use the delete command to fix that:

sed -e '/^%%CONTENT%%/r fragment.htmlf' -e '/^%%CONTENT%%/d' template.html > whole.html

This might seem slightly useless, but the power here is in the simplicity. Many times I’m generating bits of Wiki code or HTML, and this has been invaluable.

OK, now for one more command: write (w). Suppose we have a CSV file that we want to split into several files depending on the value in the last cell. We could do this with grep, or awk (coming soon), but with sed we can do it with more efficiently:

#sedscript file
/,[0-9]+$/w numbers.csv
/,[A-Za-z]+/w letters.csv
/,[^A-Za-z0-9]+/w symbols.csv

sed -r -f sedscript original.csv

Now numbers.csv will contain all rows that the last cell containing numbers, and so on for letters.csv and symbols.csv. A neat application for this might be sorting your giant contacts list into different files based on some criteria. This is a simple example, but you can probably think of a more useful scenario where you’d want to filter and split a file.

Other Examples

#Print everything outside the <html> tag (check DOCTYPES)
sed '/<html>/,/</html>/d' myfile.html

#Convert rn (DOS) to UNIX n
sed 's/$//' myfile        #Windows
sed 's/.$//' myfile       #Linux/UNIX

Conclusion

You have now learned several sed commands and patterns you can use to make editing files and some searching tasks much more efficient, and even better, scriptable so they can be automated in a process. One cool application would be to get comments from a set of files and post them to a wiki. It would sure make collaboration slick, right?

Obviously, you can bookmark this stuff but you’ll really get good at it only if you just try it out. Keep sharing your experiences and command lists, they’re great!

Related posts:

1. Get sed savvy – part 2
2. Get sed savvy – part 1
3. awk is a beautiful tool

Suppose we want to let someone else know what kinds of functions are in a given Javascript file. Think of it as a simple sort of Javadoc for CSS or Javascript. The way we are going to do this is look at all of the files modified in the last day and then extract the comments out of them and put them somewhere (on a wiki perhaps?). Doing this kind of automation will increase team communication and productivity immensely if done correctly.

Tutorial

Download and install Cygwin if you’re on Windows to follow along.

# Single-line comments - grep's better but we can use sed
sed -n '///p' blah.js > /tmp/comments.out

# Multi-line comments
sed -n '//*/,/*//p' blah.js >> /tmp/comments.out

Now, the sed commands above are tricky so here is how you can understand them: The -n option tells sed not to print anything unless you tell it specifically what to print. The comma [,] in between the two patterns tells sed to match everything between the two patterns, in this case everything between multi-line comments /* and */ and then the p-command prints whole lines that match the pattern space.

We can combine these two commands to streamline a killer process.

# sed script file
////p
//*/,/*//p

# Use the sed script to print all comments
sed -n -f sedscr blah.js > /tmp/comments.out

Now we have a nice little summary of our Javascript files we can post to a wiki or diff with another version to see what was added. Note that the sed print command prints the whole line, so if you have comments at the end of a line you will get the beginning of that line also. Not a perfect solution, but something quick and easy!

Other Examples

# Print lines longer than 80 characters
sed -n '/^.{81}/p' myfile

# Delete blank lines
sed '/^$/d' myfile

# Substitution optimized for speed
sed '/Yahoo/ s//Not Microhoo/g' myfile

Conclusion

You should now be getting pretty proficient with sed. Use it along with find and grep and you will find yourself feeling much more comfortable on the command-line.

I encourage you to experiment a bit and use this even in circumstances where you know it’s not necessary, just to get the hang of it. In the long run you’ll end up increasing your productivity by using these most powerful tools.

Related posts:

1. Get sed savvy – part 3
2. Get sed savvy – part 1
3. awk is a beautiful tool