Get sed savvy - part 3

7
Aug
2008
Filed under: Bash, Productivity, Tools | 5 Responses »

We will learn about the sed delete (d), read (r) and write (w) commands today to round out your sed toolbox. The major parts I am covering should help you through 99% of the cases where sed is your best option.

Soon we’ll be looking at awk and other tools to continue the quest for command-line fluency. If you haven’t already, install Cygwin and check out part 1 and part 2.

Tutorial

One of the best ways to crank out code quickly is by using templates. Using the Stream EDitor, you can streamline the use of templates.

Suppose we have a template HTML file that we want to reuse often. Maybe it looks like this:

<html>
    <head>
        <title>template.html</title>
     </head>
    <body>
        <div id="nav">Navigation here</div>
        <div id="content">
%%CONTENT%%
        </div>
    </body>
</html>

Now we want to replace "%%CONTENT%%" with the contents of an HTML Fragment file. The syntax is simple: ‘/<pattern>/r’:

sed '/^%%CONTENT%%/r fragment.htmlf' template.html

The above script will append the contents of fragment.htmlf immediately after "%%CONTENT%%". So we can use the delete command to fix that:

sed -e '/^%%CONTENT%%/r fragment.htmlf' -e '/^%%CONTENT%%/d' template.html > whole.html

This might seem slightly useless, but the power here is in the simplicity. Many times I’m generating bits of Wiki code or HTML, and this has been invaluable.

OK, now for one more command: write (w). Suppose we have a CSV file that we want to split into several files depending on the value in the last cell. We could do this with grep, or awk (coming soon), but with sed we can do it with more efficiently:

#sedscript file
/\,[0-9]+$/w numbers.csv
/\,[A-Za-z]+/w letters.csv
/\,[^A-Za-z0-9]+/w symbols.csv

sed -r -f sedscript original.csv

Now numbers.csv will contain all rows that the last cell containing numbers, and so on for letters.csv and symbols.csv. A neat application for this might be sorting your giant contacts list into different files based on some criteria. This is a simple example, but you can probably think of a more useful scenario where you’d want to filter and split a file.

Other Examples

#Print everything outside the <html> tag (check DOCTYPES)
sed '/<html>/,/<\/html>/d' myfile.html

#Convert \r\n (DOS) to UNIX \n
sed 's/$//' myfile        #Windows
sed 's/.$//' myfile       #Linux/UNIX

Conclusion

You have now learned several sed commands and patterns you can use to make editing files and some searching tasks much more efficient, and even better, scriptable so they can be automated in a process. One cool application would be to get comments from a set of files and post them to a wiki. It would sure make collaboration slick, right?

Obviously, you can bookmark this stuff but you’ll really get good at it only if you just try it out. Keep sharing your experiences and command lists, they’re great!

If you liked this post, please help me share it
  • Reddit
  • StumbleUpon
  • description
  • del.icio.us
  • Digg
  • co.mments
  • Google
  • Slashdot
  • Technorati
  • TwitThis
  • E-mail this story to a friend!
  • Furl

Get sed savvy - part 2

23
Jul
2008
Filed under: Bash, Productivity, Tools | 11 Responses »

Now that you know a bit about the Stream EDitor from the last sed tutorial, we are going to expand our knowledge of substitution and line printing with an interesting scenario.

Suppose we want to let someone else know what kinds of functions are in a given Javascript file. Think of it as a simple sort of Javadoc for CSS or Javascript. The way we are going to do this is look at all of the files modified in the last day and then extract the comments out of them and put them somewhere (on a wiki perhaps?). Doing this kind of automation will increase team communication and productivity immensely if done correctly.

Tutorial

Download and install Cygwin if you’re on Windows to follow along. 

# Single-line comments - grep is better but we can use sed
sed -n '/\/\/p' blah.js > /tmp/comments.out

# Multi-line comments
sed -n '/\/\*/,/\*\//p' blah.js >> /tmp/comments.out

Now, the sed commands above are tricky so here is how you can understand them: The -n option tells sed not to print anything unless you tell it specifically what to print. The comma [,] in between the two patterns tells sed to match everything between the two patterns, in this case everything between multi-line comments /* and */ and then the p-command prints whole lines that match the pattern space.

We can combine these two commands to streamline a killer process.

# sed script file
/\/\//p
/\/\*/,/\*\//p

# Use the sed script to print all comments
sed -n -f sedscr blah.js > /tmp/comments.out

Now we have a nice little summary of our Javascript files we can post to a wiki or diff with another version to see what was added. Note that the sed print command prints the whole line, so if you have comments at the end of a line you will get the beginning of that line also. Not a perfect solution, but something quick and easy!

Other Examples

# Print lines longer than 80 characters
sed -n '/^.\{81\}/p' myfile

# Delete blank lines
sed '/^$/d' myfile

# Substitution optimized for speed
sed '/Yahoo/ s//Not Microhoo/g' myfile

Conclusion

You should now be getting pretty proficient with sed. Use it along with find and grep and you will find yourself feeling much more comfortable on the command-line.

I encourage you to experiment a bit and use this even in circumstances where you know it’s not necessary, just to get the hang of it. In the long run you’ll end up increasing your productivity by using these most powerful tools.

If you liked this post, please help me share it
  • Reddit
  • StumbleUpon
  • description
  • del.icio.us
  • Digg
  • co.mments
  • Google
  • Slashdot
  • Technorati
  • TwitThis
  • E-mail this story to a friend!
  • Furl

Get sed savvy - part 1

18
Jul
2008
Filed under: Bash, Productivity, Tools | 19 Responses »

Today I’ll continue the series on command-line tools for productivity, with sed. Stream EDitor is the most complicated tool so far, an entire language in its own right. It is much too big to cover completely in one post, so I’m going to have a few posts covering the major parts of sed.

The bread and butter of sed is its search-and-replace functionality. Let’s start with that and then throw in some other fun commands.

Tutorial

As with the previous posts, if you are on Windows you’ll want to install Cygwin or one of the various other tools suggested in the previous comments. sed also uses regular expressions so you’ll want to keep your regex reference handy. From Wikipedia:

[sed] reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.

sed 's/#FF0000/#0000FF/g' main.css

We can read this like so: search [s/] for red [#FF0000/] and replace it with blue [#0000FF], globally [/g] in main.css. Two notes here: 1) This does not actually modify the file, but outputs what the file would look like if it did the replace and 2) If we left off the "g" at the end it would only replace the first occurrence. So let’s modify the file this time.

sed -i -r 's/#(FF0000|F00)\b/#0F0/g' main.css

This is an example from the find tutorial that replaces all instances of red with green in our CSS file. The -r option here gives us extra regex functionality. As Sheila mentioned in the find post, -i does not work on Solaris and she suggests something like perl -e s/foo/bar/g -i instead.

Suppose we want to change a whole color scheme though, the best way is to use a sed script file like so:

# sedscript - one command per line
s/#00CC00/#9900CC/g
s/#990099/#000000/g
s/#0000FF/#00FF00/g
...

# use sedscript with -f
sed -i -f sedscript *.css

sedscript is obviously a new file we have created. Note that we don’t quote the commands in the file. Now we have successfully changed our color scheme in our CSS files.

Other Examples

# Trim whitespace from beginning and end of line
# You *might* have to type a tab instead of \t here depending on your version of sed
sed -r 's/^[ \t]*//;s/[ \t]*$//g'

# Delete all occurances of foo
sed 's/foo//g'

Conclusion

You should start seeing how you can make a lot of changes with simple one-liners with sed. Using it effectively can really increase your efficiency with some tasks.

Here are some good references you should bookmark (including this page of course ;)

I’d say that 90% of the time I use sed for search-and-replace, so you’ve got a good start here. As I mentioned earlier, there is a LOT more to sed. Later, I’ll show you how to make deletions, add line numbers to files, print specific lines by line number, and much more. Stay Tuned, and share your favorite one-liners in the comments!

If you liked this post, please help me share it
  • Reddit
  • StumbleUpon
  • description
  • del.icio.us
  • Digg
  • co.mments
  • Google
  • Slashdot
  • Technorati
  • TwitThis
  • E-mail this story to a friend!
  • Furl
« Older Entries