Get sed savvy - part 3
We will learn about the sed delete (d), read (r) and write (w) commands today to round out your sed toolbox. The major parts I am covering should help you through 99% of the cases where sed is your best option.
Soon we’ll be looking at awk and other tools to continue the quest for command-line fluency. If you haven’t already, install Cygwin and check out part 1 and part 2.
Tutorial
One of the best ways to crank out code quickly is by using templates. Using the Stream EDitor, you can streamline the use of templates.
Suppose we have a template HTML file that we want to reuse often. Maybe it looks like this:
<html>
<head>
<title>template.html</title>
</head>
<body>
<div id="nav">Navigation here</div>
<div id="content">
%%CONTENT%%
</div>
</body>
</html>
Now we want to replace "%%CONTENT%%" with the contents of an HTML Fragment file. The syntax is simple: ‘/<pattern>/r’:
sed '/^%%CONTENT%%/r fragment.htmlf' template.html
The above script will append the contents of fragment.htmlf immediately after "%%CONTENT%%". So we can use the delete command to fix that:
sed -e '/^%%CONTENT%%/r fragment.htmlf' -e '/^%%CONTENT%%/d' template.html > whole.html
This might seem slightly useless, but the power here is in the simplicity. Many times I’m generating bits of Wiki code or HTML, and this has been invaluable.
OK, now for one more command: write (w). Suppose we have a CSV file that we want to split into several files depending on the value in the last cell. We could do this with grep, or awk (coming soon), but with sed we can do it with more efficiently:
#sedscript file /\,[0-9]+$/w numbers.csv /\,[A-Za-z]+/w letters.csv /\,[^A-Za-z0-9]+/w symbols.csv sed -r -f sedscript original.csv
Now numbers.csv will contain all rows that the last cell containing numbers, and so on for letters.csv and symbols.csv. A neat application for this might be sorting your giant contacts list into different files based on some criteria. This is a simple example, but you can probably think of a more useful scenario where you’d want to filter and split a file.
Other Examples
#Print everything outside the <html> tag (check DOCTYPES) sed '/<html>/,/<\/html>/d' myfile.html #Convert \r\n (DOS) to UNIX \n sed 's/$//' myfile #Windows sed 's/.$//' myfile #Linux/UNIX
Conclusion
You have now learned several sed commands and patterns you can use to make editing files and some searching tasks much more efficient, and even better, scriptable so they can be automated in a process. One cool application would be to get comments from a set of files and post them to a wiki. It would sure make collaboration slick, right?
Obviously, you can bookmark this stuff but you’ll really get good at it only if you just try it out. Keep sharing your experiences and command lists, they’re great!
A 24 year-old programmer for
Couple things- The light yellow boxes for commenter metadata make it insanely hard to read what you’ve already written (might just be my monitor, but doublecheck just in case)
Second- I really like this series. It seems like most of my experience trying to learn these command line utils can be condensed down to someone giving me a full solution without explaining it (helpful, but not educational) that resembles PERL Golf to the degree that I can’t even begin to read it, or the elitist “Read the man pages”. This series of concrete, easy to follow examples are invaluable- You might want to consider publishing them as cheat sheets. The one problem I’ve run into so far is that the examples sometimes require different escaping mechanisms when run under cygwin in windows (ie, ; instead of \;), but a little trial and error usually irons that stuff out.
Thanks again:)
-Alex
@Alex:
Thanks for your kind feedback. Escaping is dependent on your shell version, so it is something to keep in mind.
About #1 - I’ve emailed you a link to my upcoming design where all is happy and well contrasted ;) I guess I learned a good lesson in usability - do not color your forms because someones configuration isn’t going to display it like you want.
Nice article, always look forward to reading more of these.
One suggestion. Some of the flags (eg, -i in sed) may not be available in Solaris. It would nice to highlight this to your audience.
@chihung:
Thanks. I did mention this back in part 1. The appropriate way to handle this in Solaris is
perl -e s/foo/bar/g -iI should be highlighting this more often, though. You’re right :)
After learning perl and regular expressions these code snippets no longer look greek.
My only regret was not learning regular expressions/sed/awk earlier. To anyone that is looking at this post wondering when they would ever use this kinda stuff. Do yourself a favor and learn it right away. You wont regret it.
And Eric’s posts a good place to start.
Thanks!