Get sed savvy - part 1
Today I’ll continue the series on command-line tools for productivity, with sed. Stream EDitor is the most complicated tool so far, an entire language in its own right. It is much too big to cover completely in one post, so I’m going to have a few posts covering the major parts of sed.
The bread and butter of sed is its search-and-replace functionality. Let’s start with that and then throw in some other fun commands.
Tutorial
As with the previous posts, if you are on Windows you’ll want to install Cygwin or one of the various other tools suggested in the previous comments. sed also uses regular expressions so you’ll want to keep your regex reference handy. From Wikipedia:
[sed] reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.
sed 's/#FF0000/#0000FF/g' main.css
We can read this like so: search [s/] for red [#FF0000/] and replace it with blue [#0000FF], globally [/g] in main.css. Two notes here: 1) This does not actually modify the file, but outputs what the file would look like if it did the replace and 2) If we left off the "g" at the end it would only replace the first occurrence. So let’s modify the file this time.
sed -i -r 's/#(FF0000|F00)b/#0F0/g' main.css
This is an example from the find tutorial that replaces all instances of red with green in our CSS file. The -r option here gives us extra regex functionality. As Sheila mentioned in the find post, -i does not work on Solaris and she suggests something like perl -e s/foo/bar/g -i instead.
Suppose we want to change a whole color scheme though, the best way is to use a sed script file like so:
# sedscript - one command per line s/#00CC00/#9900CC/g s/#990099/#000000/g s/#0000FF/#00FF00/g ... # use sedscript with -f sed -i -f sedscript *.css
sedscript is obviously a new file we have created. Note that we don’t quote the commands in the file. Now we have successfully changed our color scheme in our CSS files.
Other Examples
# Trim whitespace from beginning and end of line # You *might* have to type a tab instead of t here depending on your version of sed sed -r 's/^[ t]*//;s/[ t]*$//g' # Delete all occurances of foo sed 's/foo//g'
Conclusion
You should start seeing how you can make a lot of changes with simple one-liners with sed. Using it effectively can really increase your efficiency with some tasks.
Here are some good references you should bookmark (including this page of course ;)
- http://sed.sourceforge.net/sed1line.txt - Eric Pement
- http://www.catonmat.net/blog/sed-stream-editor-cheat-sheet/ - Peteris Krumins
I’d say that 90% of the time I use sed for search-and-replace, so you’ve got a good start here. As I mentioned earlier, there is a LOT more to sed. Later, I’ll show you how to make deletions, add line numbers to files, print specific lines by line number, and much more. Stay Tuned, and share your favorite one-liners in the comments!
Why do you use [ \t] instead of \s?
@Andrew:
Call it habit. I guess I’m somehow afraid that \s will match a new line or accidentally convert Windows end of lines (\r\n) to (\n). I’ve never tested this, though :). Let me know if that works for you.
Sweet, keep these coming Eric. Regex’s and command line file manipulation are weak spots for me — these posts are extremely helpful and I’ve printed them and passed them on to the rest of my development team.
@David:
Thanks! I’m glad they’re helpful. Email me what I need to tweak about my print stylesheet for new design :)
Of course \s cannot match a newline because sed processes the program after reading each line; IOW sed’s pattern space will never contain newlines (unless you use the more advanced commands N, G, H, or put newlines in the right side of a substitution command).
It will match carriage returns, however.
@Paolo:
Just as I had thought. So using \s would also convert the files to UNIX format as a side-effect. Not bad if that’s what you want, but just be careful with it.
I made a sed cheat sheet the other day:
Sed Cheat Sheet
It contains:
command line summary
command description, if they take single address or pattern, or a range of addresses, and what they modify (input stream, output stream, pattern space or hold buffer)
command line argument summary
extensions
short summary of adderss range format
Sincerely,
Peteris Krumins
You should not forget the sed one-liners compilation by Eric Pement!
It’s here:
http://sed.sourceforge.net/sed1line.txt
Sincerely,
Peteris Krumins
ps. I was wondering if you could perhaps add my cheat sheet to your post?
I use Eric Pement’s cheatsheet all the time! You guys both deserve links…. added :)
Thanks for posting these. Sed is quite useful, especially in shell scripts.
For ad hoc regex replacements, I prefer vi so I can (u)ndo my mistakes easily, and also highlight the changes. If I’ll be using sed in a script, I may still develop the s/// command in vi for the instant feedback.
I just spent a few hours digging into sed early this week, I’ll point out a few small things I think are worth mentioning about Mac OS X.
The version of sed that comes by default on Mac OS X is POSIX sed and is not quite the same as the GNU sed used in this article. There are differences, the short story is that the GNU version is better.
If you want to run the above examples using the built in Mac OS X sed then instead of the -r switch you should use -E (like grep). You can check out the ‘regex’ and ‘re_format’ man pages for more information concerning the regular expressions allowed for this version of sed. I think its important to note that the regex support is quite poor in this version. Metacharacters like \s, \t, \n don’t work, and it is difficult to get a tab in the OS X terminal if you do end up trying to match whitespace.
It would be best to grab the GNU version of sed. I ran into a problem installing the latest version (4.1.5) on my mac, so you’ll have to instead download 4.1.4, available here:
http://ftp.gnu.org/pub/gnu/sed/sed-4.1.5.tar.gz
GNU sed has quite a few improvements. The regular expressions are actually sane and shouldn’t give you any surprises. I noticed that the maintainer for sed, Paolo Bonzini, commented above. I just emailed him about the gnu 4.1.5 issue.
As mentioned Peteris Krumins’s cheatsheet, and the sed1liners seem to be the staples for sed examples. Also the number 1 result on google (no offense to this blog) is a nice walkthrough:
http://www.grymoire.com/Unix/Sed.html
@Eric: Keep up the great articles. I’m quickly building up my own terminal skills and I like that you’re tackling the hard shell commands (find, sed, …) and you’re providing straightforward and useful examples. =)
Correction to the link for GNU sed. The link for 4.1.4 is the following:
http://ftp.gnu.org/pub/gnu/sed/sed-4.1.4.tar.gz
As an exercise to the reader, to clean up my last post:
gsed -r ’s/4\.1\.5\.tar/4.1.4.tar/’ < myLastPost
Excellent posts!
Keep it coming and thanks. :)
Here are some sed primitives I commonly use:
http://www.pixelbeat.org/cmdline.html#text
@Padraig:
Nice! There are some of these I’ll have to try out. Thanks!
[...] sed savvy - part 2′Now that you know a bit about the Stream EDitor from the last sed tutorial, we are going to expand our knowledge of substitution and line printing with an interesting [...]
I created a sed cheat sheet the other day: [Sed Cheat Sheet](http://www.catonmat.net/blog/sed-stream-editor-cheat-sheet/) It contains: * command line summary * command description, if they take single address or pattern, or a range of addresses, and * what they modify (input stream, output stream, pattern space or hold buffer) * command line argument summary * extensions * short summary of adderss range format
[...] the quest for command-line fluency. If you haven’t already, install Cygwin and check out part 1 and part [...]
[...] Get sed savvy - If you are into automating things, using regular expressions, and learning the REAL “power user tools”, this article is for you. It gives a nice introduction to sed an extremely powerful program. [...]
Hi Eric!
Guess what? I just wrote an article (part 1 of 3) explaining all the sed one-liners! The article is called “Famous Sed One-Liners Explained“.
Hope you and your blog readers find it useful :)
Sincerely,
Peteris
[...] you really want to gain an edge you can always learn how to use sed and [...]
[...] Sed Stream Editor Cheat Sheet - Savvy Sed [...]