145
submitted 11 months ago by netwren@lemmy.world to c/linux@lemmy.ml

Studying and awk came up.

Spent about an hour and I see some useful commands that extend past what "cut" can do. But really when dealing with printf() format statements is anyone using awk scripts for this?

Or is everyone just using their familiar scripting language. I'd reach for Python for the problems being presented as useful for awk.

top 50 comments
sorted by: hot top controversial new old
[-] Lydia_K@startrek.website 57 points 11 months ago

I use awk all the time, nothing too fancy, but when you need to pull out elements of text it's usually way easier than using cut.

awk {' print $3 '} will pull the third element based on your IFS variable (internal field separater, default is whitespace)

awk {' print $NF '} gets you the last element, and awk {' print $(NF-1) '} gets you one element from the last, and so on.

Basic usage but so fast and easy for so many everyday command line things.

[-] FigMcLargeHuge@sh.itjust.works 13 points 11 months ago

You can also add to the output. I use it frequently to pull a list of files, etc, from another file, and then do something like generate another script from that output. This is a weak example, but one I can think of off my head. Not firing up my work laptop to search for better examples until after the holidays. LOL.

awk {'print "ls -l "$1'}

And then send that to a file that I can then execute.

[-] ook_the_librarian@lemmy.world 25 points 11 months ago

awk will always have a soft spot for me, but I can see why not many take the time to learn it. It tends to be needed right there at the border of problem complexity where you are probably better using a full-fledged scripting tool.

But learning awk is great for that "now you're thinking in pipes" ah-hah moment.

[-] hersh@literature.cafe 23 points 11 months ago

All the time. Not always by choice!

A lot of my work involves writing scripts for systems I do not control, using as light a touch as is realistically possible. I know for a fact Python is NOT installed on many of my targets, and it doesn't make sense to push out a whole Python environment of my own for something as trivial as string manipulation.

awk is super powerful, but IMHO not powerful enough to justify its complexity, relative to other languages. If you have the freedom to use Python, then I suggest using that for anything advanced. Python skills will serve you better in a wider variety of use cases.

[-] walthervonstolzing@lemmy.ml 22 points 11 months ago

awk predates perl as well as python by a pretty large margin (1978); it's useful, of course, for processing things in a pipeline, but as it became obsolete as a general-purpose scripting language, users have had less and less of a reason to learn its syntax in detail -- so nowadays it shows up in one-liners where it could be replaced by a tiny bit of cut.

I had worked through a good bit of the O'Reilly 'sed & awk' book -- the first programming book I got, after being enticed by shell scripting in general. Once I learned a bit of Python, & got better at vim scripting, though, I started using it less and less; today I barely remember its syntax.

[-] FigMcLargeHuge@sh.itjust.works 19 points 11 months ago

I use awk constantly at work. Very useful in my opinion, and really powerful if you dig into it.

[-] calzone_gigante@lemmy.eco.br 18 points 11 months ago

Yes, for things too complex to do in sed but not complex enough to need a "normal" programming language like python.

[-] corsicanguppy@lemmy.ca 17 points 11 months ago

I use awk on the daily. It has a wider and more consistent install base than perl.

[-] sxan@midwest.social 16 points 11 months ago

Nearly every day. There was a time when I'd reach for Ruby, but in the end, the stability, ubiquity, and portability of the traditional Unix tools - among whom awk is counted - turned out to be more useful. I mainly underuse its power, though; it serves as a column aggregator or re-arranger, for the most part.

[-] OmnislashIsACloudApp@lemmy.world 15 points 11 months ago

I use awk all the time. a very common and probably simplest reason I use it is it's ability to handle variable column locations for data.

if you know you always want the last field you can do something like

awk '{print $NF}'

but usually using it as for performing more advanced operations all in one go without having to pipe something three times.

sure you can do grep cut grep printf, but you can instead do the pattern matching, the field, the formatting, whatever you need to all in one place.

it's also got a bunch of more advanced usage of course since it's its own language. one of my favorite advanced one liners is one that will recognize if it is going to print a duplicate line anywhere in your output and prevent it. in cases where you don't want to sort your output but you also want to remove duplicates it is extremely helpful and quick rather than running post-processing on the output in another way.

all that said main reason I use it is because I know it and it's fast, there's nothing you can do in awk that you can't do in Python or whatever else you're more comfortable with. The best tool for the job is the one that gets it done quickly and accurately. unless your environment is limited and it prevents the installation of tools you're more familiar with then there's no real reason to use this over Python.

[-] Ramin_HAL9001@lemmy.ml 15 points 11 months ago* (last edited 11 months ago)

I used to use the command line, Bash, Awk, Sed, Cut, Grep, and Find (often piped to one another) quite often. I can recall that the few times I used Awk was usually for collating lines from logs or CSV files.

But then I switched to using Emacs as my editor, and it gathers together the functionality of all of those tools into one, nice, neat little bundle of APIs that you can easily program in the Emacs Lisp programming language, either as code or by recording keystrokes as a "macro."

Now I don't use shell pipelines hardly at all anymore. Mostly I run a process, buffer its output, and edit it interactively. I first edit by hand, then record a macro once I know what I want to do, then apply the macro to every line of the buffer. After that, I might save the buffer to a file, or maybe stream it to another process, recapturing its output. This technique is much more interactive, with the ability to undo mistakes, and so it is easier to manipulate data than with Awk and shell pipelines.

[-] netwren@lemmy.world 2 points 11 months ago

This is fascinating to me. Do you have any links or suggestions for this workflow to learn more?

[-] Ramin_HAL9001@lemmy.ml 4 points 11 months ago* (last edited 11 months ago)

This is fascinating to me. Do you have any links or suggestions for this workflow to learn more?

I am glad you asked, because I actually wrote a series of blog posts on the topic of how Emacs replaced my old Tmux+Bash CLI-based workflow. The link there is to the introductory article, in the "contents" section there are links to each of the 4 articles in the series. The "Shell Basics" (titled "Emacs as a Shell") might be of particular interest to you.

If you have any specific questions, or if you have recommendations for something you think you would like to learn from one of my blog posts, please let me know. I would like to write a few more entries in this blog series.

[-] ctr1@fl0w.cc 11 points 11 months ago

Yes! Awk is great, I use it all the time for text processing problems that are beyond the scope of normal filters but aren't worth writing a whole program for. It's pretty versatile, and you can split expressions up and chain them together when they get too complicated. Try piping the output into sh sometime. It can be messy though and my awk programs tend to be write-only

[-] FaceButt9000@lemmy.world 11 points 11 months ago

I use it multiple times a day. I only know basic usage, but it's super useful as part of an awk/grep/sort/uniq pipeline, basically just extracting a field to work on.

[-] meiti@lemmy.world 8 points 11 months ago

Anyone interested in awk make sure to check the just published awk book second revision by original authors. Kernigan's writings are a joy to read.

[-] pelya@lemmy.world 8 points 11 months ago* (last edited 11 months ago)

Grep is fiiiiine.

sed is okay but a little nasty, when your sed script is longer that one search-replace command you gotta ask yourself what you're doing really (yes, sed is a full-featured Turing-complete programming language, if you go far enough into the man page).

When I see awk in any stackoverflow recipe, I just say 'fuck it' and rewrite the whole thing in Python. Python is included into the minimal system image in Debian, the same as awk, but is way less esoteric, and you can do python -e 'import os, sys; commands;' for a one-liner console script.

And if you want to talk about portability, try writing scripts for Android 4.4 ash shell. There's no [ ] command. You do switch/case to compare strings.

[-] netwren@lemmy.world 1 points 11 months ago
[-] pelya@lemmy.world 4 points 11 months ago

No, and I don't think I will learn another tool for something that I can already do using grep/sed/find commands, which I know by heart.

[-] netwren@lemmy.world 1 points 11 months ago

That's fair

[-] olafurp@lemmy.world 8 points 11 months ago* (last edited 11 months ago)

I think it's pretty niche but is a great tool for parsing / converting data into a format that is more easily digested by another program.

Think for example a report from an 80's system that spits out many tab separated values in a different format based on some code. Then these tables are all separated by two blank lines and order of them is randomised. To top that off you need to then pipe it all to a different program that only accepts a specific format.

You could do it in Python by doing a parse, process, stringify code but if you know awk you can do all those steps at the same time with less code.

Sure, in the age of REST the Python approach is better but awk is a very powerful tool for the "I have a specific output but need a specific input" problem.

I use awk a good bit. Not as much as when I was doing data work, though. It's better than cut at splitting on whitespace in many contexts. Lots of other nice uses like variable scope, regex, and summing

[-] AMillionNames@sh.itjust.works 7 points 11 months ago

awk is supposed to be simpler. If it isn't, just use your favorite scripting language. It comes from a period of time when a lot of the scripting languages weren't as easy to use or readily available.

[-] WalrusByte@lemmy.world 7 points 11 months ago

Sometimes I copy and paste an awk command from online, but I can never remember how to write it myself

[-] cybersandwich@lemmy.world 2 points 11 months ago

I'm convinced no one actually knows how to write awk. It's all copy and pasted from the web.

[-] caseyweederman@lemmy.ca 1 points 11 months ago

I am very, very slowly chiseling it into my long-term memory. I feel like Rincwind.

[-] picandocodigo@lemmy.world 6 points 11 months ago

I learned awk when I was studying and I still use it every now and then. It's one of those tools that come in really handy at times. I work in Ruby, but there's still times when scripting or just wanting to process some text when I end up using awk because it makes sense.

[-] jxk@sh.itjust.works 6 points 11 months ago

Awk has the advantage over Perl/Python/etc. that it's standardized by POSIX. Therefore you can rely on it on all operating systems. It's pretty much the only advanced scripting language available that is POSIX -- the alternative would be some heavy shell scripting or almost-unreadable sed.

[-] d3Xt3r@lemmy.nz 2 points 11 months ago

Therefore you can rely on it on all operating systems.

... all except that one OS which we don't like to talk about but annoyingly remains the most popular consumer OS. :P

load more comments (1 replies)
[-] WindowsEnjoyer@sh.itjust.works 6 points 11 months ago

Best use-case of AWK is that you can avoid using grep for picking a Nth word in specific line. I tend to ask GPT4 to write one-liner for me. Works super great.

[-] j4k3@lemmy.world 6 points 11 months ago

With embedded like OpenWRT on a router where you only have busybox/ash shell, awk is your primary expansions tool.

[-] Decker108@lemmy.ml 5 points 11 months ago

I think I've used it once in 15 years or so. It's typically easier to go with bash or Python.

[-] wuphysics87@lemmy.ml 5 points 11 months ago

I use sed a lot

[-] WaLLy3K@infosec.pub 4 points 11 months ago* (last edited 11 months ago)

awk is pretty damn solid. When I was completely rewriting the gravity.sh script from Pi-hole about six years back, it was easily the fastest for parsing and uniquely sorting content from files with a couple million lines. It made things much more usable on Raspberry Pi Zero hardware, since changing to another language like Python was out of the question.

[-] bizdelnick@lemmy.ml 4 points 11 months ago* (last edited 11 months ago)

Yes, but for a very specific case. I used to write highly portable scripts that could be executed in different environments (various linux distros, including minimal containers, freebsd and even solaris 10). I couldn't use bash, perl, python and even gawk. Only POSIX shell (I always tested my scripts with dash and ksh93, for solaris 10 compatibility - with its jsh), portable awk (tested with original-awk, gawk and mawk) and portable sed (better forget it if you need to support solaris).

Before that I didn't understand why should I need awk if I know perl. And awk really sucks. Once I had to replace a perl one-liner with an awk script of ~30 lines for portability.

P.S. I never use awk just for print $1 as many do. It's an overkill.

[-] bionicjoey@lemmy.ca 2 points 11 months ago

P.S. I never use awk just for print $1 as many do. It's an overkill.

cut is better for this use-case IMO. Awk is good for when cut won't cut it.

[-] neidu2@feddit.nl 3 points 11 months ago

Every day. piping srdout to a combination of awk and sed makes shell operations a lot easier. A lot of my earlier perl hacks have now been replaced by a combination of awk, sed, and xargs

[-] SpaceNoodle@lemmy.world 3 points 11 months ago

I haven't found a use for awk that couldn't be accomplished with bash builtins or other common commands.

[-] starman@programming.dev 2 points 11 months ago

No, I don't need it because of jc and nushell

[-] tanakian@lemmy.sdf.org 2 points 11 months ago

awk often can be found in my scripts.

[-] willybe@lemmy.ca 2 points 11 months ago

I used awk to migrate users from one system to another. I created template scripts for setting up the user in the new system, I dumped the data from the old system, then used awk to process the dump and create scripts for each user in the new system. That was a fun project.

[-] ninekeysdown@lemmy.world 2 points 11 months ago

Everyday. I’ve got a lot of stuff that uses it. Granted most of it was mostly created a decade ago but with minimal maintenance it works great. The most helpful script is parsing megacli outputs so I can get a heads up on drive failures and rebuilds among other things.

[-] otl@lemmy.sdf.org 2 points 11 months ago

I use it for my very basic static site generator: https://www.olowe.co/2021/01/site-build.html

[-] ikidd@lemmy.world 2 points 11 months ago

Just had to use it today to turn a key file into a single string with line breaks:

awk 'NF {sub(/\r/, ""); printf "%s\\n",$0;}' id_rsa

[-] visika@lemmy.ml 1 points 11 months ago

I used awk in a physics computer simulations course I had at the university. That's a nice tool to know how to use

[-] allywilson@sopuli.xyz 1 points 11 months ago

I don't tend to use awk in scripts as I do tend to do them in Python, but I do use awk on almost daily basis in one-liners.

Probably the most common thing for me is so I can read a config file without annoying comments and big line spaces.

grep -v "^#" krb5.conf | awk NF

[-] Legisign@lemm.ee 1 points 11 months ago

cut is actually next to useless, because it cannot understand that multiple spaces can still be a single separator in most text files in /etc. You have to use AWK.

load more comments
view more: next ›
this post was submitted on 30 Dec 2023
145 points (96.2% liked)

Linux

48653 readers
389 users here now

From Wikipedia, the free encyclopedia

Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).

Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.

Rules

Related Communities

Community icon by Alpár-Etele Méder, licensed under CC BY 3.0

founded 5 years ago
MODERATORS