Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Author Topic: bash sed and awk  (Read 2125 times)

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
bash sed and awk
« on: April 17, 2018, 07:07:09 AM »

Am having fun learning bash sed and awk.
Have spent 30 years writing very complicated .bat then .cmd file's first for dos then for Win NT CMD, so am having to convert my brain over for sh and bash.  I did a course on youtube on sh and unfortunately learnt a ton of very out of date stuff, the course was from c.2002, so am now realising the error of my ways and trying to get my mind right with bash sane replacement techniques. God what a mess it all is, although powerful in some respects.

I have written my first genuinely useful bash tool. A kind of ping program which in addition works out what ipv6 and ipv4 addresses some target box has by looking something up in the dns.

I wrote my very first tiny awk program today, simply to optionally help with reformatting textual output in a cleaner appearance, to be used by the aforementioned bash tool. Awk is nice. Not yet sure when to use it and when to stay away.

Not got very far with sed yet - bit confused by the apparent relationship between the presence or absence of the command line switches and requirements for escaping certain characters in regexes for reasons unknown to me. Different regex dialects are a pain too. Have got too used to expecting PCRE dialect to always be available. Need to do a lot more reading and dig out some meatier examples. A lot of sed tutorials I've stumbled upon get bogged down in explaining regexes, which I don't need, have been doing regexes for 30 yrs, and don't get on with the real actual sed stuff.

Some of this stuff is so sophisticated that it starts to make me wonder if I would just be getting seduced by madness and should just be writing a C program to do some job or other instead and be done with it. (The D programming language [google dlang] is my new love affair, so it would probably be D instead of C now.)

Having destroyed my raspberry pi yet again by making config mistakes that mean that ssh into it no longer works, I have given up in embarrassment for the time being.

Instead, I have rented a pi from Mythic Beasts which is physically parked somewhere near docklands I imagine. I have ipv6 (only, I think, sort-of) ssh access in to the box as root and it is running very stripped down 32-bit arm Ubuntu (no 64-bit arm AAarch64 builds yet from these guys unfortunately, great shame). The o/s comes perfectly lean for headless-only use, ie no graphics, human input devices etc. There is a choice of three different linux flavours, I picked Ubuntu at random. I did try one of the other flavours as well, one which I believe is most commonly used by pi users. For some reason I seem to have settled on Ubuntu, have no idea what the practical differences are, maybe not much especially if you are headless.

I have failed so far to get firewalling set up and I suspect there isn't any at the moment. Attempts to do so and to secure ssh slightly better simply mean I ended up locking myself out umpteen times so that I had to nuke the installation and format the disk. Luckily one can just hit a button and remotely format the disk and reinstall an o/s image of your choice in a few seconds by using Mythic Beasts’ clever remote admin tools, and I have done that a lot. :-( :-) That’s one huge advantage over owning my own pi, which I cannot nuke and reinstall without help. God knows how they do it and god only knows what it is booting from.

Fun learning bash etc on the system anyway.

I have managed to apt-get two D compilers - GDC (GCC family) and LDC (LLVM family) which are both superb. Unfortunately the packages that are available pre-built and all packaged up are quite old versions. I would love to get the latest releases built for arm, but trying to download and compile them from sources has proved a nightmare of twisty passages as I don't possess a complete shell script to do the whole job and I got lost in very long confusing instructions and was not sure exactly where some files needed to be parked after I downloaded huge dollops of stuff from amazingly slow servers on dialup or something.

I would like to think up some good things that I can do with a hosted external pi, to give it some work to do. Any suggestions?

I would also like to give it a security check up. I want make sure it can't be exploited by bad guys.  Being effectively ipv6-only helps a little bit as things are at the moment.

I am trying to get a mail server, set up it. Chose postfix somehow. Think the config is done but it needs a lot of proper testing, which I have not even started yet. I want to smtp some outbound mail into the box and then forward it onwards to AA's outbound SMTP servers. The reason is that I have some iot devices that produce email for log reports and are quite old-fashioned and cannot handle AA's recent requirements for well-secured best practice email transport. To be sure of doing the right thing, I need the email to ultimately go through AA’s outbound SMTP servers because that is what my own SPF policy rules declare. So the idea is that the pi can pick up outbound SMTP, but only from a tight range of src ip  addresses (if I have done the job right) and send everything on to AA if I have got the mail server config correct, following instructions from an article on this very topic which I found on the web.

I could do with someone trying to test the security of the mail setup, hopefully would fail to get mail injected into it from src addresses that are disallowed. I could do some tests myself using a temporarily tweaked config. A volunteer would need ipv6.

I also wonder about other kinds of abuse possibilities, whether the box could be used in traffic amplification, given the presumed lack of firewalling. Like almost all Pis, it only has a miserable 100Mbps nic, so it isn't going to be able to take down Facebook on its own. (I'm told that you can do quite a good bit better than 100mbps network i/o by using certain USB nic hardware, but the internal horrible nasty bus is the ultimate limiting factor that cannot be surmounted and gigabit nics are just throttled by that.)
Logged

vic0239

  • Reg Member
  • ***
  • Posts: 519
Re: bash sed and awk
« Reply #1 on: April 17, 2018, 07:49:33 AM »

I have failed so far to get firewalling set up and I suspect there isn't any at the moment.
Take a look at UFW. I recently set this up to restrict certain ports on my two Pi systems.
Logged
Lothian Broadband 900/900 + AAISP VDSL, Vigor2865Vac, MikroTik rb260gsp, ZyXel NWA50AX WiFi AP.

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: bash sed and awk
« Reply #2 on: April 17, 2018, 09:59:28 AM »

docs for UFW look good, straightforward non nonsense - many thanks for that tip.

Unfortunately I simply killed the box trying to set up ufw over ssh. Despite adding an exception hole to allow my client box access by src ip, I must have been doing something wrong. it could be that it is promoting me for a y/n when i can no longer respond. I did a
    ufw  enable ; ufw allow from 2001:8b0:xxx::/48

but if it stalled in the middle of that then I am dead, and I cannot see  a -y type option mentioned. I also tried piping a y into it, twice. but who knows what I did. I need something less drastic than a default with all input blocked clearly.
Logged

vic0239

  • Reg Member
  • ***
  • Posts: 519
Re: bash sed and awk
« Reply #3 on: April 17, 2018, 11:02:18 AM »

I don't recall having a confirm prompt inputting the rules, just when deleting them.

You can also set all the rules before enabling UFW.

I also employed a backdoor just in case:
Code: [Select]
$ apt-get install -y screen
$ screen -S firewall
$ while true; do sudo ufw allow from 192.168.1.0/24; ufw enable --force-enable; sleep 60;done
 (disconnect from the screen session by type in "ctrl+a d")
Logged
Lothian Broadband 900/900 + AAISP VDSL, Vigor2865Vac, MikroTik rb260gsp, ZyXel NWA50AX WiFi AP.

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: bash sed and awk
« Reply #4 on: April 17, 2018, 10:44:08 PM »

> You can also set all the rules before enabling UFW.

In that case, I did things the wrong way round then, should have done it as you say.

Don't understand why it failed though. I put the ‘allow from’ ipv6 with /48 on the same line, semicolon-separated. I thought later, what if I had been using IPv4 for SSH at the time? I didn't check that it actually was using IPv6 for SSH right at that moment, but as far as I'm aware the domain name I used to refer to it only maps to IPv6.

Perhaps I needed the force-enable switch as you had it in order to prevent it from stalling on a user prompt, that would explain what killed me.
Logged

adrianw

  • Reg Member
  • ***
  • Posts: 163
Re: bash sed and awk
« Reply #5 on: April 18, 2018, 02:27:01 AM »

How long until you move on to the Swiss Army Chainsaw of shell programming - Perl?  :yum:

It has a learning curve like a vertical cliff, but is very  powerful and need not be cryptic.
Logged

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: bash sed and awk
« Reply #6 on: April 19, 2018, 01:02:58 AM »

I hear you about perl. At that point I may have adopted awk, which seems like php or C but with strings everywhere. Or given up on all scripting languages and gone back to C. Or rather D in the light of my new love-affair. D now has a posh compile-time regex library if I remember correctly, which makes use of Ds unique ability to be able to simply evaluate most functions at compile time whenever it makes sense to do so (so called CTFE - compile-time function evaluation) thus making their code disappear altogether, unlike C which is very limited in that respect.

I have also had a massive love affair with XSLT, but only any use provided your stuff is in XML in the first place. I wrote some gigantic things in pipelined multi-Xslt, with about ten xslt transformations pipelined within the msxml engine in internal tree format. Used that to generate web pages ultimately.
Logged

displaced

  • Reg Member
  • ***
  • Posts: 270
Re: bash sed and awk
« Reply #7 on: April 19, 2018, 12:46:00 PM »

Ah, sed, awk and shell escaping rules. 

"Unterminated string literal?  What the hell!  Look, the double-quote is right there!  THERE! *stabs a finger at the screen*

...*waits*... *calms down and re-reads expression*

Oh, wait..."

Good times, good times...

These days I tend to do any utility scripting in Python, which is a language I've really come to love. 

I'm one of two 'generalist' developers in a company full of SQL people.  If I'm hit by a bus, the code needs to be understandable by someone who isn't a general-purpose programmer.  I ran a few basic Python training courses with the team to get them to a point where they can read Python and understand what it's doing.

And, it's ubiquitous.  At least Python2.6 is available by default everywhere apart from Windows.  And for Windows, it's a quick, self-contained installation that doesn't fiddle with the system.

Everything I would've done with bash/cmd scripts I do with Python these days and am happier for it :)
Logged
YouFibre 1Gbit, OPNsense on Intel N100.  Ubiquiti UAPs.

Weaver

  • Senior Kitizen
  • ******
  • Posts: 11459
  • Retd s/w dev; A&A; 4x7km ADSL2 lines; Firebrick
Re: bash sed and awk
« Reply #8 on: April 19, 2018, 09:01:03 PM »

Some good advanced sed tutorial urls would be handy.

Btw - What if you want to process utf8 or utf16le, any updated replacement tools out there that think in 32-bit unicode code points internally? I presume that much of the Unix world is still stuck in the ASCII 1970s which is why UTF-8 is so very valuable, am I correct?
Logged