Linux Basic Commands- String formating commands(cut,awk, sed,wc,head, tail,more)

TjMan 05/Jun/2019 Shell Scripting

CUT:

It is very useful for string formating in vertical faction. We can filter value of each row based there position on line. Input data:

[tjman]$ cat example.txt
1,a,b,c
2,d,e,f
3,g,h,i
4,j,k,l

Take on first character:

[tjman]$ cut -c 1 example.txt
1
2
3
4

Take on 3rd character:

[tjman]$ cut -c 3 example.txt
a
d
g
j

Take range of characters:

[tjman]$ cut -c 1-3 example.txt
1,a
2,d
3,g
4,j

Take nTH field using -f by specifing delimiter using -d:

[tjman]$ cut -d ',' -f 1 example.txt
1
2
3
4
[tjman]$ cut -d ',' -f 3 example.txt
b
e
h
k
[tjman]$ cut -d ',' -f 1,3,4 example.txt
1,b,c
2,e,f
3,h,i
4,k,l

Change output delimiter of cut command:

[tjman]$ cut -d ',' -f 1,3,4 --output-delimiter '~' example.txt
1~b~c
2~e~f
3~h~i
4~k~l

AWK:

AWK is a scripting languare for text processing in Linux. Actual structure of awk program is consist of three blocks as BEGIN,ACTION,END. It is not necessary to have all blocks every time. Check some useful examples below.

Input data:

[tjman@www ~]$ cat TextInput.txt
1 tunetotech.com technicalBlogging
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork

[tjman@www ~]$ awk 'BEGIN {printf "---|BEGIN|--\n"} {print $0} END {printf "---|END|---\n"}' TextInput.txt
---|BEGIN|--
1 tunetotech.com technicalBlogging
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork
---|END|---

Print every thing without modification:

[tjman@www ~]$ awk '{print}' TextInput.txt
1 tunetotech.com technicalBlogging
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork

Print those lines which contains the matching text:

[tjman@www ~]$ awk '/Network/{print}' TextInput.txt
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork

Print specific fields: $0 Entire lines $1 First field $2 Second field $n Nth field

[tjman@www ~]$ awk '{print $2 $3}' TextInput.txt
tunetotech.comtechnicalBlogging
google.comsearchEngin
facebook.comsocialNetwork
linkedin.comprofessionalNetwork

NR(number of rows) variable used to know the line number.

[tjman@www ~]$ awk '{print NR,$2,$3}' TextInput.txt
1 tunetotech.com technicalBlogging
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork

NR variable can be used to print specific lines: First NR is the starting point and second NR is the ending point. All lines between these will be printed.

[tjman@www ~]$ awk 'NR==1{print $0}' TextInput.txt
1 tunetotech.com technicalBlogging

[tjman@www ~]$ awk 'NR==2,NR==4{print $0}' TextInput.txt
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork

NF(numner of fields) variable used to know total number of fields and can be used to print those fileds. As an example to print last field we can do as below:

[tjman@www ~]$ awk '{print $NF}' TextInput.txt
technicalBlogging
searchEngin
socialNetwork
professionalNetwork

Check the number of fields in each line:

[tjman@www ~]$ cat TextInput_V1.txt

1 tunetotech.com technicalBlogging
google.com searchEngin
3 facebook.com socialNetwork
linkedin.com professionalNetwork

[tjman@www ~]$ awk '{print NF}' TextInput_V1.txt
3
2
3
2

Adding text in between:

[tjman@www ~]$ awk '{print NR "------>" $0}' TextInput.txt
1------>1 tunetotech.com technicalBlogging
2------>2 google.com searchEngin
3------>3 facebook.com socialNetwork
4------>4 linkedin.com professionalNetwork

Find largest length of line in a file:

[tjman@www ~]$ awk '{if (length($0)>length_max) length_max = length($0)} END {print length_max}' TextInput.txt
34

Check number of lines in a file:

[tjman@www ~]$ awk 'END{print NR}' TextInput.txt
4

You can put your logic on a awk script and use using -f command.

[tjman@www ~]$ cat my_awk_program.awk
BEGIN {printf "---|BEGIN|--\n"}
{print $0}
END {printf "---|END|---\n"}

[tjman@www ~]$ awk -f my_awk_program.awk  TextInput.txt
---|BEGIN|--
1 tunetotech.com technicalBlogging
2 google.com searchEngin
3 facebook.com socialNetwork
4 linkedin.com professionalNetwork
---|END|---

Custom file seperator:

If you want to process a file having file seperator other than space then -F is the option to specify the custom file seperator.

[tjman@www ~]$ cat TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
[tjman@www ~]$ awk '{print NF}' TextInput.txt
1
1
1
1
[tjman@www ~]$ awk -F ',' '{print NF}' TextInput.txt
3
3
3
3
[tjman@www ~]$ awk -F ',' '{print $0}' TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork

SED:

option ‘n’ suppresses printing of whole file & option ‘p’ will print only line lines from 22 to 29.

[tjman@www ~]$ sed -n 3,5p TextInput.txt
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing

[tjman@www ~]$ sed -n 5p TextInput.txt
5,youtube.com,videoShearing

‘d’ will remove the mentioned lines from output.

[tjman@www ~]$ sed 5d TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

[tjman@www ~]$ sed '5!d' TextInput.txt
5,youtube.com,videoShearing


[tjman@www ~]$ sed 3,5d TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

Add blank line after each non blank line.

[tjman@www ~]$ sed G TextInput.txt
1,tunetotech.com,technicalBlogging

2,google.com,searchEngin

3,facebook.com,socialNetwork

4,linkedin.com,professionalNetwork

5,youtube.com,videoShearing

6,amazon.com,shopping

7,wikipedia.org,encyclopedia

8,quora.com,questionAnswer

s for search and g for replace. Replace text:

[tjman@www ~]$ sed 's/Network/Site/g' TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialSite
4,linkedin.com,professionalSite
5,youtube.com,videoShearing
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

Replace text in line range(i ignore case):

[tjman@www ~]$ sed '1,3 s/com/in/gi' TextInput.txt
1,tunetotech.in,technicalBlogging
2,google.in,searchEngin
3,facebook.in,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

a to add line after word detected.

[tjman@www ~]$ sed '/org/a "ORG domain detected"' TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
"ORG domain detected"
8,quora.com,questionAnswer

i to add line before word detected.

[tjman@www ~]$ sed '/org/i "ORG domain detected"' TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing
6,amazon.com,shopping
"ORG domain detected"
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

c to change line with alternate text.

[tjman@www ~]$ sed '/org/c "ORG domain detected--Removed"' TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing
6,amazon.com,shopping
"ORG domain detected--Removed"
8,quora.com,questionAnswer

WC (Count of word,line and characters):

-l: number of line in a file

[tjman@www ~]$ wc -l TextInput.txt
8 TextInput.txt
[tjman@www ~]$ cat TextInput.txt |wc -l
8

-L: maximum length of a line on specified file

[tjman@www ~]$ cat TextInput.txt |wc -L
34

-w: number of word in a file

[tjman@www ~]$ cat TextInput.txt |wc -w
8

-m: number of character in a file

[tjman@www ~]$ cat TextInput.txt |wc -m
230

HEAD:

See lines from top of a file. Specify the number of lines you want to see. Here it will show from top to 5th line.

[tjman@www ~]$ head -5 TextInput.txt
1,tunetotech.com,technicalBlogging
2,google.com,searchEngin
3,facebook.com,socialNetwork
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing

See lines from buttom of a file. Specify the number of lines you want to see. Here it will show buttom five lines.

TAIL:

[tjman@www ~]$ tail -5 TextInput.txt
4,linkedin.com,professionalNetwork
5,youtube.com,videoShearing
6,amazon.com,shopping
7,wikipedia.org,encyclopedia
8,quora.com,questionAnswer

-f: Can be used to see rolling files. New lines will be visible as soon as it is written. Very useful to see run time logs of any application.

more command can be used to see file in dynamic scrolling faction.

Previous Post Linux Basic commands-chmod, wild cards,zip,df, du,free,top, cal,hostname, uname,sleep,ps