6 Automating Tasks with Scripts

Chapter 6
Automating Tasks with Scripts

6.1 Prerequisites
  6.1.1 Assumed Knowledge
  6.1.2 Linux and Network Setup
6.2 Introduction to Scripts
  6.2.1 Shell Scripts are Text Files
  6.2.2 Variables in Scripts
  6.2.3 For Loops
  6.2.4 If/Then/Else
  6.2.5 Input Parameters
  6.2.6 Executing Shell Scripts
6.3 More Scripting Examples
  6.3.1 First Script with echo and ls
  6.3.2 Using Variables
  6.3.3 For Loops
  6.3.4 If/Then/Else
  6.3.5 Input Arguments
  6.3.6 Reading a Text File
  6.3.7 Extra Commands

File: nsl/scripts.tex, r1670

This chapter introduces you to writing scripts that automate tasks with the Linux the command line. While you do not need to write scripts to complete tasks in most of the remaining chapters, being able to create simple scripts will simplify tasks when frequently using the command line. You may skip this chapter and return to it if (when?) you say to yourself: “I am running the same commands over and over—isn’t there a more efficient way to do this?”.

6.1 Prerequisites

6.1.1 Assumed Knowledge

This chapter assumes you have knowledge of:

Linux command line and common commands, as covered in Chapter 4.
Basic programming concepts, including variables, if/then/else statements, for and while loops, and functions.

6.1.2 Linux and Network Setup

All of the practical tasks in this chapter can be completed on a single Linux computer. Although virtnet (Chapter 3) is not required, if you do use it, as only a single computer is necessary, topology 1 is appropriate (or in fact any topology—just use a single node).

6.2 Introduction to Scripts

The shell is the software that interprets the commands you type in on a terminal. It is a program itself, and there are many different implementations: sh (the original), Bash, Csh, Tcsh, Zsh, Dash, Ksh, … . Bash is very common today and is the default on Ubuntu Linux and Mac OSX, and therefore we will focus on that.

The shell defines how you interact with the operating system on the terminal. The most common interaction is simply typing the name of an application or command, followed by optional parameters. The shell then executes that application. However a shell has much more, including features that allow you to combine multiple commands to complete more complex tasks than what a single application can do on its own. For convenience, rather than typing a set of commands on the terminal, they are usually included in a file, and then the shell executes that file. Such a file is called a shell script.

This chapter is a very quick introduction to shell scripting, first covering the concepts with short examples, and then presenting some longer examples in Section 6.3. There are many sources that explain shell scripting, including:

man bash or info bash (also available online)
Bash Reference Manual
Bash Beginners Guide
Introduction to Bash Programming
Advanced Bash Scripting

6.2.1 Shell Scripts are Text Files

Let’s create a first shell script. Use any text editor (e.g. nano, vi, emacs) to create a file containing your commands. Below is an example with the file called script-example1.sh contain just two lines.

$ cat script-example1.sh
#!/bin/bash
ls -l ~/

The script script-example1.sh is just a text file with two lines. The first line is a special line that indicates to the shell what interpreter (shell) should be used to execute the following commands. Although it is not necessary, it is good practice to include such a line. Note that later we will see everything after a # (hash) is a comment; however this is a special case where the first two characters of the file are #! (shebang), which means its not actually a comment.

The 2nd line of script-example1.sh is the only command to execute in this script: list in long format the files in my home directory.

You can execute the script by passing its name as a parameter to bash. As a result the commands inside the file are executed.

$ bash script-example1.sh
total 12
-rw-rw-r-- 1 network network  174 Mar  2  2017 lynx.cfg
-rw-rw-r-- 1 network network   21 Jan 14 17:44 script-example1.sh
drwxrwxr-x 6 network network 4096 Feb 10  2017 virtnet

Note that the output from the above demo may differ from your computer (as you may have different files).

6.2.2 Variables in Scripts

Variables can be used in shell scripts as demonstrated in script-example2.sh. You refer to the value by preceding the variable name with a $ (dollar sign). Optionally, you may enclose the variable name in {} (braces). Everything after a # (hash) is a comment and is not executed.

$ cat script-example2.sh
                                                                                      
                                                                                      
#!/bin/bash
myname="Steven Gordon"
# Variable names can be enclosed in braces { }
echo ${myname}
echo "My name is $myname" # or optionally the braces can be omitted
# It is good practice to include the braces
$ bash script-example2.sh
Steven Gordon
My name is Steven Gordon

6.2.3 For Loops

For loops can loop across numbers, using C-like syntax, as well as loop across lists, including lines in a file. Some examples:

$ cat script-data1.txt
123,456,abc
789,012,def
345,678,ghi
$ cat script-example3.sh
#!/bin/bash
for ((i=1; i<=3; i++));
do
     echo $i
done

for name in Steve John Lily;
do
     echo ${name}
done

for line in `cat script-data1.txt`;
do
     echo ${line} | cut -d "," -f 2
done
$ bash script-example3.sh
1
2
3
Steve
John
Lily
456
012
                                                                                      
                                                                                      
678

6.2.4 If/Then/Else

Conditional statements are possible using if/then/else style. The hardest part is the testing of conditions. This is normally done using the test command, which has a short form of enclosing the conditional statement in [] (square brackets). See man test to see the syntax for different conditions.

$ cat script-example4.sh
#!/bin/bash
cutoff=2
for ((i=1; i<=3; i++));
do
     if [ $i -lt $cutoff ];
     then
         echo "$i is less than $cutoff"
     elif [ $i -eq $cutoff ];
     then
         echo "$i is is equal to $cutoff"
     else
         echo "$i is not less than $cutoff"
     fi
done

for name in Steve John Lily;
do
     if [ "$name" = "Lily" ];
     then
         echo "$name is the boss"
     fi
done

filename="script-data1.txt";
if [ -e ${filename} ];
then
     echo "${filename} exists"
fi
$ bash script-example4.sh
1 is less than 2
2 is is equal to 2
3 is not less than 2
Lily is the boss
script-data1.txt exists

6.2.5 Input Parameters

A script can take input arguments/parameters, in the same way most commands do. These are called positional parameters and referred to using a number of the position listed on the command line, e.g. $1 is the first parameter, $2 is the second parameter, …

$ cat script-example5.sh
#!/bin/bash
ls -l $1 | grep $2
$ bash script-example5.sh /usr/bin sum
-rwxr-xr-x 1 root   root      30460 Feb 18  2016 cksum
-rwxr-xr-x 1 root   root    3956356 Jan 19  2017 innochecksum
-rwxr-xr-x 1 root   root      42780 Feb 18  2016 md5sum
lrwxrwxrwx 1 root   root          6 Feb 10  2017 md5sum.textutils -> md5sum
-rwxr-xr-x 1 root   root      42780 Feb 18  2016 sha1sum
-rwxr-xr-x 1 root   root      50972 Feb 18  2016 sha224sum
-rwxr-xr-x 1 root   root      50972 Feb 18  2016 sha256sum
-rwxr-xr-x 1 root   root      87836 Feb 18  2016 sha384sum
-rwxr-xr-x 1 root   root      87836 Feb 18  2016 sha512sum
-rwxr-xr-x 1 root   root       9332 Mar 13  2016 shasum
-rwxr-xr-x 1 root   root      42784 Feb 18  2016 sum

6.2.6 Executing Shell Scripts

So far we have executed the shell scripts by passing the file name as a parameter to bash. Another way is to make the script file executable (Chapter 7 explains the chmod command and permissions):

$ chmod u+x script-example1.sh

And now you can run the script like other programs:

 ./script-example1.sh
total 32
-rw-rw-r-- 1 network network  174 Mar  2  2017 lynx.cfg
-rw-rw-r-- 1 network network   36 Jan 14 17:51 script-data1.txt
-rwxrw-r-- 1 network network   21 Jan 14 17:44 script-example1.sh
-rw-rw-r-- 1 network network  209 Jan 14 17:50 script-example2.sh
                                                                                      
                                                                                      
-rw-rw-r-- 1 network network  191 Jan 14 17:53 script-example3.sh
-rw-rw-r-- 1 network network  473 Jan 14 17:55 script-example4.sh
-rw-rw-r-- 1 network network   31 Jan 14 17:58 script-example5.sh
drwxrwxr-x 6 network network 4096 Feb 10  2017 virtnet

But we need to include “./” in front of the name to tell the shell that the command/program script-example1.sh can be found in “this” directory. If you want to avoid including “./” then the directory that stores the script must by in the PATH environment variable. Let’s say we create a directory that contains all our scripts (/home/network/scripts) and move them into that directory (This is just an example; it is probably better to use the directory /home/network/bin to store your scripts and applications). Let’s also make them executable.

$ mkdir scripts
$ mv script-example*.sh scripts/
$ chmod u+x scripts/*
$ ls -l scripts/
total 20
-rwxrw-r-- 1 network network  21 Jan 14 17:44 script-example1.sh
-rwxrw-r-- 1 network network 209 Jan 14 17:50 script-example2.sh
-rwxrw-r-- 1 network network 191 Jan 14 17:53 script-example3.sh
-rwxrw-r-- 1 network network 473 Jan 14 17:55 script-example4.sh
-rwxrw-r-- 1 network network  31 Jan 14 17:58 script-example5.sh

Now lets add our directory to the PATH environment variable. First we show the current PATH, and then add our directory to it:

$ echo $PATH
/home/network/bin:/home/network/.local/bin:/usr/local/sbin:/usr/local/bin:/usr
/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/network/virtnet/bin
$ PATH=/home/network/scripts:$PATH
$ echo $PATH
/home/network/scripts:/home/network/bin:/home/network/.local/bin:/usr/local/sbin
:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/ho
me/network/virtnet/bin

Now we can execute our scripts from any directory by just typing the name.

$ script-example1.sh
total 16
-rw-rw-r-- 1 network network  174 Mar  2  2017 lynx.cfg
-rw-rw-r-- 1 network network   36 Jan 14 17:51 script-data1.txt
drwxrwxr-x 2 network network 4096 Jan 14 19:02 scripts
drwxrwxr-x 6 network network 4096 Feb 10  2017 virtnet

But be careful: some of the example scripts above referred to relative files (e.g. data1.txt), so may longer work. Try to fix them.

6.3 More Scripting Examples

The previous section introduced some basic concepts of scripting, with short examples. Here we present several more example scripts. You may use them, along with other online resources, to learn basics of shell scripts. You should run the script, and then inspect the source code, which include comments, to understand the output produced.

demo_start.sh: very simple script that shows how to get started using echo and ls.
demo_variable.sh: several examples creating and using variables.
demo_for.sh: different approaches for creating for control loops.
demo_if.sh: if-then-else statements using test.
demo_arguments.sh: reading and using command line arguments to your script
demo_readfile.sh: create a temporary text file with content, and then read that file in line by line.
demo_extras.sh: a collection of other useful commands.

The following sections show the example scripts. You can also download the individual files or a zip or tgz archive containing all the source files.

While example output of the scripts is shown, note that the output on your computer may differ (e.g. different set of files, different times).

6.3.1 First Script with echo and ls

A very simple script that shows how to get started using echo and ls.

#!/bin/bash
# Lines starting with a hash (#) are comments EXCEPT the first line above.
# The first line tells us what shell to use to run this script. You should
# include it in every script file.

# Scripts contain commands as you would execute on the command line
echo "Listing␣all␣files␣in␣reverse␣time␣order"

ls -lrta

echo "End␣of␣listing"