Bash.md

---
marp: true
title: Introduction to Bash Scripting
author: P.Y. Barriat
description: Introduction to Bash Scripting
backgroundImage: url('assets/back.png')
_backgroundImage: url('assets/garde.png')
footer: 17/10/2023 | Introduction to Bash Scripting
_footer: ""
paginate: true
_paginate: false
---

Introduction to Bash Scripting<!--fit-->
===

https://forge.uclouvain.be/barriat/learning-bash

![h:200](assets/bash_logo.png)

##### October 17, 2023

###### CISM/CÉCI Training Sessions


---

# Linux command line

A Linux terminal is where you enter Linux commands

It's called the **C**ommand **L**ine **U**ser **I**nterface

**CLUI** is one of the many strengths of Linux :

- allows to be independent of distros (or UNIX systems like OSX)
- allows to easily work at distance (SSH)
- allows to join together simple (and less simple) commands to do complex things and automate **= scripting**

In Linux, process automation relies heavily on scripting. This involves creating a file containing a series of commands that can be executed together

---

# Linux Shell

A **shell** is a program that takes commands from the keyboard and gives them to the operating system to perform

The main function is to interpret your commands **= language**

Shells have some built-in commands

A shell also supports programming constructs, allowing complex commands to be built from smaller parts **= scripts**

Scripts can be saved as files to become new commands
> many commands on a typical Linux system are scripts

---

# Bash 

The **Bash** shell is one of several shells available for Linux

It is the default command interpreter on most GNU/Linux systems. The name is an acronym for the "**B**ourne-**A**gain **SH**ell"

### Bash Scripting Demo

```bash
#!/bin/bash

# declare STRING variable
STRING="Hello World"

# print variable on a screen
echo $STRING
```

---

![h:600](assets/scripting.png)

---

# Bash environment

In a Bash shell many things constitute your environment

- the form of your 'prompt' (what comes left of your commands)
- your home directory and your working directory
- the name of your shell
- functions that you have defined
- etc.

Environment includes many variables that may have been set **by bash** or **by you**

**Access the value of a variable by prefixing its name with `$`**

---

# Environment variables

|  Variables    |         |
| ------------ | --------------- |
|     `USER`    |  the name of the logged-in user |
|     `HOME`    | the user's home directory (similar to `~` ) |
|     `PWD`    |  the current working directory |
|     `SHELL`    |  the name of the shell |
|     `UID`    |  the numeric user id of the logged-in user |

So to get the value of `USER` you would use `$USER` in bash code

> You can use special files to control bash variables : `$HOME/.bashrc`


---

# Bash Scripting basics

By naming convention, bash scripts end with `.sh`
> however, bash scripts can run perfectly fine without any extension

A good practice is to define a `shebang` : first line of the script, `shebang` is simply an absolute path to the shell interpreter (see `echo $SHELL` result)
> combination of `bash #` and `bang !`

The usual shebang for bash is `#!/bin/bash`



---

### Comments start with `#`

On a line, any characters after `#` will be ignored  (with the exception of `#!`)

```bash 
echo "A comment will follow." # Comment here.
#                            ^ Note whitespace before #
```

### There is no standard indentation

- Pick a standard in your team that you can all work to
- Use something your editor makes easy (**Vim** uses `Tab`)

---
<!-- JDF: je pnseque c'est trop tôt pour ceci
### Command separators

Commands can be combined using **meta-characters** and **control operators**

```bash 
# cmd1; cmd2
$ cd myfolder; ls   # no matter cd to myfolder successfully, run ls

# cmd1 && cmd2
$ cd myfolder && ls  # run ls only after cd to myfolder

# cmd1 || cmd2
$ cd myfolder || ls  # if failed cd to myfolder, `ls` will run
```

---
-->

# Permissions and execution

- Bash script is nothing else just a **text file** containing instructions to be executed sequentially
  > by default in Linux, a new text file's permissons are **-rw-r--r--** (or 644)
- You can run the script `hello_world.sh` using 
  * `sh hello_world.sh`
  * `bash hello_world.sh`
  * `chmod u+x run_all.sh` then `./hello_world.sh`
    > after the `chmod`, you file is **-rwxr--r--** (or 744)

---

# Hands-on exercise

Your first bash script:

1. create a folder `bash_exercises` and go there
2. use your favourite editor (vim, obviously) to create a new file called `exercise_1.sh`
3. write some code in it to display the current working directory as:
   > The current directory is : /home/me/bash_exercises
4. make the file executable
5. run it !

---

# Variables and data types in Bash

Variables let you store data : **numeric values** or **character(s)**

You can use variables to read, access, and manipulate data throughout your script

**You don't specify data types in Bash**

- assign directly : `greeting="Welcome"` or `a=4`
- assign based on variable: `b=$a`

And then access using `$`: `echo $greeting`

**!!!** no space before or after `=` in the assignation **!!!**
`myvar = "Hello World"` :boom:

---

### Quotes for character(s) `" '`

Double will do **variable substitution**, single will not:

```
$ echo "my home is $HOME" 
my home is /home/me
$ echo 'my home is $HOME'
my home is $HOME
```


### Command Substitution

```bash 
#!/bin/bash
# Save the output of a command into a variable
myvar=$( ls )
```

---

## Variable naming conventions

* Variable names **should start** with a letter or an underscore

* Variable names can contain letters, numbers, and underscores

* Variable names are **case-sensitive**

* Variable names **should not** contain spaces or **special characters**

* Use descriptive names that reflect the purpose of the variable

* Avoid using **reserved keywords**, such as `if`, `then`, `else`, `fi`, and so on...

* **Never** name your private variables using only **UPPERCASE** characters to avoid conflicts with builtins

---

## String manipulation

Consider `string=abcABC123ABCabc` and `filename=myfile.txt`

* string length : `${#string}` is `15`
* substring extraction : 
  * `${string:7}` is `23ABCabc`
  * `${string:7:3}` is `23A`
  * `${string:(-4)}` or `${string: -4}` is `Cabc` 
* substring removal from front :
  * `${string#a*C}` is `123ABCabc`
  * `${filename##myfile}` is `.txt`

---

* substring removal from back :
  * `${string%b*c}` is `abcABC123ABCa`
  * `${filename%%.txt}` is `myfile`

## Variable expansion

* `${variable-default}` : 
  if `variable` is unset or null, the expansion of `default` is substituted
* `${variable+default}` : 
  if `variable` is unset or null, nothing is substituted, otherwise the expansion of  `default` is substituted

---

# Arithmetic

|  Operator    |     Operation   |
| ------------ | --------------- |
|  `+` `-` `\*` `/` |  addition, subtraction, multiply, divide |
|  `var++` | increase the variable var by 1 |
|  `var--`    |  decrease the variable var by 1 |
|  `%`    |  modulus (remainder after division) |

Several ways to go about arithmetic in Bash scripting :

  `let`, `expr` or using **double parentheses**

<!-- JDF: probablement pas la bonne place pour ça
Return the length of a variable : `${#var}`
-->

<!-- JDF: a mon avis pas utile
---

- `let` make a variable equal to an expression

```bash 
#!/bin/bash
let a=5+4 
echo $a # 9
let a++
let "a = 4 * $a"
```

- `expr` : print out the result of the expression

```bash 
#!/bin/bash
expr 5 + 4   # 9
expr 5+4     # boom
expr "5 + 4" # boom
a=$( expr 10 - 3 )
```

- double parentheses : return the result of the expression
  > it is the **preferred method**
-->
---

```bash 
#!/bin/bash

a=$(( 4 * 5 ))
a=$(( 4 + 5 ))
a=$((3+5))

b=$(( a + 3 ))
echo $b # 11

b=$(( $a + 4 ))
echo $b # 12

(( b++ ))
(( b += 3 ))
echo $b # 16
```

---

# Conditional statements

Use:
* `if condition; then` to start conditional block
* `else` to start alternative block
* `elif` to start alternative conition block
* `fi` to close conditional block

The following operaors can be used beween conditions:

* `||` means **OR**
* `&&` mean **AND** 

---

# Conditional exemple

(Don't pay too much attention to the details yet)

```bash 
#!/bin/bash
echo "Please enter a number: "
read num

if [ $num -gt 5 ] && [ $num -le 7 ]
then
  echo "$num is 6 or 7"
elif [ $num -lt 0 ] || [ $num -eq 0 ]; then
  echo "$num is negative or zero"
else
  echo "$num is positive (but not 6, 7 or zero)"
fi
```

Read the **standard input stream** with the `read` command

---

|  Operator    |     Description   |
| ------------ | --------------- |
| `! EXPRESSION`          |    	The EXPRESSION is false|
| `-n STRING` 	          |   The length of STRING is greater than zero                          |  
| `-z STRING` 	          |   The lengh of STRING is zero (ie it is empty)                       |     
| `STR1 = STR2` 	  |   STRING1 is equal to STRING2                            |
| `STR1 != STR2` 	  |   STRING1 is not equal to STRING2                                     |                   
| `INT1 -eq INT2` |	INTEGER1 is numerically equal to INTEGER2 (or `==`)                            |
| `INT1 -gt INT2` |	INTEGER1 is numerically greater than INTEGER2                         |   
| `INT1 -lt INT2` |	INTEGER1 is numerically less than INTEGER2                            |
| `INT1 -ne INT2` |	INTEGER1 is numerically not equal to INTEGER2                            |           

---

### Build conditions with the `test` command 

```bash 
test -s /etc/hosts
```

|  Operator    |     Description   |
| ------------ | --------------- |
| `-d FILE` 	            |  FILE exists and is a directory                            |
| `-e FILE` 	            |  FILE exists                                                        |
| `-s FILE` 	            |  FILE exists and it's size is greater than zero (ie. it is **not empty**)|
| `-r FILE` 	            |  FILE exists and the read permission is granted                      |   

> `-w` and `-x` test the write and the execute permission

---

### Conditional: light variation

Check an expression in the `if` statement ? 
Use the double brackets just like we did for variables :

```bash 
#!/bin/bash
echo "Please enter a number: "
read num

if (( $num % 2 == 0 ))
then
  echo "$num is an even number !"
fi
```

---

### Case Statements

```bash 
#!/bin/bash
space_free=$( /usr/bin/df -h | awk '{ print $5 }' | sort -n | tail -n 1 | sed 's/%//' )
case $space_free in
  [1-5]*)
    echo "Plenty of disk space available"
    ;;
  [6-7]*)
    echo "There could be a problem in the near future"
    ;;
  8*)
    echo "Maybe we should look at clearing out old files"
    ;;
  9*)
    echo "We could have a serious problem on our hands soon"
    ;;
  *)
    echo "Something is not quite right here"
    ;;
esac
```

---

# Hands-on exercise

1. In your `bash_exercises` folder create a new bash file called `exercise_2.sh` and make it executable
2. Ask the user for two numbers smaller or equal to 100, put them in variables `NUMBER1` and `NUMBER2`
3. Do the following only if the number are smaller than 100
4. Check if at least one of the numbers is a multiple of 3, and tell the user
5. Otherwise, check if both numbers are even, and tell the user   
6. If the user's numbers ar too big, tell them

---

# Arrays

### Indexed arrays

```bash 
# Declare an array with 4 elements
my_array=( 'Debian Linux' 'Redhat Linux' Ubuntu OpenSUSE )
# get number of elements in the array
my_array_length=${#my_array[@]}

# Declare an empty array
my_array=( )
my_array[0]=56.45
my_array[1]=568
echo Number of elements: ${#my_array[@]}
# echo array's content
echo ${my_array[2]}
echo ${my_array[@]}
```

---

### Associative arrays = Dictionaries (Bash 4.0 or higher)

By default, a bash array is an indexed array : need to use the `declare` command

```bash
declare -A acronyms # -A for associative array
acronyms[ACK]=Acknowledgement
acronyms[EOF]="End of Frame"
echo ${acronyms[ACK]}
if [ ${acronyms[EOF]+_} ]; then echo "Found"; else echo "Not found"; fi
```

> the variable expansion `${MYVAR+ABC}` expands to `ABC` is `MYVAR` is set and to nothing otherwise

```bash
declare -A countries=( [ALB]=Albania [BHR]=Bahrain [CMR]=Cameroon [DNK]=Denmark [EGY]=Egypt )
echo ${countries[@]}
echo ${!countries[@]}
countries+=( [FJI]=Fiji )
echo ${countries[@]}
unset countries[BHR]
echo ${countries[@]}
```

<!-- 
### About `declare`

```bash
declare --help
```

Examples :

- declare a variable without a value
- force a variable to be an integer only
- declare a variable as a **parameter** (read-only)
- force a character(s) variable to store all uppercase or lowercase letters
- display the attributes and values of all variables
- etc.
-->

---

# Loops

Useful for automating repetitive tasks

Basic loop structures in Bash scripting :

* `while` : perform a set of commands while a test is true
* `until` : perform a set of commands until a test is true
* `for` : perform a set of commands for each item in a list

* controlling loops

  * `break` : exit the currently running loop
  * `continue` : stop this iteration of the loop and begin the next iteration

* last loop mechanism : `select` allows you to create a simple menu system

---

### Examples

```bash
# Basic while loop
counter=0
while [ $counter -lt 3 ]; do
  let counter+=1
  echo $counter
done
```
```bash
# Basic until loop
counter=1
until [ $counter -gt 10 ]; do
  echo $counter
  ((counter++))
done
```

---

```bash
# range
for i in {1..5}
```
```bash
# list of strings
words='Hello great world'
for word in $words
```
```bash
# range with steps for loop
for value in {10..0..2}
```
```bash
# set of files
for file in $path/*.f90
```
```bash
# command result
for i in $( cat file.txt )
```

![bg right 85%](./assets/select_example.png)

---

```bash
#!/bin/bash
# How to iterate over keys or values of an Array

declare -A fruits
fruits[south]="Banana"
fruits[north]="Orange"
fruits[west]="Passion Fruit"
fruits[east]="Pineapple"

for key in "${!fruits[@]}"
do
  echo "Key is '$key' => Value is '${fruits[$key]}'"
done
```

---

# Hands-on exercise

Speed game:

1. Use the following webiste to get a list of random words: https://randomwordgenerator.com and put them together in a variable
2. Register the start time with `date +%s` and put it in a variable `TSTART`
3. Loop over the words and ask the user to give the number of letters. Put the answers in an associative array using the words as keys and the answers as values
4. Register the end time in `TEND`
5. Display the total run time
6. Loop over the associative array to compute the score (number of good answers) and show it to the user

---

# Arguments - Positional Parameters

How to pass command-line arguments to a bash script ?

Try a simple example called `test_arg.sh` :

```bash
#!/bin/bash
echo $1 $2 $4
echo $0
echo $#
echo $*
```

```bash
bash test_arg.sh a b c d e
```

```bash
a b d
test_arg.sh
5
a b c d e
```

---

|  Special Variables    |         |
| ------------ | --------------- |
|     `$0`    |  the name of the script |
|     `$1` - `$9` | the first 9 arguments |
|     `$#`    |  how many arguments were passed |
|     `$@`    |  all the arguments supplied |
|     `$$`    |  the process ID of the current script |
|     `$?`    |  the exit status of the most recently run process |

---

### Flags

```bash
#!/bin/bash
while getopts u:a:f: flag
do
    case "${flag}" in
        u) username=${OPTARG};;
        a) age=${OPTARG};;
        f) fullname=${OPTARG};;
    esac
done
echo "Username: $username";
echo "Age: $age";
echo "Full Name: $fullname";
```

```bash
bash test_arg.sh -f 'John Smith' -a 25 -u john
```

---

# Input/Output streams

Shells use 3 standard I/O streams

- `stdin` is the standard input stream, which provides input to commands
- `stdout` is the standard output stream, which displays output from commands
- `stderr` is the standard error stream, which displays error output from commands

Shell has several **meta-characters** and **control operators** 

> `|`, `&`, `>`, `;`, `<`, etc.

---

# Control operators

| Character | Effect |
| ---- | --- |
| `;` | Normal separator between commands |
| `&&` | Execute next command only if command succeeds |
| `\|\|` | Execute next command only if command fails |
| `&` | Don't wait for result of command before starting next command |
| `\|` | Use output of command as input for the next command |
| `> file_desc` | Send stdandard output of command to file descriptor |
| `< file_desc` | Use content of file descriptor as input |

---

# Redirections

Use the meta-character `>` in order to control the output streams `stdout` and `stderr` for a command or a bash script

### From bash script

```bash
#!/bin/bash
#STDOUT to STDERR
echo "Redirect this STDOUT to STDERR" 1>&2 
#STDERR to STDOUT
cat $1 2>&1
```

### Output streams to file(s)

```bash
./my_script.sh > STDOUT.log 2> STDERR.err
```

---

### How to Read a File Line By Line : input redirection

```bash
#!/bin/bash
# How to Read a File Line By Line

input="/path/to/txt/file"
while IFS= read -r line
do
  echo "$line"
done < "$input"
```

> by default `read` removes all leading and trailing whitespace characters such as spaces and tabs

<!-- 
`while IFS= read -r line`

The internal field separator (`IFS`) is set to the empty string to preserve whitespace issues

The `-r` option is used not to allow backslashes to escape any characters
-->

---

# Return codes

Linux command returns a status when it terminates normally or abnormally

* every Linux command has an exit status
* the exit status is an integer number
* a command which exits with a **0** status has **succeeded**
* a **non-zero** (1-255) exit status indicates **failure**

How do I display the exit status of shell command ?

```bash
date
echo $?
```

> [List of special exit codes for GNU/Linux](https://tldp.org/LDP/abs/html/exitcodes.html)

---

How to store the exit status of the command in a shell variable ?

```bash
#!/bin/bash
date
status=$?
echo "The date command exit status : ${status}"
```

How to use the `&&` and `||` operators with **exit codes**

```bash
command && echo "success"
command || echo "failed"
command && echo "success" || echo "failed"
```

```bash
_files="$@"
[[ "$_files" == "" ]] && { echo "Usage: $0 file1.png file2.png"; exit 1; }
```
<!-- Attention utilisatin du [[ ]] pas encore expliqué ? -->

---

# Functions

* "small script within a script" that you may call multiple times 
* great way to reuse code
* a function is most reuseable when it performs a single task
* good to put ancillary tasks within functions : logically separate from main

```bash
#!/bin/bash
hello_world () {
   echo 'hello, world'
}
hello_world
```

> defining a function doesn’t execute it

Functions must be declared **before** they are used

---

## Variables Scope

```bash
# Define bash global variable
# This variable is global and can be used anywhere in this bash script
var="global variable"

function my_function {
# Define my_function local variable
# This variable is local to my_function only
echo $var
local var="local variable"
echo $var
}

echo $var
my_function
# Note the bash global variable did not change
# "local" is my_function reserved word
echo $var
```

---

## Return Values 

Bash functions don’t allow you to return a value when called

After completion, the return value is the **status** of the last statement (so 0-255)

It can also be specified manually by using `return` :

```bash
my_function () {
  echo "some result"
  return 55
}
my_function
echo $?
```

---

Return an arbitrary value (different from a return code) from a function : 
* Assign the result of the function

```bash
my_function () {
  func_result="some result"
}
my_function
echo $func_result
```

* Better way is to send the value to `stdout` using `echo`

```bash
my_function () {
  local func_result="some result"
  echo "$func_result"
}
func_result="$(my_function)"
echo $func_result
```

---

## Passing Arguments

In the same way than a bash script: see above (`$1`, `$*`, etc)

```bash
#!/bin/bash
print_something () {
    echo Hello $1
}
print_something Mars
```

## Be careful in case of "overriding commands"

```bash
#!/bin/bash
ls () {
  command ls -lh # 'command' keyword prevents ambiguity
}
ls
```

---

# Hands-on exercise

1. Write a script called `exercise_3.sh` expecting **2 arguments**. If not exactly two arguments are provided, exit with an error and show a "usage" message to the user.
2. Write a function taking a **folder path** and an **extension** as arguments and giving the list of matching files to the user
3. Imagine you are running jobs taking data from two folders, each with a dedicated extension. Use the two arguments of the script as the name of the two folders and **get the two lists of files**. 
4. Since your work needs to read these files, **check that all files can be read**. If some files cannot be read, display their name to the user

---

# Shell vs Environment Variables

Consider the script `test.sh` below :

```bash
#!/bin/bash
echo $var1
echo $var2
```

Then run this script :

```bash
var1=23
export var2=12
bash test.sh
```

---

# Subshells

* A subshell is a "child shell" spawned by the main shell ("parent shell")
* A subshell is a **separate** instance of the command process, run as a new process
* **Unlike calling a shell script** (slide before), subshells inherit the **same** variables as the original process
* A subshell allows you to execute commands within a separate shell environment = *Subshell Sandboxing*
  > useful to set temporary variables or change directories without affecting the parent shell's environment 
* Subshells can be used for **parallel processing** 

---

### Syntax

A command list embedded between parentheses runs as a subshell :

```bash
#!/bin/bash
( command1 ; command2 ; command3 )
```

Or :

```bash
#!/bin/bash
bash -c "command1; command2; command3"
```

> Reminder : variables in a subshell are **not** visible outside the block of code in the subshell

---

## Differences between **Sourcing** and **Executing** a script

- source a script = execution in the current shell
  > variables and functions are valid in the current shell after sourcing

- execute a script = execution in a new shell (in a subshell of the current shell)
  > all new variables and functions created by the script will only live in the subshell

Source a script using `source` or `.`

```bash
source myScript.sh
. myScript.sh
```

> official one is `.` Bash defined `source` as an alias to the `.`

---

## Example

```bash