Skip to content
Extraits de code Groupes Projets
Valider 13a1aa78 rédigé par Pierre-Yves Barriat's avatar Pierre-Yves Barriat
Parcourir les fichiers

Initialization

parent bd410755
Aucune branche associée trouvée
Aucune étiquette associée trouvée
Aucune requête de fusion associée trouvée
Bash.html
---
marp: true
title: Introduction to Bash Scripting
author: P.Y. Barriat
description: Introduction to Bash Scripting
backgroundImage: url('assets/back.png')
_backgroundImage: url('assets/garde.png')
footer: 09/11/2023 | Introduction to Bash Scripting
_footer: ""
paginate: true
_paginate: false
---
Introduction to Bash Scripting<!--fit-->
===
https://gogs.elic.ucl.ac.be/pbarriat/learning-bash
![h:200](assets/bash_logo.png)
##### October 17, 2023
###### CISM/CÉCI Training Sessions
---
# Linux command line
A Linux terminal is where you enter Linux commands
It's called the **C**ommand **L**ine **U**ser **I**nterface
**CLUI** is one of the many strengths of Linux :
- allows to be independent of distros (or UNIX systems like OSX)
- allows to easily work at distance (SSH)
- allows to join together simple (and less simple) commands to do complex things and automate **= scripting**
In Linux, process automation relies heavily on scripting. This involves creating a file containing a series of commands that can be executed together
---
# Linux Shell
A **shell** is a program that takes commands from the keyboard and gives them to the operating system to perform
The main function is to interpret your commands **= language**
Shells have some built-in commands
A shell also supports programming constructs, allowing complex commands to be built from smaller parts **= scripts**
Scripts can be saved as files to become new commands
> many commands on a typical Linux system are scripts
---
# Bash
The **Bash** shell is one of several shells available for Linux
It is the default command interpreter on most GNU/Linux systems. The name is an acronym for the "**B**ourne-**A**gain **SH**ell"
### Bash Scripting Demo
```bash
#!/bin/bash
# declare STRING variable
STRING="Hello World"
# print variable on a screen
echo $STRING
```
---
![h:600](assets/scripting.png)
---
# Shell syntax rules
Shells use 3 **"standard I/O streams"**
- `stdin` is the standard input stream, which provides input to commands
- `stdout` is the standard output stream, which displays output from commands
- `stderr` is the standard error stream, which displays error output from commands
Shell has several **meta-characters** and **control operators**
> `|`, `&`, `>`, `;` , etc.
---
# Bash environment
In a Bash shell many things constitute your environment
- the form of your prompt
- your home directory and your working directory
- the name of your shell
- functions that you have defined
- etc.
Environment includes many variables that may have been set **by bash** or **by you**
> Access the value of a variable by prefixing its name with `$`
---
# Environment variables
| Variables | |
| ------------ | --------------- |
| `USER` | the name of the logged-in user |
| `HOME` | the user's home directory (similar to `~` ) |
| `PWD` | the current working directory |
| `SHELL` | the name of the shell |
| `UID` | the numeric user id of the logged-in user |
> You can use special files to control bash variables : `$HOME/.bashrc`
---
# Bash Scripting basics
By naming convention, bash scripts end with `.sh`
> however, bash scripts can run perfectly fine without any extension
A good practice is to define a `shebang` : first line of the script, `shebang` is simply an absolute path to the shell interpreter (see `echo $SHELL` result)
> combination of `bash #` and `bang !`
### Comments start with `#`
On a line, any characters after `#` will be ignored -- *with the exception of* `#!`
```bash
echo "A comment will follow." # Comment here.
# ^ Note whitespace before #
```
---
### There is no standard indentation
- Pick a standard in your team that you can all work to
- Use something your editor makes easy (**Vim** uses `Tab`)
### Command separators
Commands can be combined using **meta-characters** and **control operators**
```bash
# cmd1; cmd2
$ cd myfolder; ls # no matter cd to myfolder successfully, run ls
# cmd1 && cmd2
$ cd myfolder && ls # run ls only after cd to myfolder
# cmd1 || cmd2
$ cd myfolder || ls # if failed cd to myfolder, `ls` will run
```
---
# Permissions and execution
- Bash script is nothing else just a **text file** containing instructions to be executed sequentially
> by default in Linux, a new text file is **-rw-r--r--** (or 644)
- You can run the script `hello_world.sh` using
* `sh hello_world.sh`
* `bash hello_world.sh`
* `chmod u+x run_all.sh` then `./hello_world.sh`
> after the `chmod`, you file is **-rwxr--r--** (or 744)
---
# Variables and data types in Bash
Variables let you store data : **numeric values** or **character(s)**
You can use variables to read, access, and manipulate data throughout your script
**There are no data types in Bash**
Set the variable values in the following ways :
- assign directly : `greeting="Welcome"` or `a=4`
- access the variable value using `$`: `echo $greeting`
- assign based on variable: `b=$a`
> !!! no space before or after `=` in the assignation !!!
> `myvar=Hello World` :boom: `-bash: World: command not found`
---
### Quotes for character(s) `" '`
Double will do variable substitution, single will not
### Command Substitution
```bash
#!/bin/bash
# Save the output of a command into a variable
myvar=$( ls )
```
### Export variables
```bash
#!/bin/bash
var1=blah
# Make the variable `var1` available to child processes
export var1
./script2.sh
```
---
## Variable naming conventions
* Variable names **should start** with a letter or an underscore
* Variable names can contain letters, numbers, and underscores
* Variable names are **case-sensitive**
* Variable names **should not** contain spaces or **special characters**
* Use descriptive names that reflect the purpose of the variable
* Avoid using **reserved keywords**, such as `if`, `then`, `else`, `fi`, and so on...
* **Never** name your private variables using only **UPPERCASE** characters
---
| Special Variables | |
| ------------ | --------------- |
| `$0` | the name of the script |
| `$1` - `$9` | the first 9 arguments |
| `$#` | how many arguments were passed |
| `$@` | all the arguments supplied |
| `$$` | the process ID of the current script |
| `$?` | the exit status of the most recently run process |
| `$RANDOM` | returns a random number |
| `$LINENO` | returns the current line number |
| `$SECONDS` | the number of seconds since the script was started |
---
# Arithmetic
| Operator | Operation |
| ------------ | --------------- |
| `+` `-` `\*` `/` | addition, subtraction, multiply, divide |
| `var++` | increase the variable var by 1 |
| `var--` | decrease the variable var by 1 |
| `%` | modulus (Return the remainder after division) |
Several ways to go about arithmetic in Bash scripting :
`let`, `expr` or using **double parentheses**
Return the length of a variable : `${#var}`
---
- `let` make a variable equal to an expression
```bash
#!/bin/bash
let a=5+4
echo $a # 9
let a++
let "a = 4 * $a"
```
- `expr` : print out the result of the expression
```bash
#!/bin/bash
expr 5 + 4 # 9
expr 5+4 # boom
expr "5 + 4" # boom
a=$( expr 10 - 3 )
```
- double parentheses : return the result of the expression
> it is the preferred method
---
```bash
#!/bin/bash
a=$(( 4 * 5 ))
a=$(( 4 + 5 ))
a=$((3+5))
b=$(( a + 3 ))
echo $b # 11
b=$(( $a + 4 ))
echo $b # 12
(( b++ ))
(( b += 3 ))
echo $b # 16
```
---
# Conditional statements
```bash
#!/bin/bash
echo "Please enter a number: "
read num
if [ $num -gt 5 ] && [ $num -le 7 ]
then
echo "$num is 6 or 7"
elif [ $num -lt 0 ] || [ $num -eq 0 ]; then
echo "$num is negative or zero"
else
echo "$num is positive (but not 6, 7 or zero)"
fi
```
> read the **standard input stream** with the `read` command
---
### `test` command
```bash
test -s /proc/cpuinfo
echo $?
```
| Operator | Description |
| ------------ | --------------- |
| `-d FILE` | FILE exists and is a directory |
| `-e FILE` | FILE exists |
| `-r FILE` | FILE exists and the read permission is granted |
| `-s FILE` | FILE exists and it's size is greater than zero (ie. it is not empty)|
| `-w FILE` | FILE exists and the write permission is granted |
| `-x FILE` | FILE exists and the execute permission is granted |
---
| Operator | Description |
| ------------ | --------------- |
| `! EXPRESSION` | The EXPRESSION is false|
| `-n STRING` | The length of STRING is greater than zero |
| `-z STRING` | The lengh of STRING is zero (ie it is empty) |
| `STR1 = STR2` | STRING1 is equal to STRING2 |
| `STR1 != STR2` | STRING1 is not equal to STRING2 |
| `INT1 -eq INT2` | INTEGER1 is numerically equal to INTEGER2 (or `==`) |
| `INT1 -gt INT2` | INTEGER1 is numerically greater than INTEGER2 |
| `INT1 -lt INT2` | INTEGER1 is numerically less than INTEGER2 |
| `INT1 -ne INT2` | INTEGER1 is numerically not equal to INTEGER2 |
---
### Conditional: light variation
Check an expression in the `if` statement ?
Use the double brackets just like we did for variables :
```bash
#!/bin/bash
echo "Please enter a number: "
read num
if (( $num % 2 == 0 ))
then
echo "$num is an even number !"
fi
```
---
### Case Statements
```bash
#!/bin/bash
space_free=$( /usr/bin/df -h | awk '{ print $5 }' | sort -n | tail -n 1 | sed 's/%//' )
case $space_free in
[1-5]*)
echo "Plenty of disk space available"
;;
[6-7]*)
echo "There could be a problem in the near future"
;;
8*)
echo "Maybe we should look at clearing out old files"
;;
9*)
echo "We could have a serious problem on our hands soon"
;;
*)
echo "Something is not quite right here"
;;
esac
```
---
# Arrays
### Indexed arrays
```bash
#!/bin/bash
# Declare an array with 4 elements
my_array=( 'Debian Linux' 'Redhat Linux' Ubuntu OpenSUSE )
# get number of elements in the array
my_array_length=${#my_array[@]}
# Declare an empty array
my_array=( )
my_array[0]=56.45
my_array[1]=568
echo Number of elements: ${#my_array[@]}
# echo array's content
echo ${my_array[@]}
```
---
### Associative arrays = Dictionaries (Bash 4.0 or higher)
By default, a bash array is an indexed array : need to use the `declare` command
```bash
#!/bin/bash
declare -A acronyms
acronyms[ACK]=Acknowledgement
acronyms[EOF]="End of Frame"
echo ${acronyms[ACK]}
if [ ${acronyms[EOF]+_} ]; then echo "Found"; else echo "Not found"; fi
```
```bash
#!/bin/bash
declare -A countries=( [ALB]=Albania [BHR]=Bahrain [CMR]=Cameroon [DNK]=Denmark [EGY]=Egypt )
echo ${countries[@]}
echo ${!countries[@]}
countries+=( [FJI]=Fiji )
echo ${countries[@]}
unset countries[BHR]
echo ${countries[@]}
```
---
### About `declare`
```bash
declare --help
```
Examples :
- declare a variable without a value
- force a variable to be an integer only
- declare a variable as a **parameter** (read-only)
- force a character(s) variable to store all uppercase or lowercase letters
- display the attributes and values of all variables
- etc.
---
# Loops
Useful for automating repetitive tasks
Basic loop structures in Bash scripting :
* `while` : perform a set of commands while a test is true
* `until` : perform a set of commands until a test is true
* `for` : perform a set of commands for each item in a list
* controlling loops
* `break` : exit the currently running loop
* `continue` : stop this iteration of the loop and begin the next iteration
* last loop mechanism : `select` allows you to create a simple menu system
---
### Examples
```bash
#!/bin/bash
# Basic while loop
counter=0
while [ $counter -lt 3 ]; do
let counter+=1
echo $counter
done
```
```bash
#!/bin/bash
# Basic until loop
counter=1
until [ $counter -gt 10 ]; do
echo $counter
((counter++))
done
```
---
```bash
# range
for i in {1..5}
```
```bash
# list of strings
words='Hello great world'
for word in $words
```
```bash
# range with steps for loop
for value in {10..0..2}
```
```bash
# set of files
for file in $path/*.f90
```
```bash
# command result
for i in $( cat file.txt )
```
![bg right 85%](./assets/select_example.png)
---
```bash
#!/bin/bash
# How to Read a File Line By Line
input="/path/to/txt/file"
while IFS= read -r line
do
echo "$line"
done < "$input"
```
The internal field separator (`IFS`) is set to the empty string to preserve whitespace issues
> by default `read` removes all leading and trailing whitespace characters such as spaces and tabs
The `-r` option is used not to allow backslashes to escape any characters
---
```bash
#!/bin/bash
# How to iterate over keys or values of an Array
declare -A fruits
fruits[south]="Banana"
fruits[north]="Orange"
fruits[west]="Passion Fruit"
fruits[east]="Pineapple"
for key in "${!fruits[@]}"
do
echo "Key is '$key' => Value is '${fruits[$key]}'"
done
```
---
# Arguments - Positional Parameters
How to pass command-line arguments to a bash script ?
Try a simple example called `test_arg.sh` :
```bash
#!/bin/bash
echo $1 $2 $4
echo $0
echo $#
echo $*
```
```bash
bash test_arg.sh a b c d e
```
```bash
a b d
test_arg.sh
5
a b c d e
```
---
### Flags
```bash
#!/bin/bash
while getopts u:a:f: flag
do
case "${flag}" in
u) username=${OPTARG};;
a) age=${OPTARG};;
f) fullname=${OPTARG};;
esac
done
echo "Username: $username";
echo "Age: $age";
echo "Full Name: $fullname";
```
```bash
bash test_arg.sh -f 'John Smith' -a 25 -u john
```
---
# Redirections
Use the meta-character `>` in order to control the output streams `stdout` and `stderr` for a command or a bash script
### From bash script
```bash
#!/bin/bash
#STDOUT to STDERR
echo "Redirect this STDOUT to STDERR" 1>&2
#STDERR to STDOUT
cat $1 2>&1
```
### Output streams to file(s)
```bash
./my_script.sh > STDOUT.log 2> STDERR.err
```
---
# Return codes
Linux command returns a status when it terminates normally or abnormally
* every Linux command has an exit status
* the exit status is an integer number
* a command which exits with a **0** status has **succeeded**
* a **non-zero** (1-255) exit status indicates **failure**
How do I display the exit status of shell command ?
```bash
date
echo $?
```
> [List of common exit codes for GNU/Linux](https://slg.ddnss.de/list-of-common-exit-codes-for-gnu-linux/)
---
How to store the exit status of the command in a shell variable ?
```bash
#!/bin/bash
date
status=$?
echo "The date command exit status : ${status}"
```
How to use the `&&` and `||` operators with **exit codes**
```bash
command && echo "success"
command || echo "failed"
command && echo "success" || echo "failed"
```
```bash
_files="$@"
[[ "$_files" == "" ]] && { echo "Usage: $0 file1.png file2.png"; exit 1; }
```
---
# Functions
* "small script within a script" that you may call multiple times
* great way to reuse code
* a function is most reuseable when it performs a single task
* good to put ancillary tasks within functions : logically separate from main
```bash
#!/bin/bash
hello_world () {
echo 'hello, world'
}
hello_world
```
> defining a function doesn’t execute it
Functions must be declared **before** they are used
---
## Variables Scope
```bash
#!/bin/bash
# Define bash global variable
# This variable is global and can be used anywhere in this bash script
var="global variable"
function my_function {
# Define my_function local variable
# This variable is local to my_function only
echo $var
local var="local variable"
echo $var
}
echo $var
my_function
# Note the bash global variable did not change
# "local" is my_function reserved word
echo $var
```
---
## Return Values
Bash functions don’t allow you to return a value when called
After completion, the return value is the status of the last statement (so 0-255)
Can be specified by using `return` :
```bash
#!/bin/bash
my_function () {
echo "some result"
return 55
}
my_function
echo $?
```
---
Return an arbitrary value from a function : assign the result of the function
```bash
#!/bin/bash
my_function () {
func_result="some result"
}
my_function
echo $func_result
```
Better way is to send the value to `stdout` using `echo`
```bash
#!/bin/bash
my_function () {
local func_result="some result"
echo "$func_result"
}
func_result="$(my_function)"
echo $func_result
```
---
## Passing Arguments
In the same way than a bash script: see above (`$1`, `$*`, etc)
```bash
#!/bin/bash
print_something () {
echo Hello $1
}
print_something Mars
```
## Be careful in case of "overriding commands"
```bash
#!/bin/bash
ls () {
command ls -lh
}
ls
```
---
# Shell vs Environment Variables
Consider the script `test.sh` below :
```bash
#!/bin/bash
echo $var1
echo $var2
```
Then run this script :
```bash
var1=23
export var2=12
bash test.sh
```
---
# Subshells
* subshell is a "child shell" spawned by the main shell ("parent shell")
* subshell is separate instance of the command process, run as a new process
* unlike calling a shell script, subshells inherit the same variables as the original process
* a subshell allow you to execute commands within a separate shell environment = *Subshell Sandboxing*
> useful to set temporary variables or change directories without affecting the parent shell's environment
* subshells can be used for parallel processing
---
### Syntax
A command list embedded between parentheses runs as a subshell :
```bash
#!/bin/bash
( command1 ; command2 ; command3 )
```
Or :
```bash
#!/bin/bash
bash -c "command1; command2; command3"
```
> Reminder : variables in a subshell are not visible outside the block of code in the subshell
---
### Sourcing Bash scripts
```bash
#!/bin/bash
COUNTRY="Belgium"
greeting() {
echo "You're in $1"
}
greeting $COUNTRY
```
```bash
COUNTRY="France"
./myScript.sh # or bash or exec
echo $COUNTRY
greeting $COUNTRY
```
```bash
source myScript.sh
echo $COUNTRY
greeting $COUNTRY
```
---
Differences between **Sourcing** and **Executing** a script
- source a script = execution in the current shell
> variables and functions are valid in the current shell after sourcing
- execute a script = execution in a new shell (in a subshell of the current shell)
> all new variables and functions created by the script will only live in the subshell
Source a script using `source` or `.`
```bash
source myScript.sh
. myScript.sh
```
> official one is `.` Bash defined `source` as an alias to the `.`
---
### Running parallel processes in subshells
Processes may execute in parallel within different subshells
> permits breaking a complex task into subcomponents processed concurrently
Exemple : `job.sh`
```bash
#!/bin/bash
i=0
while [ $i -lt 10 ]; do
echo "${i}: job $1"
i=$[$i+1]
sleep 0.2
done
```
2 ways to use it :
sequential processing (`manager_seq.sh`) or parallel processing (`manager_par.sh`)
---
```bash
#!/bin/bash
# manager_seq.sh
echo "start"
./job.sh 1
./job.sh 2
echo "done"
```
```bash
#!/bin/bash
# manager_par.sh
echo "start"
./job.sh 1 &
./job.sh 2 &
wait # Don't execute the next command until subshells finish.
echo "done"
```
```bash
time ./manager_seq.sh
time ./manager_par.sh
```
---
# Debug
Tips and techniques for debugging and troubleshooting Bash scripts
### use `set -x`
enables debugging mode : print each command that it executes to the terminal, preceded by a `+`
### check the exit code
```bash
#!/bin/bash
if [ $? -ne 0 ]; then
echo "Error occurred"
fi
```
---
### use `echo`
"classical" but useful technique : insert `echo` throughout your code
```bash
#!/bin/bash
echo "Value of variable x is: $x"
```
### use `set -e`
this option will cause Bash to exit with an error if any command in the script fails
---
Thank you for your attention<!--fit-->
===
\ No newline at end of file
Fichier ajouté
# learning-bash # learning-bash
This is the repository for the training Markdown and applications This is the repository for the training Markdown and applications
\ No newline at end of file
You can find the slides [here](./Bash.pdf).
Presentation is [here](https://www.elic.ucl.ac.be/users/pbarriat/slides/Bash.html)
\ No newline at end of file
assets/back.png

62 ko

assets/bash_logo.png

120 ko

assets/garde.png

44,7 ko

const marpKrokiPlugin = require('./kroki-plugin')
module.exports = {
engine: ({ marp }) => marp.use(marpKrokiPlugin)
}
assets/scripting.png

75,2 ko

assets/select_example.png

89,8 ko

/* @theme tum */
@import 'default';
section {
/*background-color: #fff;
color: #000;
background-image: url('images/TUM_Logo_blau_rgb_s.svg');
background-repeat: no-repeat;
background-position: right 40px top 40px;
background-size: 8%;*/
}
section.lead {
/*background-image: url('images/TUM_Uhrenturm.png');
background-position: right;
background-size: 45%;*/
}
section h1,
section h2 {
color: #1f315c;
}
section a {
color: #5fb2e6;
}
section footer,
section::after {
color: #9cb7d4;
}
section.invert {
background-color: #003359;
color: #fff;
/*background-image: url('images/TUM_Logo_weiss_rgb_s.svg');*/
}
section.lead.invert {
/*background-image: url('images/TUM_Uhrenturm_w.png');*/
}
section.invert h1,
section.invert footer,
section.invert::after {
color: #fff;
}
section.invert a {
color: #e37222;
}
/* Add "Page" prefix and total page number */
section::after {
content: attr(data-marpit-pagination) ' / ' attr(data-marpit-pagination-total);
}
#!/bin/bash
#
# PY Barriat, September 2023
#
# Download and install marp (MarkDown slides extension) from here:
# https://github.com/marp-team/marp-cli/releases
#
marp --allow-local-files --theme ./assets/tum.css Bash.md -o Bash.pdf
marp --template bespoke --bespoke.progress --allow-local-files --theme ./assets/tum.css Bash.md -o Bash.html
0% Chargement en cours ou .
You are about to add 0 people to the discussion. Proceed with caution.
Terminez d'abord l'édition de ce message.
Veuillez vous inscrire ou vous pour commenter