Scripting and Regular Expressions

Last time: Introduction to Shell Scripting

Prof. Brian D. Davison

Computer Science & Engineering, Lehigh University

Additional Credits

Announcements

• Reminder: Project #5 (debugging) due Wednesday (can't wait until day before!)
• Today's schedule
  • Finish reviewing exam #1
  • Finish Intro to Shell Scripting
  • Continue Shell Scripting with Regular Expressions

Bash shell constructs

• Functions in bash

  function_name() {
    block of code
  }
    
  function function_name {
    block of code
  }
      
  ...
  function_name
  

File Testing

• bash has a number of built-in options to test attributes of files

  if [ -f file ] 
  then
      echo file exists
  fi
  
  if [ -d file ]
  then
      echo file is a directory
  fi
  
  if [ -s file ]
  then
      echo file is nonzero size
  fi
  
  if [ -r file -a -w file ]
  then
      echo file is readable and writable
  fi
  

Ridirecting Input

• Sometimes you want to be able to pass multiline data into a program.
• If that data is in a file, you redirect stdin (e.g., with <)
• If that data is the output of another program, you use a pipe to connect the m
• But what if you want to dynamically generate that data?
  • You could use something like "echo line-n >> file"
  • and then run as "program < file"
  • A better answer is the "here document"

The here document

• The here document provides the ability to send inline text to a program until a user-defined terminator is reached.
• To use the here document, after the command receiving the input we place a << symbol and the name of the terminating string. The text to be used follows, terminated by the same user-defined string on a line by itself.

  $ cat << FINISH
  > Hello $LOGNAME
  > Time is $(date +%T).
  > FINISH
  Hello brian
  Time is 13:21:33.
  

Positional Parameters

• The scriptname and arguments can be referenced using positional parameters.
• $0 is the script, and $1 - $9 are the first 9 arguments.
• If you need access to more, but the number in braces, e.g., ${10}
  
• Positional parameters can be reset using the set command

  #!/bin/bash
  echo Script was called as $0 $*
  oldargs=$*
  set new1 new2 new3
  echo New params are $0 $*
  set $oldargs
  echo Restored $0 $*
  

  A longer example

  • Let's examine a longer shell scripting example (remember to open this and any other link in a new tab)

  As we read through the file, we see: the sh-bang line, comments, local variable declarations. It then tests for the non-existence of a file, and so if true (doesn't exist), it prints an error and exits. We then create a global (environmental) variable, followed by a local numeric variable given the result of an expression that executes the system command date (asking only for the hours) and adding 1 to it. We then make sure the number of hours is not in military (24hr time). A local array of foods is declared, and an integer too. For each person in the list generated by the system command cat on a file (except for root), run the command mail with a message (with variable substitutions) to send. The index variable is updated on each iteration, and is reset if we run through the entire set of food choices.

  Globbing

  • The shell supports wildcards/globbing to match (possibly) multiple filenames from a single expression

     bash$ ls -l
     total 2
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
     -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
     -rw-rw-r--    1 bozo  bozo       758 Jul 30 09:02 test1.txt
     
     bash$ ls -l t?.sh
     -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
    

    Some examples today are taken from chapter 19 of the Advanced Bash-Scripting Guide.

  Metacharacters: ? for single character matches, * for multi-character matches, [] for matching one of a set, and - within brackets for range matches.

  Globbing

  • The shell supports wildcards/globbing to match (possibly) multiple filenames from a single expression

     bash$ ls -l [ab]*
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
    
     bash$ ls -l [a-c]*
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
     -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
    

    Some examples today are taken from chapter 19 of the Advanced Bash-Scripting Guide.

  Metacharacters: ? for single character matches, * for multi-character matches, [] for matching one of a set, and - within brackets for range matches.

  Globbing

   bash$ ls -l [^ab]*
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
   -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
   -rw-rw-r--    1 bozo  bozo       758 Jul 30 09:02 test1.txt
   
   
   bash$ echo *
   a.1 b.1 c.1 t2.sh test1.txt
   
   bash$ echo t*
   t2.sh test1.txt
  

  • Long ago, wildcard expansion was provided by an external program called glob, but has since been incorporated into the shell directly.
  • The ability to perform globbing is also provided to unix programs, using the standard glob(3c) C library function.
  • You can read about the POSIX standard for globbing in the linux man page for glob(7)

  The ^ within brackets provides negation.

  Regular Expressions

  • Invented by Steven Kleene as a notation for describing "the algebra of regular sets". They have since become the standard way of matching patterns of text.
  • Regular Expressions (or "regexes" for short) contain much more powerful pattern matching capabilities than simple globbing.
  • Numerous Unix programs use regexes, including grep, sed, awk, perl, vi, and emacs.
  • To demonstrate, let's use grep

    bash$ grep '^Be.*m$' /usr/share/lib/dict/words 
    Beecham
    Belgium
    Bellingham
    Bentham
    Bergstrom
    Bertram
    Bethlehem
    

    This found all the dictionary words (lines) that begin with 'Be' and end with 'm'.

  Try changing this slightly to find other words in the dictionary file.

  grep

  • grep is an acronym that stands for global regular expression print.
  • Typically we'll use grep to match a string literal (a regular expression without metacharacters)

    % ps -ef | grep root
    

  • grep has a number of useful options, including
    • -i ignore case
    • -n include line numbers in output
    • -w search for words
    • -l list filenames only
    • -v show non-matching lines only
  • Today, however, we're focusing on the regular expressions such as those that grep uses (described in Hahn, Ch 20, and in grep man page).

  Regular Expressions

  • Regular Expressions
    • A regular expression is a way of describing a pattern of characters to a program so that it can either display or modify the occurrences of that pattern.
    • Regular expressions are constructed analogously to arithmetic expressions, by using various operators to combine smaller expressions.
    • The fundamental building blocks are the regular expressions that match a single character.
    • Most characters, including all letters and digits, are regular expressions that match themselves.
  • The simplest form of a regular expression is just a literal string.
  • In addition to single characters and literal strings, regular expressions use a set of metacharacters that have special meaning when used in a regular expression.

  Basic Regular Expression Metacharacters

  • Wildcard matching
    • . -- matches any one character
    • [abc] -- matches any one of the letters 'a', 'b', or 'c'
    • [^abc] -- matches any character except 'a', 'b', or 'c'
    • [a-zA-Z0-9] -- matches either a lower case letter, upper case letter, or a digit
  • Quantifiers
    • * -- match previous character zero or more times
  • Anchors
    • ^ -- matches beginning of line
    • $ -- matches end of line
    • \< -- matches beginning of word
    • \> -- matches end of word

  Regular Expression Tags

  • The \(..\) metacharacters provide the ability to tag the matching text of one part of an expression and re-use it later.
  • References later are \1 through \9 (so you can have up to 9 tags)

    Le\(.*\>\) is \1 up South Mountain
    
    would match both
    
    Lehigh is high up South Mountain
    Lee is e up South Mountain
    

    but not

    Lehigh is way up South Mountain
    

  Extended RegEx Metacharacters

  • Wildcard matching
    • | -- matches either the character before or after

      a|b|c
      

    • () -- groups characters for more complicated expressions

      Lehi(gh)*
      

  • Quantifiers
    • ? -- match previous character zero or one time
    • + -- match previous character one or more times
    
    
    
  • These are available with egrep(1)