Scripting and Regular Expressions

Prof. Brian D. Davison

Computer Science & Engineering, Lehigh University

Announcements

• Welcome back! Hope you enjoyed the extra couple of days off!
• New homework #4 online, due Friday in class
• Today's schedule
  • Return short quiz #6 -- range 3-9, mean 7
  • Continue Shell Scripting with Regular Expressions

File Testing

• bash has a number of built-in options to test attributes of files

  if [ -f file ] 
  then
      echo file exists
  fi
  
  if [ -d file ]
  then
      echo file is a directory
  fi
  
  if [ -s file ]
  then
      echo file is nonzero size
  fi
  
  if [ -r file -a -w file ]
  then
      echo file is readable and writable
  fi
  

A longer example

• First, let's review the shell scripting we saw before break in a longer example (remember to open this and any other link in a new tab)

Globbing

• We have seen previously that the shell supports wildcards/globbing to match (possibly) multiple filenames from a single expression

   bash$ ls -l
   total 2
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
   -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
   -rw-rw-r--    1 bozo  bozo       758 Jul 30 09:02 test1.txt
   
   bash$ ls -l t?.sh
   -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
   
   bash$ ls -l [ab]*
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
  
   bash$ ls -l [a-c]*
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 a.1
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 b.1
   -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
  

  Some examples today are taken from chapter 19 of the Advanced Bash-Scripting Guide.
• Here you see the use of a few metacharacters, including ? for single character matches, * for multi-character matches, [] for matching one of a set, and - within brackets for range matches.

Globbing

 bash$ ls -l [^ab]*
 -rw-rw-r--    1 bozo  bozo         0 Aug  6 18:42 c.1
 -rw-rw-r--    1 bozo  bozo       466 Aug  6 17:48 t2.sh
 -rw-rw-r--    1 bozo  bozo       758 Jul 30 09:02 test1.txt
 
 
 bash$ echo *
 a.1 b.1 c.1 t2.sh test1.txt
 
 bash$ echo t*
 t2.sh test1.txt

• Long ago, wildcard expansion was provided by an external program called glob, but has since been incorporated into the shell directly.
• The ability to perform globbing is also provided to unix programs, using the standard glob(3c) C library function.
• You can read about the POSIX standard for globbing in the linux man page for glob(7)

Regular Expressions

• Regular Expressions were invented by Steven Kleene as a notation for describing "the algebra of regular sets". They have since become the standard way of matching patterns of text.
• Regular Expressions (or "regexes" for short) contain much more powerful pattern matching capabilities than simple globbing.
• Numerous Unix programs use regexes, including grep, sed, awk, perl, vi, and emacs.
• To demonstrate, let's use grep

  bash$ grep '^Be.*m$' /usr/share/lib/dict/words 
  Beecham
  Belgium
  Bellingham
  Bentham
  Bergstrom
  Bertram
  Bethlehem
  

  This found all the dictionary words (lines) that begin with 'Be' and end with 'm'.

grep

• grep is an acronym that stands for global regular expression print.
• Typically we'll use grep to match a string literal (a regular expression without metacharacters)

  % ps -ef | grep root
  

• grep has a number of useful options, including
  • -i ignore case
  • -n include line numbers in output
  • -w search for words
  • -l list filenames only
  • -v show non-matching lines only
• Today, however, we're focusing on the regular expressions such as those that grep uses (described in Quigley, Ch 4, and in grep man page).

Regular Expressions

• Regular Expressions
  • A regular expression is a way of describing a pattern of characters to a program so that it can either display or modify the occurrences of that pattern.
  • Regular expressions are constructed analogously to arithmetic expressions, by using various operators to combine smaller expressions.
  • The fundamental building blocks are the regular expressions that match a single character.
  • Most characters, including all letters and digits, are regular expressions that match themselves.
• The simplest form of a regular expression is just a literal string.
• In addition to single characters and literal strings, regular expressions use a set of metacharacters that have special meaning when used in a regular expression.

Basic Regular Expression Metacharacters

• Wildcard matching
  • . -- matches any one character
  • [abc] -- matches any one of the letters 'a', 'b', or 'c'
  • [^abc] -- matches any character except 'a', 'b', or 'c'
  • [a-zA-Z0-9] -- matches either a lower case letter, upper case letter, or a digit
• Quantifiers
  • * -- match previous character zero or more times
• Anchors
  • ^ -- matches beginning of line
  • $ -- matches end of line
  • \< -- matches beginning of word
  • \> -- matches end of word

Regular Expression Tags

• The \(..\) metacharacters provide the ability to tag the matching text of one part of an expression and re-use it later.
• References later are \1 through \9 (so you can have up to 9 tags)

  Le\(.*\>\) is \1 up South Mountain
  
  would match both
  
  Lehigh is high up South Mountain
  Lee is e up South Mountain
  

Extended Regular Expression Metacharacters

• Wildcard matching
  • | -- matches either the character before or after

    a|b|c
    

  • () -- groups characters for more complicated expressions

    Lehi(gh)*
    

• Quantifiers
  • ? -- match previous character zero or one time
  • + -- match previous character one or more times
  
  
  
• These are available with egrep(1)

Regular Expressions in Emacs

• UNIX editors such as vi and emacs also support regular expressions to search and transform files
• M-C-S will allow you to type a regular expression and interactively find the next one (C-S will move to the next, as with regular search)
• M-X replace-regexp will prompt for a regular expression for matching, and then prompt for the replacement string

Lab exercise (from Quigley)

• Given the file datebook.txt
  • Print all lines containing the string San.
  • Print all lines where the person's first name starts with J.
  • Print all lines ending in 700.
  • Print all lines that don't contain 834.
  • Print all lines where birthdays are in December.
  • Print all lines where the phone number is in the 408 area code.
  • Print all lines containing an uppercase letter, followed by four lowercase letters, a comma, a space, and one uppercase letter.
  • Print lines where the last name begins with a K or k.
  • Print lines preceded by a line number where the salary (last field) is a six-figure number.
  • Print lines containing Lincoln or lincoln.