Scripting 101

About writing shell scripts and making the most of your shell
Forum rules
Topics in this forum are automatically closed 6 months after creation.
Locked
mph426

Scripting 101

Post by mph426 »

Hi all,

In the spirit of this thread, I thought it might be nice to take a simple script and build on it. This will allow us to learn as we go.

For the first example I've taken a script that I wrote a few years ago that searches for a text string, inserted some comments on what's going on in the program. This is a VERY simple start. From there I'll be adding features to "build" it to something a bit more complex and making it more usable.

This example is find-string.sh. As is, it searches ALL files under the current directory for a specified "string" of text. This is the the most basic start so, you more experienced scripters don't be put off by it's simplicity, we'll build it as we go.

A little background. This is my first opportunity to use a debian based distro. I cut my teeth on IBM's AIX operating system. I've since administered Sun Solaris, Silicon Grahics IRIX, SCO Unix, Red Hat and other RPM based Linux distros. I gotta say, this distro rocks!!! Since my start was with AIX (back in the '80s), I use the korn shell (ksh) almost exclusively. There are subtle differences in the function output, amongst other idiosyncrasies that over the years I've become accustomed to. For this example I'll be using bash.

I'll try to put more than enough comments in the code to explain what's going on. I don't mean to insult any ones intelligence, just trying to make it easy for the beginners to catch on to. That being said, I'll try to break down each statement as we go. This will probably make for more comments than code, but what the hay! :D

P.S. I don't get as much time to peruse the site as I'd like, so understand if it takes me a day or two to respond.

Here's the code;

Code: Select all

#! /bin/bash
# ^ the shebang.  What this really does is spawn a new instance of the bash
# shell to run the program in.  Otherwise, the current instance (the window 
# you're running it in) is susceptible to any problems that may occur. This 
# can be very important on console based systems. Unlike DOS a window,
# output from  a program can be interpreted and run. This could cause some
# undesirable issues.  The shebang is not absolutely 100% necessary, simple
# scripts will work fine without it.  However the more intense the program the 
# greater the need for it.  It can NEVER hurt to have it.
###########################################################################
# MPH (2010/11/19) 
# find-string.sh
###########################################################################
# Purpose:  Search all files under the current directory for a string of 
#           text.
# Note: This is a very basic program on which we'll build.  Future              
#       enhancements will include error checking, argument handleing, file      
#       specification, search for whole words or partial and more... As I      
 #       think of them ;) Suggestions are always welcome.
###########################################################################     
# Statements:
# find: The "./" tells it to use the current directory. The "-type f" tells
# it to only search for files.  
#
# The "|" (pipe sign) tells it to pipe the output through the following command.
#
# while: One could use a "for" statement here, but if there are any spaces      
# in the file names, you may encounter unexpected results.  Without getting    
# into IFS (Input Field Separator) etc...  The while statement works best in
# this case.  Also note that the variables are double quoted ""  It is almost
# always a good practice to do this.
find ./ -type f | while read file 
do
  # i= sets the value of "i" to the output of the command.
  # grep:  -i tells grep to be case insensitive.  Note the `` (back ticks) the
  # command line is encased in, run the command and return it's value.
  # 2> /dev/null outputs the stderr stream to the bit bucket.
  # That being said. 1> = stdio (standard I/O)  2> = stderr (Standard Error)
  i=`grep -i "$1" "$file" 2> /dev/null`
  # if: For one reason or another I find that negating the -z will help reduce
  # false positives.
  if [ ! -z "$i" ]
  then
    echo "$file:"
    echo "$i"
    echo
  fi
done
In the output below we use it to search for the string "grep"

Sample output:

Code: Select all

./find-string.sh grep 

./find-string.sh:
  # grep:  -i tell grep to be case insensitive. 2> /dev/null outputs the stderr
  i=`grep -i "$1" "$file" 2> /dev/null`

Last edited by LockBot on Wed Dec 28, 2022 7:16 am, edited 1 time in total.
Reason: Topic automatically closed 6 months after creation. New replies are no longer allowed.
dagon
Level 7
Level 7
Posts: 1655
Joined: Mon Dec 06, 2010 4:33 am
Location: Kungälv, Sweden
Contact:

Re: Scripting 101

Post by dagon »

Cool project... might as well continue with a man page and make a .deb out of it too!
mph426

Re: Scripting 101

Post by mph426 »

Hi all,

It's time to add some more code. Here we're adding just a quick little command line check to see if any parameters have been supplied. If so, run the program, if not give a brief syntax and description, then exit.

Now, the check that we're adding does no checking of what kind of parameters been given, just that there is one or more there. We'll add the checking later. You'll also notice that the comments, for the most part have been removed from what's already been covered and added to the new content. Also note the file name has changed. Always a good idea to keep iterations along the way. If you ever want / need to go back, you can.

Who knows, one day we may even get to man page. Never done that before... should be interesting.

Here's the code:

Code: Select all

#! /bin/bash
###########################################################################
# MPH (2010/11/19) 
# find-string01.sh
###########################################################################
# Purpose:  Search all files under the current directory for a string of 
#           text.
# Note: This is a very basic program on which we'll build.  Future 
#       enhancements will include argument handling, file specification,
#       search for whole words or partial and more... As I think of them ;)
#       Suggestions are most welcome.
###########################################################################
# Here we'll add a little basic checking.  This will only test if an 
# argument was given.  If not, then give the usage message.
#
# Statements:
# if: Test to see if a text string was given.  If not, give a brief 
# description on it's useage and exit.  The $1 is the first argument given
# after the command itself.  I don't know the limit of parameters / 
# arguments that can be given but I'm sure it's a bunch. ;) Note that We're
# using ! (not) -n (non-zero) for some reason I've had better luck than        
# using -z (zero-lenth). There are times when using either of them in their     
# inteded form will give a false positive.  Personnaly I've found it more       
# accurate negating the statement.  Again, note the back ticks encasing 
# another command, as in the first example, this time displaying the output     
# instead of setting it's value to a variable.                                 
# exit: The exit 0 says to exit the program with no errors.
if [ ! -n "$1" ]
then
  echo
  echo "Usage: `basename $0` string"
  echo
  echo "  Search all files under the current directory including binary"
  echo "  files for a given string of text."
  echo 
  exit 0
fi
# This part you should already know. If not check the earlier post. ;)
find ./ -type f | while read file 
do
  i=`grep -i "$1" "$file" 2> /dev/null`
  if [ ! -z "$i" ]
  then
    echo "$file"
    echo "$i"
    echo
  fi
done
Sample output: Note that I was editing the file while i ran it. It picked up a match in the swap file.

Code: Select all

./find-string01.sh

Usage: find-string01.sh string

  Search all files under the current directory including binary
  files for a given string of text.

>./find-string01.sh grep
./.find-string01.sh.swp
Binary file ./.find-string01.sh.swp matches

./find-string01.sh
  i=`grep -i "$1" "$file" 2> /dev/null`

ChickenPie4Tea

Re: Scripting 101

Post by ChickenPie4Tea »

erm, it's a great idea for a thread here but I'll have to come back when I've learnt the basic basics as it's starting off too complicated for me!!!
mph426

Re: Scripting 101

Post by mph426 »

ChickenPie4Tea,

Copy the script into a directory by itself. Look at the code in gedit or whatever your favorite editor may be. Don't be too concerned with the command, back ticks, brackets etc... Just start to get used to the structure. Read the lines, refer to the comments. They'll start to make some sense.

Stick with it and get what you can out of it. The more you stick with it the more you'll get.
mph426

Re: Scripting 101

Post by mph426 »

Hi all,

Today we'll change up things a bit. We're going to add the getopts statement and change the way that $1 (the first argument) is looked at. Again, if $1 is not present, take an action, this time however, it won't be the usage statement itself, but rather a call to a function that shows the usage. This ties in with the getopts to separate the options from the arguments or parameters, whichever you'd like to call them.

The getops is used to find arguments flaged by the - (minus sign). This makes the rest of the command line easier to parse. Once the arguments and the parameters are out of the way, we can get back to the simple search command. Again, notice that the file name has bumped up to 02.

Here's the code:

Code: Select all

#! /bin/bash
#
# MPH (2010/11/19) 
#
# find-string02.sh
#
###########################################################################
# Purpose:  Find a string of text within all or a specified group of files.
###########################################################################
# Variables:  Put preset variables here.
###########################################################################
# None
###########################################################################
# Functions:  Fuctions and their descriptions go here.
###########################################################################
# usage() A brief help for the program.
###########################################################################
# Note the use of "basename" this strips off the path to the file. I.E. if 
# you called by it's full path "/usr/local/bin/find-string.sh" it will only
# show the program we called.  Very useful for seperating paths from files.
# Also, you can use dirname to strip off the path to the file.
###########################################################################
usage() {
  echo "Usage: `basename $0` string [-h] string"
  echo 
  echo "  -h This help"
  echo
  echo "  Search all files under the current directory including binary"
  echo "  files for a given string of text."
  echo
  exit 0
}
###########################################################################
# Main
###########################################################################
# Get any command line options.  This is a very simple example.  Functions
# can be called, variables set, etc... We'll get more in depth as we go.
#
# Statements: 
# while getopts;  This will parse the given options. enclose the options
# your looking for between the two : (colons)  In this case we're looking
# for h and ?.  You do not need the - (minus) sign as the getops will pick
# that up automatically. 
# case;  This statement is kinda like a if elseif but a bit cleaner.
# esac;  the end of a case statment, case spelled backward, how cleaver.
while getopts :h?: opt
do
  case $opt in
    h) usage ;;
    ?) usage ;;
  esac
done
# Here were changeing the way that $1, or the first argument is being 
# looked at.  If there is no argument given, show the usage.
if [ ! -n "$1" ]
then
  usage
fi
echo
# This part you should already know. If not check the earlier post. ;)
find ./ -type f | while read file 
do
  i=`grep -i "$1" "$file" 2> /dev/null`
  if [ ! -z "$i" ]
  then
    echo
    echo "$file:"
    echo "$i"
  fi
done
echo
mph426

Re: Scripting 101

Post by mph426 »

Hi again,

Like I said in the beginning, I don't get the time I'd like to do this. My work laptop crashed Friday morning while running updates. I've spent all me "free" time since rebuilding it. I'm getting close but have more to do.

Since I've haven't had time to work on the next point release, I thought I'd take this opportunity to give a little personal insight when it comes to script development. I always like to keep iterations of my scripts as I go. My script directory is /util. Scripts that I'm working on go in /util/prog. When working on them I'll create a link from the last known working version to the the eventual name of the file. I.E. the working script is /util/find-string.sh, the link is to /util/prog/find-string02.sh. This is done by the following command.
Note: The -s creates a symbolic link instead of a hard link.

Code: Select all

 ln -s /util/prog/find-string02.sh /util/find-string.sh
In the sub directory prog are the find-string01.sh through the latest version being worked on. That way I can edit /util/prog/find-string[latest version].sh and work on it, while being able to use the last known good version on a day to day basis. If there is a problem and I need to go back for ANY reason. They're all there. Once you're finished with working on the latest version, remove and replace the link to the new version.

It's kind of like keeping a generic version of CVS (Concurrent Versions System) for development of your scripts. This can save you a TON of headaches in the long run.
mph426

Re: Scripting 101

Post by mph426 »

Hi all,

Here's the final release.

In this version we'll add the ability to only look in a given file set. It can be either a single file or a group of file signified by a wildcard.

Here's the code:

Code: Select all

#! /bin/bash
#
# MPH (2010/11/19) 
#
# find-string03.sh
#
###########################################################################
# Purpose:  Find a string of text within all or a specified group of files.
###########################################################################
# Variables:  Put preset variables here.
###########################################################################
# None
###########################################################################
# Functions:  Fuctions and their descriptions go here.
###########################################################################
# usage() A brief help for the program.
###########################################################################
# Note the use of "basename" this strips off the path to the file. I.E. if 
# you called by it's full path "/usr/local/bin/find-string.sh" it will only
# show the program we called.  Very useful for separating paths from files.
# Also, use dirname to strip off the path to the file.
#
# In addition we're gonna add filetype to the usage statement.
###########################################################################
usage() {
  echo "Usage: `basename $0` string [-h] [filetype]"
  echo 
  echo "  -h This help"
  echo
  echo "  If no filetype is given, all files under the current"
  echo "  directory will be searched.  This will include binary"
  echo "  files."
  echo 
  echo "  Note: you can you wildcards, but they must follow normal syntax"
  echo "        I.E. for *.sh you must escape the wildcard \*.sh"
  echo "        to get the true result."
  echo
  exit 0
}
###########################################################################
# Main
###########################################################################
# Get any command line options.  This is a very simple example.  Functions
# can be called, variables set, etc... We'll get more in depth as we go.
#
# This part you should already know. If not chech the earlier post. ;)
while getopts :h?: opt
do
  case $opt in
    h) usage ;;
    ?) usage ;;
  esac
done
if [ ! -n "$1" ]
then
  usage
fi
###########################################################################
# Now, we're looking for another argument.  $2 will be the filetype.  
# There is no real error checking here, be aware of erroneous input.
#
# Statements: 
# if: $2 is not empty, the value is set in $filetype.  Assign output the 
# results of the find statement. If there's none given do the earlier basic 
# find statement.  
###########################################################################
if [ ! -z "$2" ]
then
  filetype="$2"
  output=`find ./ -name "$filetype" -type f`
else
  output=`find ./ -type f`
fi
# echo:  $output and read it one line at a time. This will take into account
# a single file or wild cards.
echo "$output" | while read file 
do
  # now, we're back to the same thing we did in the last release.
  i=`grep -i "$1" "$file"` 2> /dev/null
  if [ ! -z "$i" ]
  then
    echo "$file:"
    echo "$i"
    echo
  fi
done
DrHu

Re: Scripting 101

Post by DrHu »

Also it may help to provide scripting site links, so I'll start
http://www.freeos.com/guides/lsst/
http://steve-parker.org/sh/exitcodes.shtml
http://comp.eonworks.com/scripts/scripts.html

Actually there have been various post referencing sites for hardware software etc that some users/posters have provided
  • And apart from having to keep the links up-to-date/remove dead links; it provides a good resource beyond the user guides..
--maybe a resources web links page is also a good project for Mint, the forum moderators also must have a large collection of such data, that they might consider sharing..
Locked

Return to “Scripts & Bash”