Revision $Id: webjob-run-chain.base,v 1.3 2008/01/16 01:16:15 klm Exp $ Purpose This recipe demonstrates a simple script-based scheme that can be used within a WebJob framework to execute an arbitrary number of links in a job chain. Motivation The motivation for this recipe was to create a simple scheme that can be used to create a lifeline during time-sensitive, man-in-the-loop, or risky tasks. An example of a time-sensitive task would be performing a live investigation where multiple jobs need to be carried out faster than the regular check-in period (e.g., hourly, daily, etc.). An example of a man-in-the-loop task would be one where job B can't run until job A has finished and its results have been blessed. An example of a risky task would be updating a WebJob client's config files -- if that operation fails, the client could be cut-off from the server, which would be really inconvenient if you had no other remote access to the client. Requirements Cooking with this recipe requires an operational WebJob server. If you do not have one of those, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. The latest source distribution is available here: http://sourceforge.net/project/showfiles.php?group_id=40788 Each client must be running UNIX and have basic system utilities and WebJob (1.7.0 or higher) installed. The server must be running UNIX and have basic system utilities, PaD utilities, and WebJob installed. The commands presented throughout this recipe were designed to be executed within a Bourne shell (i.e., sh or bash). This recipe assumes that you have read and implemented the following recipes: http://webjob.sourceforge.net/Files/Recipes/webjob-run-periodic.txt In particular, you need to know how to schedule a oneshot job. Time to Implement Assuming that you have satisfied all the requirements/prerequisites, this recipe should take less than 30 minutes to implement. Solution The solution is to have the clients download and run the chain script. The following steps describe how to implement this solution. 1. Set WEBJOB_CLIENT and WEBJOB_COMMANDS as appropriate for your server. Next, extract the chain and chain-worker scripts at the bottom of this recipe, and install them to the appropriate commands directory. If you want these scripts to be bound to a particular client, set WEBJOB_CLIENT as appropriate before running the following commands. Once the files are in place, set their ownership and permissions to 0:0 and mode 644, respectively. # WEBJOB_CLIENT=common # WEBJOB_COMMANDS=/webjob/profiles/${WEBJOB_CLIENT}/commands # sed -e '1,/^--- chain ---$/d; /^--- chain ---$/,$d' webjob-run-chain.txt > chain # sed -e '1,/^--- chain-worker ---$/d; /^--- chain-worker ---$/,$d' webjob-run-chain.txt > chain-worker # TARGETS="chain chain-worker" # for TARGET in ${TARGETS} ; do cp ${TARGET} ${WEBJOB_COMMANDS}/ && chmod 644 ${WEBJOB_COMMANDS}/${TARGET} && chown 0:0 ${WEBJOB_COMMANDS}/${TARGET} ; done If you are using DSV, be sure to sign the newly installed scripts. The following example shows how this might be done. # webjob-dsvtool -s -k chain chain-worker If security is a top priority, you should sign these scripts on an isolated system, and manually move them to the WebJob server (e.g., via USB thumb drive). If you are OK with storing your signing key on the WebJob server, be sure to use a key that is protected with a strong passphrase. When finished, you should have the following files in the commands directory: chain chain.sig chain-worker chain-worker.sig 2. Test out the chain by running the following job from one of your clients: $ webjob -e -f upload.cfg chain -d 60 -l 10 This should result in the client executing 10 jobs in a 600-second interval (one job every 60 seconds). No real work will be done since the chain script contains no default jobs. 3. When the need arises to do some real work, uncomment the following line in the chain script. --- snip --- ... #${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_CONFIG} chain-worker & ... --- snip --- Next, add jobs to the chain-worker script as needed. If you are using DSV, be sure to re-sign both scripts. Note that if this step is not done, the chain will not work. Finally, schedule a oneshot job in the periodic script (e.g., hourly, daily, etc.) to execute the chain. Closing Remarks If you are using DSV, you should avoid editing the chain script. This will help prevent broken chains, which would happen if a client requested an unsigned chain job. Editing the chain-worker script, on the other hand, is fairly safe to do (even if the administrator fails to sign that particular script) because it would only result in a failed job (i.e., not a broken chain). Credits This recipe was brought to you by Klayton Monroe. Appendix 1 --- chain --- #!/bin/sh ###################################################################### # # $Id: chain,v 1.4 2008/01/16 01:12:37 klm Exp $ # ###################################################################### # # Copyright 2006-2007 The WebJob Project, All Rights Reserved. # ###################################################################### # # Purpose: Execute an arbitrary number of links in a job chain. # ###################################################################### IFS=' ' PATH=/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin PROGRAM=`basename $0` HOSTNAME=`hostname | awk -F. '{print $1}'` ###################################################################### # # Usage # ###################################################################### Usage() { echo 1>&2 echo "Usage: ${PROGRAM} [-c count] [-d delay] [-f config] [-H webjob-home] -l links" 1>&2 echo 1>&2 exit 1 } ###################################################################### # # Main # ###################################################################### CHAIN_COUNT=1 CHAIN_LINKS= CHAIN_DELAY=300 CONFIG=upload.cfg while getopts "c:d:f:l:H:" OPTION ; do case "${OPTION}" in c) CHAIN_COUNT="${OPTARG}" ;; d) CHAIN_DELAY="${OPTARG}" ;; f) CONFIG="${OPTARG}" ;; l) CHAIN_LINKS="${OPTARG}" ;; H) WEBJOB_HOME="${OPTARG}" ;; *) Usage ;; esac done if [ ${OPTIND} -le $# ] ; then Usage fi if [ -z "${CHAIN_LINKS}" ] ; then Usage fi MY_CHAIN_REGEXP="[0-9]+" echo "${CHAIN_COUNT}" | egrep "${MY_CHAIN_REGEXP}" > /dev/null 2>&1 if [ $? -ne 0 ] ; then # The value is not valid. exit 2; fi echo "${CHAIN_DELAY}" | egrep "${MY_CHAIN_REGEXP}" > /dev/null 2>&1 if [ $? -ne 0 ] ; then # The value is not valid. exit 3; fi echo "${CHAIN_LINKS}" | egrep "${MY_CHAIN_REGEXP}" > /dev/null 2>&1 if [ $? -ne 0 ] ; then # The value is not valid. exit 4; fi PATH=${PATH}:${WEBJOB_HOME=/usr/local/webjob}/bin export PATH WEBJOB_CONFIG=${WEBJOB_HOME}/etc/${CONFIG} if [ ! -f ${WEBJOB_CONFIG} -o ! -r ${WEBJOB_CONFIG} ] ; then exit 5; fi ###################################################################### # # Insert custom/emergency jobs here. # # NOTE: If this script is digitally signed using DSV and you want to # make changes, you must re-sign the script before the next client # request. If that is not done, the chain for that client will be # broken (assuming the client is checking DSV signatures). Therefore, # the recommended approach is to leave this script alone and do all # work from a script called chain-worker. # ###################################################################### #${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_CONFIG} chain-worker & ###################################################################### # # Conditionally execute the next link in the chain. # ###################################################################### #CHAIN_LINKS=1 # To break a running chain, uncomment this variable. if [ ${CHAIN_COUNT} -lt ${CHAIN_LINKS} ] ; then CHAIN_COUNT=`expr ${CHAIN_COUNT} + 1` ( sleep ${CHAIN_DELAY} && ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_CONFIG} chain -H ${WEBJOB_HOME} -f ${CONFIG} -d ${CHAIN_DELAY} -l ${CHAIN_LINKS} -c ${CHAIN_COUNT} ) & fi --- chain --- Appendix 2 --- chain-worker --- #!/bin/sh ###################################################################### # # $Id: chain-worker,v 1.1.1.1 2007/10/18 18:25:12 klm Exp $ # ###################################################################### # # Copyright 2006-2007 The WebJob Project, All Rights Reserved. # ###################################################################### # # Purpose: Execute arbitrary tasks as part of a job chain. # ###################################################################### IFS=' ' PATH=/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin PROGRAM=`basename $0` HOSTNAME=`hostname | awk -F. '{print $1}'` ###################################################################### # # GetHostGroups # ###################################################################### GetHostGroups() { #################################################################### # # To associate a client with a particular group, add it's hostname # to the appropriate list. If you want to add a new group, then # make sure you update MY_GROUPS. Note: fully qualified hostnames # aren't supported by this implementation. # #################################################################### MY_GROUP1="" MY_GROUP2="" MY_GROUPS="MY_GROUP1 MY_GROUP2" #################################################################### # # Figure out which groups this host belongs to. # #################################################################### MY_HOST=`hostname | awk -F. '{print $1}'` MY_LIST="" for GROUP in ${MY_GROUPS} ; do eval HOSTS=\$${GROUP} for HOST in ${HOSTS} ; do if [ "${HOST}"X = "${MY_HOST}"X ] ; then MY_LIST="${MY_LIST} ${GROUP}" fi done done echo ${MY_LIST} } ###################################################################### # # RunJobs # ###################################################################### RunJobs() { #################################################################### # # Common jobs. This area is for jobs that should be executed by all # clients. # #################################################################### : # REPLACE WITH ONE OR MORE COMMON JOBS #################################################################### # # Custom jobs. This area is for jobs that should be executed by all # clients belonging to one of the pre-defined groups. Group members # are listed by hostname. # #################################################################### for GROUP in `GetHostGroups` ; do case "${GROUP}" in MY_GROUP1) : # REPLACE WITH ONE OR MORE MY_GROUP1 JOBS ;; MY_GROUP2) : # REPLACE WITH ONE OR MORE MY_GROUP2 JOBS ;; esac done } ###################################################################### # # Usage # ###################################################################### Usage() { echo 1>&2 echo "Usage: ${PROGRAM} [-H webjob-home]" 1>&2 echo 1>&2 exit 1 } ###################################################################### # # Main # ###################################################################### while getopts "H:" OPTION ; do case "${OPTION}" in H) WEBJOB_HOME="${OPTARG}" ;; *) Usage ;; esac done if [ ${OPTIND} -le $# ] ; then Usage fi PATH=${PATH}:${WEBJOB_HOME=/usr/local/webjob}/bin RunJobs --- chain-worker ---