Revision $Id: webjob-manage-cronjob.base,v 1.8 2005/04/20 04:24:10 klm Exp $ Purpose This recipe demonstrates how to add or remove a specified cron job from one or more WebJob clients. Motivation WebJob clients typically check in and do work on an hourly or daily basis. Sometimes, they'll check in as often as every 15 minutes (e.g., heartbeat job). However, checking in more frequently than that may not be practical -- especially for large client populations. This creates a 15-60 minute hole in the admin's ability to schedule higher frequency jobs and get work done on the clients. Since cron has minute resolution, it's a natural choice for managing higher frequency work and filling that hole. However, this requires a modification to each client's crontab. That's not a problem for a few hosts, but if you have tens to hundreds of hosts, it could quickly turn into a real headache. That's where WebJob and this recipe come in. Requirements Cooking with this recipe requires an operational WebJob server. If you do not have one of those, refer to the instructions provided in the README.INSTALL file that comes with the source distribution. The latest source distribution is available here: http://sourceforge.net/project/showfiles.php?group_id=40788 The server must be running UNIX and have basic system utilities, Apache, and WebJob (1.4.0 or higher) installed. Each client must be running UNIX and have basic system utilities and WebJob (1.4.0 or higher) installed. This recipe assumes that you have read and implemented the hourly script from the following recipe: http://webjob.sourceforge.net/Files/Recipes/webjob-run-periodic.txt Time to Implement Assuming that you have satisfied all the requirements/prerequisits, this recipe should take less than one hour to implement. Solution The following steps describe how to implement this solution. 1. Set WEBJOB_CLIENT and WEBJOB_COMMANDS as appropriate for your server. Next, extract the cronjob_manager script at the bottom of this recipe, and install it in the appropriate commands directory. If you want this script to be bound to a particular client, set WEBJOB_CLIENT as appropriate before running the following commands. Once the file is in place, set its ownership and permissions to 0:0 and mode 644, respectively. # WEBJOB_CLIENT=common # WEBJOB_COMMANDS=/var/webjob/profiles/${WEBJOB_CLIENT}/commands # sed -e '1,/^--- cronjob_manager ---$/d; /^--- cronjob_manager ---$/,$d' webjob-manage-cronjob.txt > cronjob_manager # cp cronjob_manager ${WEBJOB_COMMANDS}/ # chmod 644 ${WEBJOB_COMMANDS}/cronjob_manager # chown 0:0 ${WEBJOB_COMMANDS}/cronjob_manager This script has the following usage: Usage: cronjob_manager {-d|--deploy} -c 'command-line' -t 'time-specification' cronjob_manager {-r|--remove} -e 'remove-expression' cronjob_manager {-l|--lsjobs} 2. Decide what your job is going to be. For this recipe, we'll use the following example: * * * * * uptime >> /var/log/uptime.log 2> /dev/null This job runs uptime once a minute and appends any output to /var/log/uptime.log. 3. Test deployment and removal of your job on a small group of test clients. To do this, add the hostnames of each client to MY_GROUP1 (or any available group). Note that group variables are defined in GetHostGroups(), which is part of the hourly script. Assuming your test clients are named host1, host2, and host3, your group variable would look like this: MY_GROUP1="host1 host2 host3" Next, insert the following job under the appropriate group case statement in RunOneShotJobs(): ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg cronjob_manager --deploy -t '* * * * *' -c 'uptime >> /var/log/uptime.log 2> /dev/null' The group case statement should look similar to the one shown here: --- snip --- for GROUP in `GetHostGroups` ; do case "${GROUP}" in MY_GROUP1) : # REPLACE WITH ONE OR MORE MY_GROUP1 JOBS ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg cronjob_manager --deploy -t '* * * * *' -c 'uptime >> /var/log/uptime.log 2> /dev/null' ;; MY_GROUP2) : # REPLACE WITH ONE OR MORE MY_GROUP2 JOBS ;; done --- snip --- Once the oneshot has completed, inspect the WebJob output to ensure that it was successful. The .out file should look similar to this: COMMAND_LINE=uptime >> /var/log/uptime.log 2> /dev/null TIME_SPECIFICATION=* * * * * USER=root --- crontab.bak --- existing jobs... --- crontab.bak --- --- crontab.new --- existing jobs... * * * * * uptime >> /var/log/uptime.log 2> /dev/null --- crontab.new --- This output consists of three parts: runtime variables, a listing of the original crontab, and a listing of the new crontab. The job you deployed should show up at the bottom of the new crontab (i.e., crontab.new). To reverse the process (i.e., remove the cron job), shedule a new oneshot job. Replace the old oneshot with this one: ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg cronjob_manager --remove -e 'uptime >> /var/log/uptime.log 2> /dev/null' Once this oneshot has completed, inspect the WebJob output to ensure that it was successful. The .out file should look similar to this: REMOVE_EXPRESSION=uptime >> /var/log/uptime.log 2> /dev/null USER=root --- crontab.bak --- existing jobs... * * * * * uptime >> /var/log/uptime.log 2> /dev/null --- crontab.bak --- --- crontab.new --- existing jobs... --- crontab.new --- 4. Once you're satisfied that the deployment/removal process works as advertised, you can schedule a new oneshot job that'll run on all clients. After that, sit back, wait for the output to roll in, and check it to ensure that there weren't any complications. Closing Remarks Modifying crontabs can be risky -- especially if you rely on cron for mission critical operations. These risks can be reduced or eliminated through solid testing and a pinch disaster recovery planning -- i.e., you'd better make backups of all your crontabs *before* you go messing with them. This can be done as follows: ${WEBJOB_HOME}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg cronjob_manager --lsjobs As a defensive practice, you should get in the habit of enclosing the command-line ('-c'), remove-expression ('-e'), and time-specification ('-t') arguments in single quotes. This will help prevent unexpected variable expansion. Consider the following examples: # sh cronjob_manager --deploy -t '30 0 * * *' -c '${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1' or # sh cronjob_manager --deploy -t '30 0 * * *' -c "${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1" The first command does what you'd expect -- i.e., no variable expansion, and you get the following crontab entry: 30 0 * * * ${WEBJOB_HOME=/opt/local/webjob}/bin/webjob -e -f ${WEBJOB_HOME}/etc/upload.cfg daily > /dev/null 2>&1 The second command, however, encloses the command-line ('-c') argument with double quotes, so its variables are evaluated and replaced with their actual values. Assuming that WEBJOB_HOME was not already defined, the entry that ends up in the crontab would look like this: 30 0 * * * /opt/local/webjob/bin/webjob -e -f /opt/local/webjob/etc/upload.cfg daily > /dev/null 2>&1 Take care to ensure that your remove-expression ('-e') is unique to the job you want to remove. If you specify an expression that matches multiple jobs, you'll end up removing all of them, which is probably not what you wanted. Just remember that the expression you specify is passed to egrep inside the script. Credits This recipe was brought to you by Klayton Monroe. References Appendix 1 --- cronjob_manager --- #!/bin/sh ###################################################################### # # $Id: cronjob_manager,v 1.1 2005/04/20 03:35:56 klm Exp $ # ###################################################################### # # Copyright 2003-2005 The WebJob Project, All Rights Reserved. # ###################################################################### # # Purpose: Deploy or remove cron jobs, and list crontab contents. # ###################################################################### IFS=' ' PATH=/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin PROGRAM=`basename $0` ###################################################################### # # CjmCleanup # ###################################################################### CjmCleanup() { MY_BAK_FILE=crontab.bak MY_NEW_FILE=crontab.new rm -f ${MY_BAK_FILE} ${MY_NEW_FILE} } ###################################################################### # # CjmGetNewCronTab # ###################################################################### CjmGetNewCronTab() { MY_NEW_FILE=crontab.new echo "--- ${MY_NEW_FILE} ---" crontab -l echo "--- ${MY_NEW_FILE} ---" } ###################################################################### # # CjmGetOldCronTab # ###################################################################### CjmGetOldCronTab() { MY_BAK_FILE=crontab.bak echo "--- ${MY_BAK_FILE} ---" crontab -l | tee ${MY_BAK_FILE} echo "--- ${MY_BAK_FILE} ---" } ###################################################################### # # CjmDeployCronJob # ###################################################################### CjmDeployCronJob() { MY_TIME_SPECIFICATION="$1" MY_COMMAND_LINE="$2" MY_BAK_FILE=crontab.bak MY_NEW_FILE=crontab.new cat ${MY_BAK_FILE} > ${MY_NEW_FILE} # NOTE: Don't use egrep here (see CVS log). grep "${MY_COMMAND_LINE}" ${MY_BAK_FILE} > /dev/null || cat - >> ${MY_NEW_FILE} << EOF ${MY_TIME_SPECIFICATION} ${MY_COMMAND_LINE} EOF if [ -r ${MY_NEW_FILE} -a -s ${MY_NEW_FILE} ] ; then crontab ${MY_NEW_FILE} fi } ###################################################################### # # CjmRemoveCronJob # ###################################################################### CjmRemoveCronJob() { MY_REMOVE_EXPRESSION="$1" MY_BAK_FILE=crontab.bak MY_NEW_FILE=crontab.new if [ -r ${MY_BAK_FILE} -a -s ${MY_BAK_FILE} ] ; then if egrep "${MY_REMOVE_EXPRESSION}" ${MY_BAK_FILE} > /dev/null ; then egrep -v "${MY_REMOVE_EXPRESSION}" ${MY_BAK_FILE} > ${MY_NEW_FILE} if [ -r ${MY_NEW_FILE} ] ; then crontab ${MY_NEW_FILE} else echo "${PROGRAM}: File='${MY_NEW_FILE}' Error='File does not exist or is not accessible." 2>&1 fi fi fi } ###################################################################### # # CjmUsage # ###################################################################### CjmUsage() { echo 1>&2 echo "Usage: ${PROGRAM} {-d|--deploy} -c 'command-line' -t 'time-specification'" 1>&2 echo " ${PROGRAM} {-r|--remove} -e 'remove-expression'" 1>&2 echo " ${PROGRAM} {-l|--lsjobs}" 1>&2 echo 1>&2 exit 1 } ###################################################################### # # CjmMain # ###################################################################### CjmMain() { #################################################################### # # Punch in and go to work. # #################################################################### COMMAND_LINE="" REMOVE_EXPRESSION="" TIME_SPECIFICATION="" umask 077 trap "CjmCleanup ; exit 2" 1 2 15 #################################################################### # # Process command line arguments. # #################################################################### if [ $# -lt 1 ] ; then CjmUsage fi case "$1" in -d|--deploy) options="c:t:" ACTION=deploy ;; -l|--lsjobs) ACTION=lsjobs ;; -r|--remove) options="e:" ACTION=remove ;; *) CjmUsage ;; esac shift # Remove the mode argument. while getopts "${options}" OPTION ; do case "${OPTION}" in c) COMMAND_LINE="${OPTARG}" ;; e) REMOVE_EXPRESSION="${OPTARG}" ;; t) TIME_SPECIFICATION="${OPTARG}" ;; *) CjmUsage ;; esac done if [ ${OPTIND} -le $# ] ; then CjmUsage fi #################################################################### # # Do the user's bidding. # #################################################################### case "${ACTION}" in deploy) if [ -z "${COMMAND_LINE}" -o -z "${TIME_SPECIFICATION}" ] ; then CjmUsage fi echo "COMMAND_LINE=${COMMAND_LINE}" echo "TIME_SPECIFICATION=${TIME_SPECIFICATION}" echo "USER=${USER}" CjmGetOldCronTab CjmDeployCronJob "${TIME_SPECIFICATION}" "${COMMAND_LINE}" CjmGetNewCronTab CjmCleanup ;; lsjobs) crontab -l ;; remove) if [ -z "${REMOVE_EXPRESSION}" ] ; then CjmUsage fi echo "REMOVE_EXPRESSION=${REMOVE_EXPRESSION}" echo "USER=${USER}" CjmGetOldCronTab CjmRemoveCronJob "${REMOVE_EXPRESSION}" CjmGetNewCronTab CjmCleanup ;; *) CjmUsage ;; esac } CjmMain "$@" --- cronjob_manager ---