The FTimes Project The HashDig Project The WebJob Project The PaD Project
Location: / Home / WebJob / Man Pages / webjob
Man Pages
WebJob
Home
                                                                                                                                                                                                       

NAME

WebJob - Download and execute a program over HTTP/HTTPS


SYNOPSIS

webjob {-e|--execute} {-f|--file} config[,config[...]] program [options]

webjob {-g|--get-url} [{-f|--file} config[,config[...]]] url

webjob {-h|--hashsum} {-t|--type} digest file

webjob {-r|--run-embedded} {lua|perl} [...]

webjob {-v|--version}

Note: Command line arguments are position dependent, and argument snuggling (e.g., '-ef', '-gf', or '-ht') is not supported. All arguments and their values (if any) must be separated by whitespace. However, any options associated with program, lua, or perl are not bound by these restrictions. Instead, they are bound to the argument syntax for program, lua, or perl, respectively.


DESCRIPTION

WebJob's --execute mode downloads and executes a program in one unified operation. WebJob's runtime behavior is controlled through a config file identified by the --file argument. Among other things, this file identifies a server willing to satisfy WebJob GET requests. Once initialized, WebJob downloads the target program, stores it on the local file system, and executes it in a sub-process. The options, if any, are bound to the target program at invocation. By default, any output generated by the target program is directed to stdout/stderr. However, output may also be directed to a WebJob server through a WebJob PUT request.

WebJob's --get-url mode downloads a target url and writes the output to stdout. This is useful in situations where a file (accessible through the Web) is needed, but the target platform lacks the necessary tools (e.g., curl, wget, etc.) to fetch it.

WebJob's --hashsum mode provides the ability to compute a message digest of the specified file. This is useful in situations where the downloaded executable needs to compute a hash, but the target platform lacks the necessary tools (e.g., md5, md5sum, etc.) to perform that task.

WebJob's --run-embedded mode provides the ability to execute Lua or Perl scripts depending on how WebJob was compiled. This is useful in situations where the target platform lacks the necessary scripting tools (or the native tools are unable) to perform the task at hand. It is also useful in that it provides a cross-platform scripting environment, which allows you to standardize various jobs across multiple platforms.

WebJob's GET/PUT requests may be bound to any available outbound port. This increases the likelihood that distant WebJob servers can be reached in spite of firewalls and filtering routers. WebJob also provides limited proxy support.

When compiled with OpenSSL, WebJob supports HTTPS requests and mutual certificate authentication.

WebJob also has many controls which dictate how it will execute. The following table summarizes what controls apply to each mode of operation. An 'X' indicates that the given control applies to the selected mode.

                          ===== MODES =====
                                      r
                                      u
                                      n
                                      -
                                      e
                          e   g   h   m   v
                          x   e   a   b   e
                          e   t   s   e   r
                          c   -   h   d   s
                          u   u   s   d   i
                          t   r   u   e   o
                          e   l   m   d   n
   ======   CONTROL  ======================
   ClientId               X   .   .   .   .
   DsvCertificateTree     X   .   .   .   .
   DsvVerifySignature     X   .   .   .   .
   GetHookCommandLine     X   .   .   .   .
   GetHookEnable          X   .   .   .   .
   GetHookStatus          X   .   .   .   .
   GetHookSuffix          X   .   .   .   .
   GetTimeLimit           X   .   .   .   .
   HashType               X   .   .   .   .
   Import                 X   .   .   .   .
   OverwriteExecutable    X   .   .   .   .
   Priority               X   .   .   .   .
   PutTimeLimit           X   .   .   .   .
   RunTimeLimit           X   .   .   .   .
   RunType                X   .   .   .   .
   SslBundledCAsFile      X   X   .   .   .
   SslExpectedPeerCN      X   X   .   .   .
   SslMaxChainLength      X   X   .   .   .
   SslPassPhrase          X   X   .   .   .
   SslPrivateKeyFile      X   X   .   .   .
   SslPublicCertFile      X   X   .   .   .
   SslUseCertificate      X   X   .   .   .
   SslVerifyPeerCert      X   X   .   .   .
   StrictControls         X   X   .   .   .
   TempDirectory          X   .   .   .   .
   TimeoutSignal          X   .   .   .   .
   UnlinkExecutable       X   .   .   .   .
   UnlinkOutput           X   .   .   .   .
   UploadOnGetFailure     X   .   .   .   .
   UrlAuthType            X   X   .   .   .
   UrlDownloadLimit       X   X   .   .   .
   UrlGetProxyAuthType    X   X   .   .   .
   UrlGetProxyEnable      X   X   .   .   .
   UrlGetProxyHost        X   X   .   .   .
   UrlGetProxyPassword    X   X   .   .   .
   UrlGetProxyPort        X   X   .   .   .
   UrlGetProxyUsername    X   X   .   .   .
   UrlGetUrl              X   .   .   .   .
   UrlPassword            X   X   .   .   .
   UrlPutProxyAuthType    X   .   .   .   .
   UrlPutProxyEnable      X   .   .   .   .
   UrlPutProxyHost        X   .   .   .   .
   UrlPutProxyPassword    X   .   .   .   .
   UrlPutProxyPort        X   .   .   .   .
   UrlPutProxyUsername    X   .   .   .   .
   UrlPutUrl              X   .   .   .   .
   UrlUsername            X   X   .   .   .


MODES OF OPERATION

The modes of operation described in this section are mutually exclusive. In other words, only one mode may be specified per invocation.

{-e|--execute} {-f|--file} config[,config[...]] program [options]

Use config file settings to download and execute the specified program. The config file (or comma delimited list of files), as specified by the --file argument, may be the name of a regular file or '-', which indicates that configuration information is to be read from stdin. If a list of files is supplied, then the value for stdin may only be specified once. Also, note that path information in the target program argument is not allowed. The options, if any, are bound to the target program at invocation. The various config file controls and syntax rules are described in the CONFIGURATION CONTROLS section of this document.

This mode of operation is roughly split into three stages: GET, RUN, and PUT. Each stage provides an independent timer. In the event that a given stage doesn't complete within the alloted time, WebJob will attempt to abort gracefully. A timeout value of zero means there is no time limit.

{-g|--get-url} [{-f|--file} config[,config[...]]] url

Optionally use config file settings to download the specified url. The config file (or comma delimited list of files), as specified by the --file argument, may be the name of a regular file or '-', which indicates that configuration information is to be read from stdin. If a list of files is supplied, then the value for stdin may only be specified once. The various config file controls and syntax rules are described in the CONFIGURATION CONTROLS section of this document.

URLs must use a scheme of http or https and satisfy the following regular expression:

    scheme://(user(:pass)?@)?host(:port)?(/path(\?query)?)?
{-h|--hashsum} {-t|--type} digest file

Use digest algorithm, as specified by the --type argument, to compute a hash of file. If file is specified as '-', then input will be read from stdin. Currently, the following algorithms are supported: MD5, SHA1, and SHA256. The digest value is not case sensitive. In other words, 'md5' is equivalent to 'MD5'.

{-r|--run-embedded} {lua|perl} [...]

Use the embedded Lua or Perl interpreter to execute or process the lua or perl arguments, which can be combined in any way to form valid command line expression. This mode of operation is only available if embedded Lua or Perl support is compiled into the binary. Note that only one embedded language can be compiled into a given WebJob binary.

{-v|--version}

Display version information and exit.


OPTIONS

--NoUpload

This option disables PUT requests (i.e., uploads) even if a PUT URL has been specified via UrlPutUrl.

--GetTimeLimit

This option is the command line counterpart to the GetTimeLimit control, which is described in the CONFIGURATION CONTROLS Section. If this value is specified in both the config file and on the command line, the config file value will trump the the command line value. Note that this behavior will likely be reversed in a future release.

--PutTimeLimit

This option is the command line counterpart to the PutTimeLimit control, which is described in the CONFIGURATION CONTROLS Section. If this value is specified in both the config file and on the command line, the config file value will trump the the command line value. Note that this behavior will likely be reversed in a future release.

--RunTimeLimit

This option is the command line counterpart to the RunTimeLimit control, which is described in the CONFIGURATION CONTROLS Section. If this value is specified in both the config file and on the command line, the config file value will trump the the command line value. Note that this behavior will likely be reversed in a future release.

--TimeoutSignal

This option is the command line counterpart to the TimeoutSignal control, which is described in the CONFIGURATION CONTROLS Section. If this value is specified in both the config file and on the command line, the config file value will trump the the command line value. Note that this behavior will likely be reversed in a future release.


CONFIGURATION CONTROLS

This section describes the various controls that WebJob recognizes. In general, controls either shape runtime behavior or provide information needed by the application to perform a specific function. Controls and their values, one pair/line, are written to a file having the following format.

    <control> = <value>

All controls are case insensitive, but, in general, their values are not. Comments may occur anywhere on a given line, and must begin with a pound character (i.e., '#'). In any given line, all text to the right of the first comment will be ignored. White space surrounding controls and values is ignored.

CONTROL DESCRIPTIONS

This section describes each control that may be specified, defines what values it may have, and states which modes of operation recognize the control.

ClientId: <client>

Applies to execute.

ClientId is required. It defines the identity of the client for which the GET/PUT requests are being made. Typically, the ClientId and UrlUsername are the same when basic authentication is enabled. However, this is not a requirement.

DsvCertificateTree: <directory>

Applies to execute.

DsvCertificateTree is optional. This control specifies a directory that contains one or more PEM-encoded certificates whose public keys may be used to verify the payload's signature -- invalid and non-PEM-encoded files are silently ignored. Each certificate in the certificate tree is used, in turn, to verify the payload's integrity. The verification process does not stop until a match is found or all certificates have been tried. Note that WebJob will not search for certificates in sub-directories of the specified tree. This control is ignored if DsvVerifySignature is disabled.

For more information on signing and verifying payloads refer to the WebJob-DsvTool man page (i.e., webjob-dsvtool(1)). That document also provides examples of how to create the necessary PEM-encoded keys and certificates.

Note: If DsvCertificateTree does not exist or does not contain at least one valid certificate, the program will abort.

Note: Digital Signature Verification (DSV) support is a compile-time option, which is enabled by default. However, if DSV support is not compiled into the binary, WebJob will abort with an error if you try to use this (or any) DSV control in the config file. To determine if DSV support is compiled into the binary, run the following command:

    webjob --version
DsvVerifySignature: [Y|N]

Applies to execute.

DsvVerifySignature is optional. When enabled ('Y' or 'y'), it instructs the application to verify the signature of the requested payload. Signatures are verified against one or more certificates located in DsvCertificateTree. The default value is 'N'.

If this control is enabled and the payload's signature is missing or invalid, WebJob will abort, and the downloaded payload will not be executed. This helps to protect the client in the case where the WebJob server can no longer be trusted (e.g., because it has been compromised).

Note: Certificate chains and expiration dates are not checked in the verification process. It is the responsibility of the WebJob administrator to periodically manage (i.e., add, remove, update) any certificates installed on the clients.

Note: Digital Signature Verification (DSV) support is a compile-time option, which is enabled by default. However, if DSV support is not compiled into the binary, WebJob will abort with an error if you try to use this (or any) DSV control in the config file. To determine if DSV support is compiled into the binary, run the following command:

    webjob --version
GetHookCommandLine: <command>

Applies to execute.

GetHookCommandLine is required when GetHookEnable is enabled. The value of this control is a command or pipeline that will be executed by the system's command interpreter once the requested file has been downloaded. If GetHookSuffix is defined, a variant of the target program is requested. This variant must be transformed by the specified command to produce the target program. If the command is successful, as defined by GetHookStatus, then the target program will be executed as usual. Otherwise, the target program will not be executed. The tokens '%tp', '%vp', '%pid', and '%s' may be used as place holders in the specified command line. When the hook is activated, these tokens are expanded as the target program, variant program, current Process ID, and suffix, respectively. The character '%' may be used to produce a literal token. For example, '%%tp' would be expanded to '%tp'.

GetHookEnable: [Y|N]

Applies to execute.

GetHookEnable is optional. When enabled ('Y' or 'y'), the caller has the ability to interrupt or hook the normal course of events by running a single system command. This mechanism provides users with the ability veto or bless, as the case may be, subsequent execution of the target program. The default value is 'N'.

GetHookStatus: [0-255]

Applies to execute.

GetHookStatus is required when GetHookEnable is enabled. The value of this control determines the meaning of success for the hook. If the specified value matches the exit status as returned by system(3), then the target program will be executed as usual. Otherwise, the target program will not be executed.

GetHookSuffix: <suffix>

Applies to execute.

GetHookSuffix is optional when GetHookEnable is enabled. The hook suffix is appended to the target (program) name to create a variant name. This variant name takes the place of the target name in the GET request. To maintain order, the hook command must transform the downloaded variant into the target program. If this control is not defined, then the variant and target names will be the same, and no transformation would be required.

GetTimeLimit: [0-31536000]

Applies to execute.

GetTimeLimit is optional. The value of this control specifies how long the program is willing wait before giving up on download (i.e. GET) requests. If GetTimeLimit is non zero and the time limit expires, the program will abort. If GetTimeLimit is not set or is set to zero, no limit will be imposed. The default value is zero.

HashType: [md5|sha1]

Applies to execute.

HashType is optional. This control specifies the hash algorithm that is to be used when calculating message digests. Currently, the following algorithms are supported: MD5, SHA1, and SHA256. If not specified, HashType defaults to MD5. This control is not case sensitive.

Import: <file>

Applies to execute and get-url.

Import is optional. When specified, the directives contained in the file referenced by this control are included in the current configuration. Multiple instances of this control are allowed per file, and recursion is permitted up to three levels deep. Imports may be specified using a relative path.

OverwriteExecutable: [Y|N]

Applies to execute.

OverwriteExecutable is optional. When enabled ('Y' or 'y'), existing executables in the specified TempDirectory will be overwritten by subsequent GET requests. The default value is 'N'.

Priority: [low|below_normal|normal|above_normal|high]

Applies to execute.

Priority is optional. Use it to set the priority of the target process. The environment variable, WEBJOB_PRIORITY, trumps the value of this control when set. No priority is set by default.

On UNIX platforms, the priority values listed above correspond to the following priority levels:

    low .............. 20
    below_normal ..... 10
    normal ...........  0
    above_normal .....-10
    high .............-20

On WINX platforms, the priority values listed above correspond to the following priority classes:

    low .............. IDLE_PRIORITY_CLASS
    below_normal ..... BELOW_NORMAL_PRIORITY_CLASS
    normal ........... NORMAL_PRIORITY_CLASS
    above_normal ..... ABOVE_NORMAL_PRIORITY_CLASS
    high ............. HIGH_PRIORITY_CLASS

Note: If the specified priority can't be set, the target process will abort.

Note: If an environment variable is defined but exceeds the maximum allowed length, it will be rejected.

PutTimeLimit: [0-31536000]

Applies to execute.

PutTimeLimit is optional. The value of this control specifies how long the program is willing wait before giving up on upload (i.e. PUT) requests. If PutTimeLimit is non zero and the time limit expires, the program will abort. If PutTimeLimit is not set or is set to zero, no limit will be imposed. The default value is zero.

RunTimeLimit: [0-31536000]

Applies to execute.

RunTimeLimit is optional. The value of this control specifies how long the program is willing wait for the target application to finish. If RunTimeLimit is non zero and the time limit expires, the program will kill the spawned process and abort. If RunTimeLimit is not set or is set to zero, no limit will be imposed. The default value is zero.

Note: Killing the spawned (i.e., child) process is done via TerminateProcess() and kill() on Windows- and UNIX-based platforms, respectively. This may result in orphaned grandkids in either case. To help prevent this situation, proactively identify processes that block or take a long time to run and make sure that they run as kids. Using the shell's built-in exec or the family of exec() calls can help in this regard.

RunType: [linktest|snapshot]

Applies to execute.

RunType is optional. This control sets a corresponding flag that classifies uploaded data as linktest or snapshot, The value of this control does not affect the format or content of WebJob output. When output is uploaded to a Web server, the value of this control may be used to determine how the data is processed. If not specified, RunType defaults to snapshot.

SslBundledCAsFile: <file>

Applies to execute and get-url.

SslBundledCAsFile is required when SslVerifyPeerCert is enabled. This control specifies the name of a PEM (Privacy Enhanced Mail) encoded file that contains a bundled set of Certificate Authority (CA) certificates. Any validated peer certificate that is signed by one of these CAs will be accepted provided that the SslMaxChainLength and SslExpectedPeerCN checks are also satisfied. SslBundledCAsFile may be specified as a relative path.

SslExpectedPeerCN: <name>

Applies to execute and get-url.

SslExpectedPeerCN is required when SslVerifyPeerCert is enabled. The value of this control represents the peer's expected Common Name (CN). Conventionally, CNs are specified as fully qualified domain names. This control eliminates the need to perform a DNS lookup at the time of certificate validation. This, in turn, may help to prevent attacks involving DNS spoofing.

SslMaxChainLength: [1-10]

Applies to execute and get-url.

SslMaxChainLength is optional when SslVerifyPeerCert is enabled. The value of this control determines how deep a certificate chain may be before it is considered invalid. If not specified, this control defaults to a value of 1.

SslPassPhrase: <passphrase>

Applies to execute and get-url.

SslPassPhrase is optional when SslUseCertificate is enabled. Its value, if specified, is used to decrypt the contents of the client's private key file (see SslPrivateKeyFile). If a passphrase is required to load the private key and this control has not been set, the user will be prompted for one. Naturally, this will cause problems for automated tasks, so keep that in mind.

SslPrivateKeyFile: <file>

Applies to execute and get-url.

SslPrivateKeyFile is required when SslUseCertificate is enabled. This control specifies the name of a PEM (Privacy Enhanced Mail) encoded key file that can be used to sign SSL certificates. SslPrivateKeyFile may be specified as a relative path.

SslPublicCertFile: <file>

Applies to execute and get-url.

SslPublicCertFile is required when SslUseCertificate is enabled. This control specifies the name of a PEM (Privacy Enhanced Mail) encoded certificate to be used during SSL handshakes. SslPublicCertFile may be specified as a relative path.

SslUseCertificate: [Y|N]

Applies to execute and get-url.

SslUseCertificate is optional. When enabled ('Y' or 'y'), it instructs the application to provide client side certificate authentication, if necessary. The default value is 'N'.

SslVerifyPeerCert: [Y|N]

Applies to execute and get-url.

SslVerifyPeerCert is optional. When enabled ('Y' or 'y'), it instructs the application to verify the credentials of the peer server. The default value is 'N'.

StrictControls: [Y|N]

Applies to execute and get-url.

StrictControls is optional. When enabled ('Y' or 'y'), any unknown or unsupported configuration controls will cause the program to abort. When disabled ('N' or 'n'), any unknown or unsupported controls will be silently ignored. The default value is 'N'.

Note: If you want to enforce strict controls within a single config file (or chain of config files), StrictControls must be the first control processed. You may also enable/disable this control multiple times. This allows you to control the strictness of certain sections or starting at a particular point in the entire configuration. Once set, this control takes effect immediately.

Note: For versions 1.7.0 and below, this control was not supported, and the default behavior was to abort on any unknown or unsupported configuration control.

TempDirectory: <directory>

Applies to execute.

TempDirectory is optional. This control specifies a directory in which all temporary files are to be created. If TempDirectory is not specified, the program attempts to obtain its value from the environment. First, it looks for TMPDIR (TEMP for WIN32). Next, it looks for TMP (all platforms). If a suitable environment variable is not found, TempDirectory defaults to /tmp (\temp for WIN32). TempDirectory may be specified as a relative path.

Note: If TempDirectory does not exist, the program will abort.

Note: If an environment variable is defined but exceeds the maximum allowed length, it will be rejected.

TimeoutSignal: <signal-number>

Applies to execute.

TimeoutSignal is optional. When the run timer expires (see RunTimeLimit), the signal specified by this control is sent to the spawned process. This control is rarely needed, and it should be used with care. Sending the wrong signal to a child process may not yield the desired result. However, there are cases when sending the default signal (SIGKILL = 9) is not the preferred course of action either. For example, you may want to run a tool that will perform an orderly shutdown upon receipt of a particular signal (e.g., SIGINT = 2).

Note: This control is UNIX-specific, and its value must be a valid signal number (e.g., 1, 2, 9, etc.). Because the number of signals can vary from OS to OS, you should check the man pages to determine what values are valid for your systems. On many systems, the following command will produce a mapping of signal names to their assigned numbers:

  $ kill -l
UrlAuthType: [basic|none]

Applies to execute and get-url.

UrlAuthType is optional. It identifies what authentication scheme to use when issuing HTTP/HTTPS requests. The value specified by this control applies to any requests involving UrlGetUrl or UrlPutUrl. When UrlAuthType is set to basic, user credentials are base 64 encoded and incorporated into the request header. User credentials specified in the URL take precedence over credentials specified in the UrlUsername and UrlPassword controls. If not specified, UrlAuthType defaults to none.

UrlDownloadLimit: [0-100000000]

Applies to execute and get-url.

UrlDownloadLimit is optional. The value of this control specifies how much data the client is willing to receive for any given HTTP/HTTPS request. If UrlDownloadLimit is not set or is set to zero, no limit will be imposed. The default value is zero.

Note: No limit, here, means that the Content-Length in the response header can be any value up to 4 GB.

UrlGetUrl: <url>

Applies to execute.

UrlGetUrl is required. It defines the scheme, user credentials, host address, port, and CGI application to be used when making requests. If a username/password pair is specified in the URL, that pair takes precedence over the values specified by UrlUsername/UrlPassword, if any. The tokens '%user' and '%pass' may be used as place holders in the specified URL. Before a GET/PUT request is issued, these tokens are replaced with the actual values assigned to UrlUsername and UrlPassword, respectively. URLs must use a scheme of http or https and satisfy the following regular expression:

    scheme://(user(:pass)?@)?host(:port)?/(path(\?query)?)?

Note: This has been relaxed to the following in version 1.7.0:

    scheme://(user(:pass)?@)?host(:port)?(/path(\?query)?)?
UrlPassword: <password>

Applies to execute and get-url.

UrlPassword is optional. It identifies the password to use when accessing a remote Web server. The value specified by this control is used in conjunction with UrlGetUrl and UrlPutUrl unless those controls supply their own username/password pair.

UrlPutUrl: <url>

Applies to execute.

UrlPutUrl is optional. It defines the scheme, user credentials, host address, port, and CGI application to be used when making PUT requests. If a username/password pair is specified in the URL, that pair takes precedence over the values specified by UrlUsername/UrlPassword, if any. In any event, user credentials are only sent when basic authentication has been requested (See UrlAuthType). UrlPutUrl uses the same syntax as UrlGetUrl.

UrlUsername: <username>

Applies to execute and get-url.

UrlUsername is optional. It identifies the username to use when accessing a remote Web server. The value specified by this control is used in conjunction with UrlGetUrl and UrlPutUrl unless those controls supply their own username/password pair.

UnlinkExecutable: [Y|N]

Applies to execute.

UnlinkExecutable is optional. When enabled ('Y' or 'y'), the downloaded executable is deleted before the program exits. The default value is 'N'.

UnlinkOutput: [Y|N]

Applies to execute.

UnlinkOutput is optional. When enabled ('Y' or 'y'), any output files are unlinked before the program exits. This control has no effect when output data is directed to stdout/stderr. The default value is 'N'.

UploadOnGetFailure: [Y|N]

Applies to execute.

UploadOnGetFailure is optional. When enabled ('Y' or 'y'), the program will attempt to upload its results even if the GET fails. Normally, attempting to upload results after a failed download is futile (e.g., authentication failures). However, there may be times when this behavior is desired. For example, suppose that UrlGetUrl and UrlPutUrl point to different servers. In this scenario, a GET failure does not imply that the PUT will fail too, so by monitoring uploads, you could detect jobs that aren't functioning properly. The default value for this control is 'N'.

Note: If multiple config files are specified on the command line, this control only applies to the last one.

UrlGetProxyAuthType: [basic|none]

Applies to execute and get-url.

UrlGetProxyAuthType is optional when UrlGetProxyEnable is enabled. It identifies what authentication scheme to use when issuing HTTP/HTTPS GET requests that must first pass through an outbound proxy server. When UrlGetProxyAuthType is set to basic, user credentials are base64 encoded and incorporated into the proxy request header. If not specified, UrlGetProxyAuthType defaults to 'none'.

UrlGetProxyEnable: [Y|N]

Applies to execute and get-url.

UrlGetProxyEnable is optional. When enabled ('Y' or 'y'), the client submits its GET request to a proxy server that must be specified using the UrlGetProxyHost control. All other controls in the GET-Proxy group are optional. The default value is 'N'.

UrlGetProxyHost: [<hostname>|<IP>]

Applies to execute and get-url.

UrlGetProxyHost is required when UrlGetProxyEnable is enabled. The value of this control represents the proxy server's hostname or IP address. Generally, hostnames should be specified as fully qualified domain names.

UrlGetProxyPassword: <password>

Applies to execute and get-url.

UrlGetProxyPassword is optional when UrlGetProxyEnable is enabled. It identifies the password to use when issuing HTTP/HTTPS GET requests that must first pass through an outbound proxy server.

UrlGetProxyPort: <port>

Applies to execute and get-url.

UrlGetProxyHost is optional when UrlGetProxyEnable is enabled. It identifies the port to use when issuing HTTP/HTTPS GET requests that must first pass through an outbound proxy server. The default value is '8080'.

UrlGetProxyUsername: <username>

Applies to execute and get-url.

UrlGetProxyUsername is optional when UrlGetProxyEnable is enabled. It identifies the username to use when issuing HTTP/HTTPS GET requests that must first pass through an outbound proxy server.

UrlPutProxyAuthType: [basic|none]

Applies to execute.

UrlPutProxyAuthType is optional when UrlPutProxyEnable is enabled. It identifies what authentication scheme to use when issuing HTTP/HTTPS PUT requests that must first pass through an outbound proxy server. When UrlPutProxyAuthType is set to basic, user credentials are base64 encoded and incorporated into the proxy request header. If not specified, UrlPutProxyAuthType defaults to 'none'.

UrlPutProxyEnable: [Y|N]

Applies to execute.

UrlPutProxyEnable is optional. When enabled ('Y' or 'y'), the client submits its PUT request to a proxy server that must be specified using the UrlPutProxyHost control. All other controls in the PUT-Proxy group are optional. The default value is 'N'.

Note: All controls in the PUT-Proxy group will be ignored if UrlPutUrl has not been defined.

UrlPutProxyHost: [<hostname>|<IP>]

Applies to execute.

UrlPutProxyHost is required when UrlPutProxyEnable is enabled. The value of this control represents the proxy server's hostname or IP address. Generally, hostnames should be specified as fully qualified domain names.

UrlPutProxyPassword: <password>

Applies to execute.

UrlPutProxyPassword is optional when UrlPutProxyEnable is enabled. It identifies the password to use when issuing HTTP/HTTPS PUT requests that must first pass through an outbound proxy server.

UrlPutProxyPort: <port>

Applies to execute.

UrlPutProxyHost is optional when UrlPutProxyEnable is enabled. It identifies the port to use when issuing HTTP/HTTPS PUT requests that must first pass through an outbound proxy server. The default value is '8080'.

UrlPutProxyUsername: <username>

Applies to execute.

UrlPutProxyUsername is optional when UrlPutProxyEnable is enabled. It identifies the username to use when issuing HTTP/HTTPS PUT requests that must first pass through an outbound proxy server.


RETURN VALUES

Upon successful completion, a value of 0 (XER_OK) is returned. Otherwise, one of the following error codes is returned:

  • 1 = XER_Usage

  • 2 = XER_Abort

  • 3 = XER_BootStrap

  • 4 = XER_ProcessArguments

  • 5 = XER_ReadProperties

  • 6 = XER_CheckDependencies

  • 7 = XER_Configure

  • 8 = XER_GetStage

  • 9 = XER_RunStage

  • 10 = XER_PutStage

  • 11 = XER_MultiStage

  • 12 = XER_GetUrl

  • 13 = XER_CloneProperties

  • 20 = XER_Cleanup

  • 27 = XER_Cleanup_Configure (XER_Cleanup + XER_Configure)

  • 28 = XER_Cleanup_GetStage (XER_Cleanup + XER_GetStage)

  • 29 = XER_Cleanup_RunStage (XER_Cleanup + XER_RunStage)

  • 30 = XER_Cleanup_PutStage (XER_Cleanup + XER_PutStage)

  • 31 = XER_Cleanup_MultiStage (XER_Cleanup + XER_MultiStage)

If the command line does not conform to the required syntax, a value of 1 (XER_Usage) is returned. A value of 2 (XER_Abort) means the program encountered a critical or unexpected error and had to abort immediately. The remaining exit codes roughly indicate which subsystem encountered the fatal error. If the error code is XER_MultiStage, then multiple stages (e.g., GET/RUN/PUT) failed.


FILES

WebJob Configuration File

The config file specified on the command line contains a set of controls that are used to configure WebJob's runtime behavior. For more details on what controls may be specified, see the CONFIGURATION CONTROLS section of this document.

Target Program

The target program specified on the command line is retrieved from the WebJob server and written to the local file system. The exact location varies depending on the value of TempDirectory. When UnlinkExecutable is enabled, the program is deleted before WebJob exits. However, if WebJob aborts, this file may not be deleted.

WebJob Output Files (webjob_<time>_<nonce>[.d]/webjob.<suffix>)

The values for <time>, <nonce>, and <suffix> are the job epoch in seconds, a random hex string used to help prevent name collisions and guessing attacks, and one of {out,err,env}, respectively. The suffix is used to associate a given file with its corresponding output stream -- i.e., std{out,err,env}. When UrlPutUrl has been set, the sub-process writes its data to the named out and err streams. Once the sub-process has finished, WebJob writes information regarding environment settings and exit status to the env stream. When UnlinkOutput is enabled, these files are deleted after they have been uploaded. However, if WebJob aborts or the upload fails, these files may not be deleted. The ``.d'' extension is used to tag the work directory as done. When WebJob starts, it creates an untagged work directory. Before WebJob deletes any files and exits, it attempts to tag (i.e., rename) the work directory as done. This is an external indicator that the job has finished and it's OK to remove the directory.

Public Certificate and Private Key

When SslUseCertificate has been enabled, WebJob expects to find certificate and key files in the locations specified by SslPublicCert and SslPrivateKey controls, respectively. If these files do not exist or have the proper format, WebJob will abort.

Bundled Certificate Authorities

When SslUseCertificate has been enabled, WebJob expects to find a bundled certificate authorities file in the location specified by SslBundledCAs control. If this file does not exist or have the proper format, WebJob will abort.


EXAMPLES

The following examples are intended to demonstrate different ways of configuring and using WebJob. Any text encapsulated between '--- XXX ---' delimiters represents config file content.

Example 1. Download and execute a program over HTTP -- output goes to stdout/stderr

This example demonstrates how to download and run ps(1) with output being directed to stdout/stderr. It is assumed that a WebJob server has already been configured to serve the appropriate ps(1) binary for the system in question. Also, assume that the following information has been provided.

    ClientId = client_1
    URL = http://webjob.server.net/cgi-client/nph-webjob.cgi
    Username = client_1
    Password = access

The simplest configuration file that will carry out this task is as follows:

    --- webjob.cfg ---
    ClientId=client_1
    UrlGetUrl=http://client_1:access@webjob.server.net/cgi-client/nph-webjob.cgi
    UrlAuthType=basic
    --- webjob.cfg ---

or, alternatively

    --- webjob.cfg ---
    ClientId=client_1
    UrlGetUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
    UrlUsername=client_1
    UrlPassword=access
    UrlAuthType=basic
    --- webjob.cfg ---

Once the config file has been created, WebJob may be invoked as follows:

    webjob --execute --file webjob.cfg ps -aux

or in short form:

    webjob -e -f webjob.cfg ps -aux

WebJob will also accept the config file via stdin as follows:

    webjob -e -f - ps -aux < webjob.cfg

or

    cat webjob.cfg | webjob -e -f - ps -aux

WebJob stores the target program in a job-specific directory (webjob_<time>_<nonce>) under TempDirectory. In this case, TempDirectory hasn't been specified, so data will be written to the default TempDirectory, which is '/tmp' or '\temp' for UNIX or Windows environments, respectively. To automatically remove the target program and the sub-directory in which it resides after job completion, enable UnlinkExecutable.

Example 2. Download and execute a program over HTTP -- output is uploaded

Suppose that you want to upload the output of ps(1) to a WebJob server instead of stdout/stderr. Given that that the URL from the previous example may be used for both GET and PUT requests, the new config file becomes:

    --- webjob.cfg ---
    ClientId=client_1
    UrlGetUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
    UrlPutUrl=http://webjob.server.net/cgi-client/nph-webjob.cgi
    UrlUsername=client_1
    UrlPassword=access
    UrlAuthType=basic
    --- webjob.cfg ---

This job may be invoked as follows:

    webjob -e -f webjob.cfg ps -aux

This time, WebJob will create three files, webjob.{out,err,env}, in a job-specific directory (webjob_<time>_<nonce>) under TempDirectory. These files contain the output data that will be uploaded to the WebJob server in a PUT request. To automatically remove these files after they have been uploaded, enable UnlinkOutput. To automatically remove these files and the sub-directory in which they reside, enable both UnlinkOutput and UnlinkExecutable.

Example 3. Download and execute a program over HTTPS -- explore advanced options

Suppose that you want to run netstat(1) over a secure, authenticated channel (i.e., SSL with mutual certificate authentication). Suppose you also want to delete the target program and its associated output files upon job completion. Assuming that the WebJob server is properly configured, then the information provided below is sufficient to create the following config file.

    GET/PUT URL = https://webjob.server.net:443/cgi-client/nph-webjob.cgi
    SSL Public Cert File = /usr/local/webjob/ssl/mycrt.pem
    SSL Private Key File = /usr/local/webjob/ssl/mykey.pem
    SSL Bundled CAs File = /usr/local/webjob/ssl/CAs.pem
    SSL PassPhrase = flimflam
    Certificate Chain Length = 2
    Web Resource Validates Client Certificates = Y
    --- webjob.cfg ---
    ClientId=client_1
    UrlGetUrl=https://webjob.server.net:443/cgi-client/nph-webjob.cgi
    UrlPutUrl=https://webjob.server.net:443/cgi-client/nph-webjob.cgi
    UrlUsername=client_1
    UrlPassword=access
    UrlAuthType=basic
    UnlinkOutput=Y
    UnlinkExecutable=Y
    SslUseCertificate=Y
    SslPublicCertFile=/usr/local/webjob/ssl/mycrt.pem
    SslPrivateKeyFile=/usr/local/webjob/ssl/mykey.pem
    SslPassPhrase=flimflam
    SslVerifyPeerCert=Y
    SslBundledCAsFile=/usr/local/webjob/ssl/CAs.pem
    SslExpectedPeerCN=webjob.server.net
    SslMaxChainLength=2
    --- webjob.cfg ---

This job may be invoked as follows:

    webjob -e -f webjob.cfg netstat -an

Example 4. Compute the MD5 hash of /bin/ps

To compute the MD5 hash of /bin/ps, run the following command:

    webjob --hashsum --type md5 /bin/ps

or in short form:

    webjob -h -t md5 /bin/ps

Example 5. Compute the SHA1 hash of /bin/ps

To compute the SHA1 hash of /bin/ps, run the following command:

    webjob --hashsum --type sha1 /bin/ps

or in short form:

    webjob -h -t sha1 /bin/ps

Example 6. Use GetHook to unzip gzipped executables

This example demonstrates how to download and run a gzipped copy of ps(1). It is assumed that a WebJob server has already been configured to serve unzipped and zipped copies of ps(1) as ps and ps.gz, respectively.

The first step is to enable GetHook in the client's config file and define the appropriate hook command, suffix, and return status. The following config file snippet shows how this is done.

    --- webjob.cfg ---
    ...
    GetHookEnable=Y
    GetHookCommandLine=gzip -d %vp
    GetHookStatus=0
    GetHookSuffix=.gz
    ...
    --- webjob.cfg ---

In general, any valid command or pipeline may be specified for the hook as long as it produces a valid ps executable. When the following WebJob command is issued, the hook modifies the GET request such that ps.gz is requested. After ps.gz has been downloaded, the hook executes the specified command. In this case, '%vp' will be expanded to 'ps.gz'. If the hook command returns a zero status, then the operation is deemed successful and normal WebJob execution ensues. If the return status is not zero, the target command will not be executed.

    webjob -e -f webjob.cfg ps -aux

Example 7. Use GetHook to verify GPG signed executables

This example demonstrates how to download and run a GPG signed copy of ps(1). It is assumed that a WebJob server has already been configured to serve unsigned or signed copies of ps(1) as ps or ps.gpg, respectively.

The first step is to enable GetHook in the client's config file and define the appropriate hook command, suffix, and return status. The following config file snippet shows how this is done.

    --- webjob.cfg ---
    ...
    GetHookEnable=Y
    GetHookCommandLine=gpg --batch --decrypt %vp > %tp
    GetHookStatus=0
    GetHookSuffix=.gpg
    ...
    --- webjob.cfg ---

When the following WebJob command is issued, the hook modifies the GET request such that ps.gpg is requested. After ps.gpg has been downloaded, the hook executes the specified command. In this case, '%tp' and '%vp' will be expanded to 'ps' and 'ps.gpg', respectively. If the hook command returns a zero status, then the operation is deemed successful and normal WebJob execution ensues. If the return status is not zero, the target command will not be executed.

    webjob -e -f webjob.cfg ps -aux

Example 8. Use GetHook to log WebJob activity

This example demonstrates how to log, using logger(1), the fact that WebJob downloaded and attempted to run ps(1). It is assumed that a WebJob server has already been configured to serve copies of ps(1) as ps.

The first step is to enable GetHook in the client's config file and define the appropriate hook command and return status. The following config file snippet shows how this is done. Note how the suffix is commented out -- it could be left blank also.

    --- webjob.cfg ---
    ...
    GetHookEnable=Y
    GetHookCommandLine={ logger -p user.alert -t "webjob[%pid]" "%vp ---> %tp" ; true ; }
    GetHookStatus=0
    #GetHookSuffix=
    ...
    --- webjob.cfg ---

When the following WebJob command is issued, the hook does not modify the GET request because a suffix was not defined. After ps has been downloaded, the hook executes the specified command. In this case, '%tp', '%vp', and '%pid' will be expanded to 'ps', 'ps', and the current Process ID (PID), respectively. The specified command always returns a zero status due to true(1), so normal WebJob execution ensues.

    webjob -e -f webjob.cfg ps -aux

Example 9. Download a file and write it to stdout

To download the file test.cfg located on www.somewhere.net and write it to stdout, run the following command:

    webjob --get-url http://www.somewhere.net/test.cfg

or in short form:

    webjob -g http://www.somewhere.net/test.cfg

Example 10. Download a file through a proxy server and write it to disk

This example is similar to the previous one except that you must now include a config file. The purpose of the config file is to enable proxy support and to specify the server's address and port. Given that the proxy server's address is 192.168.1.1, its port is 8888, and it requires no authentication, the following config file would be sufficient to perform the task:

    --- webjob.cfg ---
    UrlGetProxyEnable=Y
    UrlGetProxyHost=192.168.1.1
    UrlGetProxyPort=8888
    --- webjob.cfg ---

To download the file test.cfg located on www.somewhere.net through the specified proxy server and write it to a file called test.cfg on disk, run the following command:

    webjob --get-url --file webjob.cfg http://www.somewhere.net/test.cfg > test.cfg

or in short form:

    webjob -g -f webjob.cfg http://www.somewhere.net/test.cfg > test.cfg

Example 11. Execute simple Perl one-liners

This example demonstrates how to use execute a simple Perl one-liner script to URL-decode a string.

    echo %74%65%73%74 | webjob -r perl -p -e 's/%([0-9A-Fa-f]{2})/pack("C", hex($1))/seg;'

This example demonstrates how to execute a simple Perl one-liner script to generate a sequence of numbers between 0 and 255.

    webjob -r perl -e 'foreach $i (0..255) { printf("%d\n", $i); }'

This example demonstrates how to execute a simple Perl one-liner script to insert a comment above a line of selected text in a config file called test.cfg.

    webjob -r perl -p -i -e 's/^(DsvVerifySignature=.*)$/# No comment.\n$1/;' test.cfg


SEE ALSO

openssl(1), webjob-dsvtool(1)


AUTHOR

Klayton Monroe


HISTORY

WebJob was initially written to assist Incident Response Handlers in their efforts to investigate potentially compromised systems. Often, these Handlers must work around the constraints imposed by the surrounding environment. For example, lack of physical or shell access, untrusted diagnostic programs, lack of encryption, many machines in need of investigation, and so on. Therefore, I felt that Handlers, or their eyes and ears in the field, needed an efficient way to import and run known good diagnostic tools when investigating live systems.

WebJob is lightweight, portable, and easy to use. This makes it a good candidate for establishing a foothold on the target system. Once there, all that is needed to begin diagnostics is a small configuration file and access to a remote WebJob server. When there are many machines to investigate, WebJob can be deployed in parallel, and all harvested output can be directed to and aggregated on a single server. This reduces the amount of manual data processing involved in collecting, tagging, and storing evidence. It also significantly reduces the amount time, effort, and resources needed to arrive at a determination.

Message digest (i.e., --hashsum) support first appeared in WebJob 1.3.0.

GetHook and Jid (Job ID) support first appeared in WebJob 1.4.0.

Digital Signature Verification (DSV) support and server-side GET/PUT triggers first appeared in WebJob 1.6.0.

Proxy support, get URL (i.e., --get-url) support, and dynamic content via server-side GET hooks first appeared in WebJob 1.7.0.

Queueing support via Job Queue Directories (JQD) and embedded Perl scripting (i.e., --run-embedded) first appeared in WebJob 1.8.0.

Embedded Lua scripting (i.e., --run-embedded) first appeared in WebJob 1.9.0.

Copyright 2000-2014 The WebJob Project, All Rights Reserved.
The FreeBSD Project SourceForge Logo KoreLogic, Inc.