Pegasus LSF Job Scripts

The command bsub < ScriptFile will submit the given script for processing. Your script must contain the information LSF needs to allocate the resources your job requires, handle standard I/O streams, and run the job. For more information about flags, type bsub -h or man bsub at the Pegasus prompt. Example scripts and descriptions are below.

You must be a member of a project to submit jobs to it. See Projects for more information.

On submission, LSF will return the jobID which can be used to track your job.

[username@pegasus ~]$ bsub < test.job
Job <4225> is submitted to the default queue <general>.

Example script for a serial Job

test.job

---

#!/bin/bash
#BSUB -J myserialjob
#BSUB -P myproject
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:00
#BSUB -q general
#BSUB -n 1
#BSUB -R "rusage[mem=128]"
#BSUB -B
#BSUB -N
#BSUB -u myemail@miami.edu
#
# Run serial executable on 1 cpu of one node
cd /path/to/scratch/directory
./test.x a b c

Here is a detailed line-by-line breakdown of the keywords and their assigned values listed in this script:

ScriptFile_keywords

---

#!/bin/bash
specifies the shell to be used when executing the command portion of the script.
The default is Bash shell.

#BSUB -J serialjob
assign a name to job. The name of the job will show in the bjobs output.

#BSUB -P myproject
specify the project to use when submitting the job. This is required when a user has more than one project on Pegasus.

#BSUB -e %J.err
redirect std error to a specified file

#BSUB -W 1:00
set wall clock run time limit of 1 hour, otherwise queue specific default run time limit will be applied.

#BSUB -q general
specify queue to be used. Without this option, default 'general' queue will be applied.

#BSUB -n 1
specify number of processors. In this job, a single processor is requested.

#BSUB -R "rusage[mem=128]"
specify that this job requests 128 megabytes of RAM per core. Without this, a default RAM setting will be applied:  1500MB per core

#BSUB -B
send mail to specified email when the job is dispatched and begins execution.

#BSUB -u example@miami.edu
send notification through email to example@miami.edu.

#BSUB -N
send job statistics report through email when job finishes.

Example scripts for parallel jobs

We recommend using Intel MPI unless you have specific reason for using OpenMP.  Intel MPI scales better and has better performance than OpenMP.

Submit parallel jobs to the parallel job queue with -q parallel.

For optimum performance, the default resource allocation on the parallel queue is ptile=16. This requires the LSF job scheduler to allocate 16 processors per host, ensuring all processors on a single host are used by that job. *Without prior authorization, any jobs using a number other than 16 will be rejected from the parallel queue.* Reserve enough memory for your jobs. Memory reservations are per core. Parallel job performance may be affected, or even interrupted, by other badly-configured jobs running on the same host.

Example script for Intel/Intel MPI

testparai.job

---

#!/bin/bash
#BSUB -J mpijob
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:30
#BSUB -q parallel
#BSUB -n 32                             # Request 32 cores
#BSUB -R "span[ptile=16]"               # Request 16 cores per node
#BSUB -R "rusage[mem=128]"              # Request 128MB per core
#

mpiexec foo.exe

foo.exe is the mpi executable name. It can be followed by its own argument list.

Example script for MPI/OpenMP

testparao.job

---

#!/bin/bash
#BSUB -J mpijob
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -W 1:30
#BSUB -q parallel
#BSUB -n 32                             # Request 32 cores
#BSUB -R "span[ptile=16]"               # Request 16 cores per node
#BSUB -R "rusage[mem=128]"              # Request 128MB per core
#

mpiexec --mca btl self,sm,openib foo.exe

The command line is similar to Intel MPI job above. Option --mca self,sm,openib tells OpenMP to use lookback, shared memory, and openib for inter-process communication.