.. _g-scripts:


Serial/Parallel Job Scripts
=======================

The command ``bsub < ScriptFile`` will submit the given script for
processing. Your script must contain the information LSF needs to
allocate the resources your job requires, handle standard I/O streams,
and run the job. For more information about flags, type ``bsub -h`` or
``man bsub`` at the Pegasus prompt. Example scripts and descriptions are
below.

You must be a member of a project to submit jobs to it. See :ref:`g-projects` for more information.

On submission, LSF will return the ``jobID`` which can be used to track
your job.

::

    [username@pegasus ~]$ bsub < test.job
    Job <4225> is submitted to the default queue <normal>.

**Example script for a serial Job**

``test.job``

--------------

.. code:: bash

    #!/bin/bash
    #BSUB -J myserialjob
    #BSUB -P myproject
    #BSUB -o %J.out
    #BSUB -e %J.err
    #BSUB -W 1:00
    #BSUB -q normal
    #BSUB -n 1
    #BSUB -R "rusage[mem=128]"
    #BSUB -B
    #BSUB -N
    #BSUB -u myemail@miami.edu
    #
    # Run serial executable on 1 cpu of one node
    cd /path/to/scratch/directory
    ./test.x a b c

Here is a detailed line-by-line breakdown of the keywords and their
assigned values listed in this script:

``ScriptFile_keywords``

--------------

.. code:: bash

    #!/bin/bash
    specifies the shell to be used when executing the command portion of the script.
    The default is Bash shell.

    #BSUB -J serialjob
    assign a name to job. The name of the job will show in the bjobs output.

    #BSUB -P myproject
    specify the project to use when submitting the job. This is required when a user has more than one project on Pegasus.

    #BSUB -e %J.err
    redirect std error to a specified file

    #BSUB -W 1:00
    set wall clock run time limit of 1 hour, otherwise queue specific default run time limit will be applied.

    #BSUB -q normal
    specify queue to be used. Without this option, default 'normal' queue will be applied.

    #BSUB -n 1
    specify number of processors. In this job, a single processor is requested.

    #BSUB -R "rusage[mem=128]"
    specify that this job requests 128 megabytes of RAM per job. Without this, a default RAM setting will be applied:  1500MB per job

    #BSUB -B
    send mail to specified email when the job is dispatched and begins execution.

    #BSUB -u example@miami.edu
    send notification through email to example@miami.edu.

    #BSUB -N
    send job statistics report through email when job finishes.

**Example scripts for parallel jobs**

We recommend using Intel MPI unless you have specific reason for using
OpenMP.  Intel MPI scales better and has better performance than OpenMP.

Submit parallel jobs to the **normal** job queue with ``-q normal``.

For optimum performance, the default resource allocation on the parallel
queue is ``ptile=16``. This requires the LSF job scheduler to allocate
16 processors per host, ensuring all processors on a single host are
used by that job. ***Without prior authorization, any jobs using a
number other than 16 will be rejected from the parallel queue.***
**Reserve enough memory for your jobs.** Memory reservations are per
job. Parallel job performance may be affected, or even interrupted, by
other badly-configured jobs running on the same host.

**Example script for Intel/Intel MPI**

``testparai.job``

--------------

.. code:: bash

    #!/bin/bash
    #BSUB -J mpijob
    #BSUB -o %J.out
    #BSUB -e %J.err
    #BSUB -W 1:30
    #BSUB -q normal
    #BSUB -n 32                             # Request 32 cores
    #BSUB -R "span[ptile=16]"               # Request 16 cores per node
    #BSUB -R "rusage[mem=128]"              # Request 128MB per job
    #

    mpiexec foo.exe

``foo.exe`` is the mpi executable name. It can be followed by its own
argument list.

**Example script for MPI/OpenMP**

``testparao.job``

--------------

.. code:: bash

    #!/bin/bash
    #BSUB -J mpijob
    #BSUB -o %J.out
    #BSUB -e %J.err
    #BSUB -W 1:30
    #BSUB -q normal
    #BSUB -n 32                             # Request 32 cores                
    #BSUB -R "span[ptile=16]"               # Request 16 cores per node
    #BSUB -R "rusage[mem=128]"              # Request 128MB per job
    #

    mpiexec --mca btl self,sm,openib foo.exe

The command line is similar to Intel MPI job above. Option
``--mca self,sm,openib`` tells OpenMP to use lookback, shared memory,
and openib for inter-process communication.


How to Request GPUs
=======================
GPUs can be requested on a per node basis. Please see below an example job script. More information on the gpu queues and their restrictions can be found here: https://acs-docs.readthedocs.io/clusters/general-usage/5-jobs/2-queues.html

``testgpu.job``

--------------

.. code:: bash

    #!/bin/bash
    #BSUB -J GPUjob
    #BSUB -P <project_ID>
    #BSUB -o %J.out
    #BSUB -e %J.err
    #BSUB -W 1:30
    #BSUB -q gpu_titan                     # On Pegasus the queues are "gpu_titan" or "gpu_h100". On Triton the queue is "normal"
    #BSUB -n 1                             # Request 1 core               
    #BSUb -gpu "num=1"                     # Request 1 gpu. Can request up to 2.
    #BSUB -R "rusage[mem=128]"             # Request 128MB per job

    python cuda_test.py    

You can request up to 2 gpus in your job script. On Pegasus, only the gpu_titan and gpu_h100 queues contain gpus. On Triton, every node in the normal queue contains 2 gpus. Virtual Memory of the gpu is allocated dynamically as needed.