NOTE 1: This FAQ is about using the scheduling manager to submit jobs to blueshark nodes. For information on using installed packages on blueshark, please see this FAQ for more information on using environment modules.
NOTE 2: To access the blueshark HPC cluster, you must have an SSH client and use it to connect to <code>code><code>blueshark.fit.educode>. Authentication is performed using your TRACKS username and password. Please be aware that your TRACKS username is not your email address. It is only the part before the '@' symbol. Please see this SSH FAQ for more information on using SSH.
NOTE 3: All code of processes must be run on a compute node using the SLURM scheduler. Blueshark is a shared resource; therefore, if a user runs code on the login node, i.e., blueshark.fit.edu itself, it will cause login and performance issues for others. Users who run code or processes on the login node will have their code terminated, and continued disregard will result in the user being blocked from using blueshark.
Table of Contents
- Blueshark Resource Management
- Common Errors and Solutions
- slurmstepd: error: Exceeded job memory limit at some point.
- error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
- SSH: Access denied: user [username] (uid=[uid]) has no active jobs.
- SFTP: Received message too long [random number]
- sbatch: error: Batch script contains DOS[MAC] line breaks (\r\n)sbatch: error: instead of expected UNIX line breaks (\n).
To run a job on blueshark, you must submit it through blueshark's scheduler, SLURM. This differs from how you'd normally run a command, as you need to prepare a submission script and optionally make your program MPI capable.
Blueshark Resource Management
Blueshark uses SLURM (Simple Linux Utility for Resource Management) to manage available resources and to distribute jobs to free nodes. Slurm also provides a queuing system; if not enough resources are available, it will hold your job until it can run it.
SLURM Example Submission Script
In order to submit a job to SLURM, a job submission script must be created. An example submission script is provided below.
#SBATCH --job-name TestJob
#SBATCH --nodes 2 #SBATCH --ntasks 2 #SBATCH --mem=50MB #SBATCH --time=00:15:00
#SBATCH --partition=short #SBATCH --error=testjob.%J.err #SBATCH --output=testjob.%J.out module load mpich echo "Starting at date" echo "Running on hosts: $SLURM_NODELIST" echo "Running on $SLURM_NNODES nodes." echo "Running on $SLURM_NPROCS processors." echo "Current working directory is pwd"
The only options you absolutely need are:
- <code>--job-name code> — a unique name for your job. This can be set to anything.
- <code>--nodes code> — the number of nodes to request.
- <code>--ntasks code> — the number of tasks in total across all nodes. Note that this differs from torque's PPN, which assigns this number of tasks to every node.
- <code>--mem code> — amount of memory to request on each node. This is a hard limit and you will run into out-of-memory errors if you fail to provide the correct amount.
- <code>--partition code> — the partition for your job. Valid partitions can be found by using <code>sinfocode>.
If you need MPICH to run your jobs, set it to load at login using:
<code>module initadd mpichcode>
or by placing
<code>module load mpichcode>
in your submission script like above. Many more options are available for Slurm's submission scripts, and more information can be found here.
Submitting a Job
Like Torque, SLURE has its own set of commands for job management. To submit your submission script, use
Some other commands you may want to use are listed below.
- <code>squeue code> — similar to <code>showqcode> for torque; lists the jobs that are currently running for everyone.
- <code>sinfo code> — show node status.
- <code>scancel code> — cancel a currently running job.
- <code>sstat code> — show statistics for a job.
For a more in-depth look into Slurm and its respective commands, check out their quick start guide.
SLURM's partitions are very similar to Torque's queue system. Jobs that only need a short amount of time to run, but a large number of processors, will have their jobs categorized differently than jobs that may need to run for days and require fewer processors. In addition, partitions can be used to group together nodes that have general hardware that others don't (ex, GPU partition has GPUs in its nodes). Currently, these partitions exist:
|Partition Name||Max Compute Time||Max Nodes||Allowed Groups|
|10 minutes||6||Parallel Programming class|
|infinite||10||blueshark GPU users|
To set a partition, use:
in your submission script, or specify it on the command line using --partition.
Running GPU Jobs
Running GPU jobs is very similar to running regular jobs. An extra parameter has to be passed (--gres) and the partition must be set to gpu.
will set your partition.
<code>#SBATCH --gres=gpu:[#] code>
will set the number of GPUs you want per node. Note that this differs from ntask, specified earlier. --gres will request n number of GPUs from each node. Thus, if you request 4 nodes with --gres=gpu:2, you will have [4 nodes] * [2 GPU/node] = 8 GPUs in total. This option will not exceed 4, as we only have 4 GPUs per node.
GPUs can also be selected based on whether or not they support GPUDirect technology. Each GPU node has 2 standard GPUs and 2 GPUDirect enabled GPUs. To select between the two, you can use:
<code>#SBATCH --gres=gpu:[type]:[#] code>
where [type] is either gpudirect or standard.
Optimizing your Submission Script
SLURM will attempt to run your job wherever it can place it, however, this is hugely dependent on how your submission script specifies its resources. Thus, if you can reduce your submission script requirements, your job has a much higher chance of being scheduled faster.
--mem is most often used to specify the amount of memory your job will take per node. However, this is largely dependent on how many tasks you can fit in a node or the number of nodes you'll require. If you don't specify the number of nodes you need, slurm won't balance out the tasks, often leading to out-of-memory errors on nodes where more jobs were placed than expected. Another issue arises when the cluster is under heavy use. Small pockets of resources are scattered through the cluster, and won't be easy to acquire when your job needs a fixed amount of memory per node.
To prevent this, we can use --mem-per-cpu instead. If each task only requires a certain amount of memory, you can specify this amount instead. This way, the scheduler can better allocate resources -- if tasks require more memory than what's available on a node, they'll be split, and if there are pockets of resources a single task can fit in, it will allocate that spot.
Common Errors and Solutions
slurmstepd: error: Exceeded job memory limit at some point.
The job you ran tried using more memory than what was defined in your submission script. As a result, slurm automatically killed your job. A simple fix is to increase the amount of memory dedicated to your job, using --mem at the command line or "#SBATCH --mem" in your submission script.
error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)
You attempted to submit a job to a partition that didn't support your --time option. The solution is to move your job to a partition with a longer execution time (med, long, etc.). By default, jobs are sent to the short queue, which only permits at most 45 minutes. Specify a partition in your submission script, or reduce your --time option.
SSH: Access denied: user [username] (uid=[uid]) has no active jobs.
This error comes up when you attempt to ssh into a node that you're not currently running a job on. Under normal circumstances, you should not run jobs directly on the nodes as this can confuse the scheduler and prevent other users from submitting jobs. If you're unable to use the scheduler to submit your job, and you absolutely need to ssh in (for example, X11 forwarding), see this section of SLURM's FAQ.
SFTP: Received message too long [random number]
This error comes up when you attempt to use SFTP and you have a message printed via a *profile config in your directory (.bash_profile, etc.) To correct the issue, remove the printing message and try again.
sbatch: error: Batch script contains DOS[MAC] line breaks (\r\n)
sbatch: error: instead of expected UNIX line breaks (\n).
Sometimes, if you download a SLURM submission script to a Windows or Mac computer and re-upload to Blueshark, you may get this error when attempting to submit the script using sbatch. The solution is to run "dos2unix" on the file.