Moab and SLURM Exercise

Exercise 1

  1. Login to the workshop machine

    Workshops differ in how this is done. The instructor will go over this beforehand.

  2. Create a subdirectory for the Moab exercise files and cd to it. Then copy the exercise files:

    mkdir ~/moab
    cd  moab
    cp  /usr/global/docs/training/blaise/moab/*   ~/moab

  3. List the contents of your moab subdirectory. You should notice the following files:

    File Name Description
    moab.ex1 Simple shell script used for first exercise
    mpi_array.c Parallel program source code - C version
    mpi_array.f Parallel program source code - Fortran version
    mpithreads.c Hybrid parallel (MPI + threads) program source code - C version
    Solutions to exercises

Review your cluster's batch configuration

  1. Try all of the commands below:
    news job.lim.machine - where machine is the name of the cluster
    sinfo -s
  2. Questions:
    • Which pools are configured?
    • How many nodes are there in each pool?
    • What are the batch queue node and time limits?
    • What states are the nodes in (alloc, idle, etc.)?
    • Who's running jobs and which/how many nodes are they using?

Find out which banks are available

  1. To see which banks are available to you on this cluster, simply issue the mshare command. Note that it displays your bank allocation and usage information also.

  2. To view your default bank and all banks across the entire Moab grid, use the mdiag -u classXX command, where classXX is your unique workshop username. Look for the ADEF (account default) and ALIST (account list) fields. For the workshop accounts, they will be the same, but for some users, there will be a choice of accounts.

  3. To view the entire bank hierarchy, use the mshare -t root command.

Create and run a Moab job script

  1. Using your favorite text editor (vi/vim, emacs, nedit, gedit, nano...), create a job script that does the following:
    • Sets a time limit of 5 minutes
    • Runs in the partition being used for the workshop (machine name)
    • Requests 1 node
    • Runs in the pReserved queue
    • Combines stderr and stdout
    • Writes the batch output to a file name of your choosing
    • Gives your job a unique name
    • Ignores lscratch file system dependencies
    • Runs under your login shell (or preferred shell)
    • Changes to your moab subdirectory
    • Issues the name of the host it is running on
    • Issues the Moab/SLURM jobid for this job
    • Shows your path
    • Runs the moab.ex1 executable provided to you
    • Sleeps for a few minutes (so you can have time to check on it)

    For reference, you can review the job script.

  2. Submit your job to Moab with the msub command
    • Did Moab accept it?
    • What was its jobid number?
    • Problems? Check your script against the example and/or the Moab tutorial for errors.

Monitor your job

  1. The tutorial described several ways to monitor your job, including:
    • showq, showq -r, showq -i
    • checkjob, checkjob -v
    • mdiag -j, mdiag -j -v
    • squeue
    • sview
    • ju
    • mjstat

  2. Try any/all of these commands, noting their similarities and differences.
    Hint: you may want to pipe the output of the more verbose commands into grep with the jobid or your workshop username. For example:
    showq | grep class04
    If you run out of time, you can submit another job with a longer "sleep".

  3. If you have questions about the output of Moab commands, check the Moab tutorial and/or man pages.

  4. After your job completes, examine its status using the checkjob and showq -c | grep jobid commands.

Check your job's output

  1. Review the output file from your job.
    • Where did you find it?
    • Is it named what you specified?
    • Is the output what should be expected? Compare your output file to the solution1.out file. You may want to look at the moab.ex1 file also.

This completes Exercise 1

Exercise 2

  1. Still logged into the workshop cluster?

    If so, then continue to the next step. If not, then login as you did previously for Exercise 1.

Holding and releasing a job

  1. Using your same job script, submit the job so that it is held. This can be done on the msub command line, or from within the script itself. Try both ways. If you have any questions see the tutorial.

  2. Verify that your job is actually in a holding state

  3. Release your job(s) so that they run to completion.

  4. Verify that the job release actually took effect

Canceling a job

  1. Once again, submit your job script.

  2. Try to cancel it before it completes. You can do this when its queued or when it's running. If you have any questions see the tutorial.

  3. Confirm that the job is actually cancelled. Also, check its post-execution status with the checkjob command.

Running in standby mode

  1. Modify your job script so that it will run in standby mode. The example file is provided for reference.

  2. Submit your job script.

  3. When your job starts to run, verify that it is running in standby mode. One way to do this is use the checkjob command and look for qos:standby near the top of the output.

  4. Submit your job script again but be sure to have the job HELD.

  5. Confirm that the job is held.

  6. Now change the qos from standby to normal for this job. If you have any questions see the tutorial.

  7. Confirm that the qos was changed.

  8. Cancel the job (or release it and let it run) when you're sure that it was changed.

Run a parallel job

  1. Using your solution1 example file, copy it to a new file.

  2. Modify your new file so that:
    • Four nodes are requested
    • A new output file name is used
    • It compiles either mpi_array.c or mpi_array.f. Your choice of compilers can be found HERE (use a parallel compiler command).
    • Lists the names of the nodes used to run the job
    • Runs an 48-task MPI job using the mpi_array executable you created in the previous step.

    The example file is provided for reference.

  3. Submit your job and monitor it, making sure it is using the number of nodes/tasks specified.

  4. Check your output file to verify that things worked. See as a comparison.

Run a hybrid (MPI + threads) parallel job

  1. The example file mpithreads.c combines MPI with pthreads. The basic idea is to run one MPI task per node, and then spawn one thread for each core on that node. The threads do the actual work and MPI is used to collect the results across all nodes. Feel free to examine the source code if you'd like.

  2. Using your solution3 example file, copy it to a new file.

  3. Modify your new file so that:
    • Four nodes are requested
    • A new output file name is used
    • Compiles the mpithreads.c file (use a parallel C compiler command)
    • Runs a 4-task MPI job using the executable you created in the previous step. However, this time run with only one task per node. This will permit the threads spawned by each MPI task to use the available cores on a node without competition from the threads of other MPI tasks.

    The example file is provided for reference.

  4. Submit your job and monitor it, making sure it is using the number of nodes/tasks specified.

  5. Check your output file to verify that things worked. See as a comparison.

When will my job start?

    Some of the most frequently asked questions by users include;

    There are several common answers to these questions. Assuming that there are no system problems or errors in the user's job submission script, one of the most common reasons has to do with a job's calculated priority and the Moab fair-share scheduling algorithms.

  1. Try / compare the following two commands, which provide you with a sorted list of eligible jobs and their relative priorities:
    showq -i
    mdiag -p
  2. You can also use the showstart command to "guesstimate" when a particular job might start. Pick a jobid or two from the mdiag -p list of prioritized jobs and try:
    showstart jobid
  3. Understanding how a job's priority is calculated is non-trivial, but is explained in the Moab document: Understanding Job Priority Calculation on Livermore Computing's Moab-Scheduled Machines. For your reference reading later, possibly.

Try sview

    The sview utility provides a graphical view of all user jobs running on a cluster. Give it a try if you haven't already.

Documentation - if you still have time

    The best launching spot for LC Moab documentation is:

This completes the exercise.

Evaluation Form       Please complete the online evaluation form if you have not already done so for this tutorial.

Where would you like to go now?