MPI Exercises

Exercise 1

  1. Login to the workshop machine

    Workshops differ in how this is done. The instructor will go over this beforehand

  2. Copy the example files

    1. In your home directory, create a subdirectory for the MPI test codes and cd to it.
      mkdir ~/mpi
      cd  ~/mpi

    2. Copy either the Fortran or the C version of the parallel MPI exercise files to your mpi subdirectory:

      C:
      cp  /usr/global/docs/training/blaise/mpi/C/*   ~/mpi
      Fortran:
      cp  /usr/global/docs/training/blaise/mpi/Fortran/*   ~/mpi

    3. Some of the example codes have serial versions for comparison. Use the appropriate command below to copy those files to your mpi subdirectory also.

      C:
      cp  /usr/global/docs/training/blaise/mpi/Serial/C/*   ~/mpi
      Fortran:
      cp  /usr/global/docs/training/blaise/mpi/Serial/Fortran/*   ~/mpi 
      

  3. List the contents of your MPI subdirectory

    You should notice quite a few files. The parallel MPI versions have names which begin with or include mpi_. The serial versions have names which begin with or include ser_. Makefiles are also included.

    Note: These are example files, and as such, are intended to demonstrate the basics of how to parallelize a code. Most execute in a second or two. The serial codes will be faster because the problem sizes are so small and there is none of the overhead associated with parallel setup and execution.

    C Files Fortran Files Description
    mpi_hello.c mpi_hello.f Hello World
    mpi_helloBsend.c mpi_helloBsend.f Hello World modified to include blocking send/receive routines
    mpi_helloNBsend.c mpi_helloNBsend.f Hello World modified to include nonblocking send/receive routines
    mpi_array.c
    ser_array.c
    mpi_array.f
    ser_array.f
    Array Decomposition
    mpi_mm.c
    ser_mm.c
    mpi_mm.f
    ser_mm.f
    Matrix Multiply
    mpi_pi_send.c
    ser_pi_calc.c
    mpi_pi_send.f
    ser_pi_calc.f
    pi Calculation - point-to-point communications
    mpi_pi_reduce.c
    ser_pi_calc.c
    mpi_pi_reduce.f
    ser_pi_calc.f
    pi Calculation - collective communications
    mpi_wave.c
    draw_wave.c
    ser_wave.c
    mpi_wave.f
    mpi_wave.h
    draw_wavef.c
    ser_wave.f
    Concurrent Wave Equation
    mpi_heat2D.c
    draw_heat.c
    ser_heat2D.c
    mpi_heat2D.f
    mpi_heat2D.h
    draw_heatf.c
    ser_heat2D.f
    2D Heat Equation
    mpi_latency.c
    mpi_latency.f
    Round Trip Latency Timing Test
    mpi_bandwidth.c
    mpi_bandwidth_nonblock.c
    mpi_bandwidth.f
    mpi_bandwidth_nonblock.f
    Bandwidth Timing Tests
    mpi_prime.c
    ser_prime.c
    mpi_prime.f
    ser_prime.f
    Prime Number Generation

    mpi_ping.c
    mpi_ringtopo.c
    mpi_scatter.c
    mpi_contig.c
    mpi_vector.c
    mpi_indexed.c
    mpi_struct.c
    mpi_group.c
    mpi_cartesian.c

    mpi_ping.f
    mpi_ringtopo.f
    mpi_scatter.f
    mpi_contig.f
    mpi_vector.f
    mpi_indexed.f
    mpi_struct.f
    mpi_group.f
    mpi_cartesian.f
    From the tutorial...
    Blocking send-receive
    Non-blocking send-receive
    Collective communications
    Continguous derived datatype
    Vector derived datatype
    Indexed derived datatype
    Structure derived datatype
    Groups/Communicators
    Cartesian Virtual Topology
    Makefile.MPI.c
    Makefile.Ser.c
    Makefile.MPI.f
    Makefile.Ser.f
    Makefiles
    batchscript.c
    batchscript.f
    Batch job scripts
    mpi_bug1.c
    mpi_bug2.c
    mpi_bug3.c
    mpi_bug4.c
    mpi_bug5.c
    mpi_bug6.c
    mpi_bug7.c
    mpi_bug1.f
    mpi_bug2.f
    mpi_bug3.f
    mpi_bug4.f
    mpi_bug5.f
    mpi_bug6.f
    mpi_bug7.f
    Programs with bugs

  4. Compilers - What's Available?

    1. Visit the Compilers Currently Installed on LC Platforms webpage.

    2. Click on the name of the workshop Linux cluster (sierra) in the summary table near the top of the page. This will take you to a table of available compilers, including the MPI compiler script commands.

    3. You can also view the MPI compiler script commands in the LLNL MPI Implementations and Compilers section of the MPI tutorial, and the Compilers section of the Linux Clusters Overview tutorial.

    4. Now, in your cluster login window, try the use -l compilers command to display available compilers. You should see GNU, Intel and PGI compilers - several versions of each.
      • Question: Which version is the default version?
      • Answer: Use the dpkg-defaults command and look for the asterisk.

  5. Create, compile and run an MPI "Hello world" program

    1. Using your favorite text editor (vi/vim, emacs, nedit, gedit, nano...) open a new file - call it whatever you'd like.

    2. Create a simple MPI program that does the following:
      • Uses the appropriate MPI include file
      • Identifies task 0 as the "master" task
      • Initializes the MPI environment
      • Gets the total number of tasks
      • Gets the task's rank (who it is)
      • Gets the name of the processor it is executing on
      • Prints a hello message that includes its task rank and processor name
      • Has the master task alone print the total number of tasks
      • Terminates the MPI environment

      If you need help, see the provided example files or

    3. Using your choice of compiler (see above section 4), compile your hello world MPI program. This may take several attempts if there are any code errors. When you get a clean compile, proceed.

    4. Using the srun command, run your MPI executable. Be sure to use the pReserved partition with 8 tasks on two different nodes. For example:

      srun -N2 -n8 -ppReserved hello

      You may see a message like below while the job gets ready to run:

      srun: Job is in held state, pending scheduler release
      srun: job 1139098 queued and waiting for resources

      Did your job run successfully? Based on the output, did it behave as expected? If not, figure out any problems and fix them before proceeding.

    5. Run your program a few more times, but vary the number of nodes and total tasks. Observe the task output statements to confirm.

This completes Exercise 1




Exercise 2

  1. Still logged into the workshop cluster?

    If so, then continue to the next step. If not, then login as you did previously for Exercise 1.

  2. Create, compile and run an MPI blocking send/receive program

    1. Assuming you were able to create a successful "hello world" MPI program in Exercise 1, copy your source file to a new file and call it something like helloBsend.c or helloBsend.f. If you were not successful, you can use the provided mpi_hello.c or mpi_hello.f. as your starting point.

    2. Edit your new helloBsend source file and modify it to do the following - after the master task has printed the number of tasks, but before MPI_Finalize:
      • Have each task determine a unique partner task to send/receive with. One easy way to do this:
        if (taskid < numtasks/2) then partner = numtasks/2 + taskid
        else if (taskid >= numtasks/2) then partner = taskid - numtasks/2
      • Each task sends its partner a single integer message: its taskid
      • Each task receives from its partner a single integer message: the partner's taskid
      • For confirmation, after the send/receive, each task prints something like "Task ## is partner with ##" where ## is the taskid of the task and its partner.

      If you need help, see the provided example files or

  3. Using your choice of compiler as in Exercise 1, compile your helloBsend program until you get a clean compile.

  4. Using the srun command as in Exercise 1 to run your executable. For example:

    srun -N2 -n8 -ppReserved helloBsend

    Did your job run successfully? Based on the output, did it behave as expected? If not, figure out any problems and fix them before proceeding.

  5. Run your program a few more times, but vary the number of nodes and total tasks. Observe the task output statements to confirm.

  6. If time permits, copy your helloBsend source file to a new helloNBsend source file. Then convert the blocking routines to non-blocking routines. See the provided or source files if you need assistance.

  7. Compile and run as before.

This completes Exercise 2




Exercise 3

  1. Still logged into the workshop cluster?

    1. If so, then continue to the next step. If not, then login as you did previously for Exercises 1 and 2.

  2. Review the array decomposition example code

    1. First, review the serial version of this example code, either ser_array.c or ser_array.f.

    2. After you understand what's going on, review the parallel MPI version, either mpi_array.c or mpi_array.f. The comments explain how MPI is used to implement a parallel data decomposition on an array.

  3. Compile the parallel MPI and serial versions of the array decomposition example code

    As with Exercises 1 & 2, use the compiler commands of your choice to compile the mpi_array example code. If you need a refresher on the available compiler commands see: computing.llnl.gov/tutorials/linux_clusters/index.html#Compilers. For example:

    C:
    gcc ser_array.c  -o ser_array
    mpicc mpi_array.c  -o mpi_array
    
    Fortran:
    gfortran ser_array.f -o ser_array
    mpif77 mpi_array.f -o mpi_array
    

  4. Run the executables interactively

    For the MPI executable, use the special workshop pool and 4 tasks. For example:

    Serial:
    ser_array
    MPI:
    srun -n8 -ppReserved mpi_array

    Note: The srun command is covered in detail in the "Starting Jobs" section of the Linux Clusters Overview tutorial, located at: computing.llnl.gov/tutorials/linux_clusters/index.html#Starting. There is also a man page.

  5. Compare other serial codes to their parallel version

    If we had more time, you might even be able to start with a serial code or two and create your own parallel version. Feel free to try if you'd like.

  6. Try any/all of the other MPI example codes

    1. First, review the code(s) so that you understand how MPI is being used.

    2. Then, using the MPI compiler command(s) of your choice, compile the codes of interest.

    3. For convenience, the included Makefiles can be used to compile any or all of the exercise codes. Note: you will need to edit the Makefiles to select your choice of compiler(s) and flags. For example:

      C:
      make -f Makefile.MPI.c
      make -f Makefile.MPI.c  mpi_mm
      make -f Makefile.Ser.c 
      Fortran:
      make -f Makefile.MPI.f
      make -f Makefile.MPI.f  mpi_mm
      make -f Makefile.Ser.f 

    4. Run the executables interactively in the special workshop pool. Use the srun command for this as shown previously.
      Most of the executables only need 4 MPI tasks or less. Some exceptions and notes:

      mpi_array Requires that the number of tasks be evenly divisible by 4
      mpi_bandwidth Requires an even number of tasks.
      mpi_cartesian Requires 16 MPI tasks
      mpi_group Requires 8 MPI tasks
      mpi_heat2D
      mpi_wave
      These examples attempt to generate an Xwindows display for results. You will need to make sure that your Xwindows environment and software is setup correctly if you want to see the graphic. Ask the instructor if you have any questions.
      mpi_latency Requires only 2 MPI tasks that should be on DIFFERENT nodes

    5. Some things to try:
      • Different compilers
      • Experiment with compiler flags (see respective man pages)
      • Vary the number of tasks and nodes used

  7. Compare on-node communications vs off-node communications

    1. Compile the mpi_bandwidth code if you haven't already.

    2. Run the code interactively with 2 tasks on the same node:
      srun -n2 -ppReserved mpi_bandwidth

    3. Note the average bandwidths. These numbers reflect communications "on-node", which is done in shared memory - not using the IB switch.

    4. Now run the code interactively with 2 tasks on two different nodes.
      srun -N2 -ppReserved mpi_bandwidth
    5. Note the average bandwidths. These numbers reflect communications "off-node", which is done using the IB switch.

    6. What are your observations? Why?

  8. Compare blocking send/receive with non-blocking send/receive

    1. Copy your mpi_bandwidth source file to another file called mpi_bandwidthNB. Modify your new file so that it performs non-blocking sends/receives instead of blocking. An example mpi_bandwidth_nonblock file has been provided in case you need it.

    2. After you're satisfied with your new non-blocking version of the bandwidth code, compile both.

    3. Run each code using two tasks on two different nodes in the special workshop pool:
      srun -N2 -ppReserved mpi_bandwidth
      srun -N2 -ppReserved mpi_bandwidthNB

    4. Compare the results. Which one performs best?

  9. Run some batch MPI jobs

    1. Review the provided Moab job script: batchscript.c or batchscript.f. Moab is covered in detail in the Moab tutorial.

    2. Submit your job script:
      msub batchscript.c
         -or-
      msub batchscript.f

    3. You can monitor your job's progress with the command:
      showq | grep classXX
      where classXX is your assigned workshop username.

    4. After your job completes, review the output file to confirm its successful execution.

    5. If you are interested, you can substitute other exercise files and run those also.

  10. When things go wrong...

    There are many things that can go wrong when developing MPI programs. The mpi_bug series of programs demonstrate just a few. See if you can figure out what the problem is with each case and then fix it.

    Compile with the compile command(s) of your choice and run interactively using 4 tasks in the special workshop pool.

    The buggy behavior will differ for each example. Some hints are provided below.

    Code Behavior Hints/Notes
    mpi_bug1 Hangs
    mpi_bug2 Wrong results or abnormal termination
    mpi_bug3 Error message and/or abnormal termination
    mpi_bug4 Gives the wrong result for "Final sum". Compare to mpi_array Number of MPI tasks must be divisible by 4.
    mpi_bug5 Dies or hangs - depends upon platform and MPI library
    mpi_bug6 Terminates or is ignored (depends on platform/language) Requires 4 MPI tasks.
    mpi_bug7 Hangs


This completes the exercise.

Evaluation Form       Please complete the online evaluation form if you have not already done so for this tutorial.

Where would you like to go now?