As LLNL’s investment in GPU architectures grows, so too does exploration of new programming paradigms to support code portability on parallel computing platforms. For example, when Sierra came online in 2018, we began porting our existing codes to pragma-based technologies (e.g., OpenMP) and vendor language extensions (e.g., CUDA) to run on such IBM OpenPOWER machines. It was quickly discovered that the smaller FORTRAN community had a more challenging task compared to their more numerous C/C++ colleagues. FORTRAN developers had to tackle less mature OpenMP and CUDA support in compilers, limited code examples available online, and more limited documentation than that available for C/C++. These items highlighted the importance of FORTRAN developers collaborating across Department of Energy (DOE) laboratories to leverage multiple projects that address technical challenges.

The FGPU repository is one such collaboration and provides FORTRAN examples and documentation for next-generation GPU machines. It is intended as an aid for both FORTRAN developers and FORTRAN compiler teams. FGPU provides code examples for developers and code tests for compiler vendors. These code tests are derived from the many compiler bugs we encountered in early Sierra FORTRAN efforts.

While FGPU has a heavy FORTRAN emphasis, some examples provided also include C++ usage for demonstrating OpenMP or CUDA with mixed C++/FORTRAN language executables. Some areas of the repository are:

• interoperability – Examples of how to use OpenMP with other technologies or libraries.
• iso_c_bindings – The OpenMP API for FORTRAN only provides a subset of the C API. This directory contains iso_c_bindings of some of these missing API routines.
• macro_layer – An example of using a macro layer over OpenMP pragmas.
• partially_supported_constructs – Examples of OpenMP constructs that need enhancements to meet FORTRAN needs.
• testing – A collection of code examples that encountered issues during compiler testing, performance testing, and tool testing with CUDA and OpenMP on the DOE Sierra platform using IBM XL OpenMP and CUDA FORTRAN. May be useful as a test suite for compilers wanting to augment the simpler OpenMP examples available on openmp.org, especially in areas using FORTRAN-derived or other features not well covered in those simpler examples.
• usability_issues – Examples highlighting general usability issues with OpenMP.