Using OpenMP on the HEC
What is OpenMP?
OpenMP is a set of compiler directives, library routines and environment variables that can be used to specify shared-memory parallelism in C, C++ and Fortran codes.
OpenMP compiler directives can be inserted into source code to indicate to the compiler which section of code can be readily parallelised, allowing a programmer to highlight the core sections of code that can benefit from parallelism. When compiled with a special compiler flag, the compiler will create a multi-threaded version of the application that can automatically distribute the highlighted parallel sections of code to different CPUs on the same node. On the current HEC, this offers up to eight-way parallelism.
It should be stressed that not all codes will benefit from such attempts at parallelism, and not all sections of code can be parallelised. Users should test serial and parallel versions of their code to ensure that the parallel version is making good use of the additional processors.
Programming and Compiling with OpenMP
A detailed explanation of the OpenMP compiler directives can found in the guides for both the PGI and Intel compiler suites (see the Further Reading section)
To compile OpenMP code using the PGI compiler, compile with your normal set of PGI compiler flags, and add the compiler argument -mp.
To compile OpenMP code using the Intel compiler, compile with your normal set of Intel compiler flags, and set the -openmp
Don't forget to make sure that the correct module for your preferred compiler suite has already been added to your environment.
Submitting OpenMP batch jobs
The following job template will run an 8-processor version of the program omptest compiled with intel compilers:
#$ -q parallel
#$ -N myjob
#$ -l nodes=1,ppn=1,tpp=8
module add intel
The additional scheduler directives above serve the same purpose as for MPI jobs; they specify the parallel queue, and the additional cores required. Note the format: one node, one process per node, 8 threads per process. Using this syntax, the scheduler will launch the job with the OMP_NUM_THREADS environment variable set to the correct value (8), allowing the omptest OpenMP application to launch the correct number of threads.
As with MPI jobs, a whole node is reserved for the job; no other job or users may use the node while your OpenMP job is running. To ensure resources aren't wasted, please ensure that any OpenMP job gives a significant speedup over a single-core run.
As the whole node is reserved, the memory resource request is automatically set to the maximum for the node. By defualt, this a 24G node. A larger memory node can be scheduled by adding the directive:
to the job submission script. As large memory nodes are a rarer and high demand resource, you will likely experience a longer wait time for jobs to run.
- The Official OpenMP home page
- Online Compiler Guides