This is a biased account of the preparation required for compiling parallel codes on the cluster (supercomputer) that I use on a daily basis. I had to figure a lot of stuff out myself, and hopefully this rough guide will help me again in the future. Perhaps some random internet denizen might find some useful information here as well.
Right off the bat, we need to find out which compilers are currently installed, and where they are located. This is relatively straightforward using the “which” command. First the Intel Fortran and C compilers:
[me@machine ~]$ which ifort /opt/intel/composerxe-2011.3.174/bin/intel64/ifort [me@machine ~]$ which icc /opt/intel/composerxe-2011.3.174/bin/intel64/icc
as well as the GNU Fortran and c:
[me@machine ~]$ which gfortran /usr/bin/gfortran [me@machine ~]$ which gcc /usr/bin/gcc
This doesn’t REALLY tell you if these compilers are installed or not, only whether or not the paths to them are in your $PATH variable. Whenever you type a command, Linux checks to see if that command is in any of the paths in your $PATH. If it is not, then you will receive an error. To see what paths are currently in your $PATH variable, a simple echo command will suffice:
[me@machine ~]$ echo $PATH
If you suspect that the desired compilers are indeed installed, but not in your path, start looking in the /opt/<vendor> directories for the correct paths. It’s highly unlikely that gcc and gfortran will not be installed (and furthermore, not be in your path), but just in case, they usually live in /usr/bin
Another method for finding the paths to compilers is to use the “locate” command:
[me@machine ~]$ locate ifort
This will bring up a list of paths that contain the phrase “ifort”. This can be very useful, especially in conjunction with the “grep” command.
Once you have located the paths to the compilers, you can add them to your $PATH variable with the following command in BASH (for other shells, see Google):
[me@machine ~]$ export PATH=$PATH:/home/kyle/tau-2.21/x86_64/bin
Echo-ing your $PATH variable should now reveal that the path has been added. A good idea is to add that line to one of your login files (.bashrc), so that it executes automatically each and every time you log into the machine.
Of course, one of the primary reasons for running codes on a cluster, is to take advantage of parallelism, and this means compiling parallel programs. We want to make sure that our “MPI Compilers” are accessible as well:
[me@machine geo_benchmark]$ which mpif77 /usr/local/openmpi/bin/mpif77 [me@machine geo_benchmark]$ which mpif90 /usr/local/openmpi/bin/mpif90 [me@machine geo_benchmark]$ which mpicc /usr/local/openmpi/bin/mpicc [me@machine geo_benchmark]$ which mpiCC /usr/local/openmpi/bin/mpiCC
So far, so good. Notice that these are the “openmpi” versions of these compilers. It’s entirely possible that other versions exist as well (MPICH, MPICH2, MPICH-GM, MVAPICH, MVAPICH2). If you require (or want to switch to) one of these versions, start looking in the /usr/local/ directories to see if they are available. The “locate” command is helpful here as well.
We may also need Intel’s Math Kernel Library (MKL). It is usually installed in paths similar to Intel’s compilers. The following are example include and library directories, respectively, for MKL:
/opt/intel/composerxe-2011.3.174/mkl/include /opt/intel/composerxe-2011.3.174/mkl/lib/intel64
Once you have all of the paths sorted out, figuring out the actual compile line, especially with MKL, can be a nightmare. Luckily there is a web app to help, the Intel® Math Kernel Library Link Line Advisor. Check it out.
Finally, with all of this information, you can create a Makefile. Here’s a stripped down example to help get you started:
FCC = mpif90 CC = icc INCDIR = /opt/intel/composerxe-2011.3.174/mkl/include LIBDIR = /opt/intel/composerxe-2011.3.174/mkl/lib/intel64 LLAPACK = -L${LIBDIR} -lmkl_intel_lp64 -lmkl_sequential -lmkl_core LFLAGS = -lm $(LLAPACK) -I${INCDIR} OBJ = program.o program.e: $(OBJ) $(FCC) $(OBJ) -o program.e $(LFLAGS) program.o: program.f $(FCC) -c program.f clean: rm -f *.o program.e
If all has gone well, you now have a Makefile with all of the appropriate paths. If you cannot find all of the correct paths, or straight up suspect that certain software has not been installed, you may want to contact your system administrator. Good luck.
