Documentation:FAQ

(MKL) what's the difference between "lp64" and "ilp64"?
Integer type length: ILP64 uses the 64-bit integer type, while LP64 use the 32-bit integer.

By default, the compilers take integers as 32-bit length. So most applications need link with LP64 MKL libraries.

How to mix MPI flavors and compilers (please someone verify)
On Bellatrix, the following compilers are available:


 * système: gcc 4.4.6
 * modules: gcc/4.6.3 gcc/4.7.2 intel/13.0.1

The following mpis are available:


 * mvapich2/1.8.1, openmpi/1.6.3, intelmpi/4.1.0

Here is what you have to do to use a specific mpi with a specific compiler, after having loaded the correct module

When using Intel MPI, you can also set some environment variables (instead of using the specific wrappers):

export I_MPI_CC=icc export I_MPI_CXX=icpc export I_MPI_F77=ifort export I_MPI_F90=ifort

What is the NUMA topology / cache hierarchy of the node I am computing on?
Likwid allows you to display useful information about the nodes' architecture. If it is not installed in the modules, you can download it and compile it in your home or scratch directory. For example, on Bellatrix you can do:

> qsub -q Q_free -I > module load likwid > likwid-topology -g - CPU type:     Intel Core SandyBridge EP processor ************************************************************* Hardware Thread Topology ************************************************************* Sockets:      2 Cores per socket:     8 Threads per core:     1 - HWThread      Thread          Core            Socket 0             0               0               0 1              0               1               0 2              0               2               0 3              0               3               0 4              0               4               0 5              0               5               0 6              0               6               0 7              0               7               0 8              0               0               1 9              0               1               1 10             0               2               1 11             0               3               1 12             0               4               1 13             0               5               1 14             0               6               1 15             0               7               1 - Socket 0: ( 0 1 2 3 4 5 6 7 ) Socket 1: ( 8 9 10 11 12 13 14 15 ) - ************************************************************* Cache Topology ************************************************************* Level: 1 Size: 32 kB Cache groups:  ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) - Level: 2 Size: 256 kB Cache groups:  ( 0 ) ( 1 ) ( 2 ) ( 3 ) ( 4 ) ( 5 ) ( 6 ) ( 7 ) ( 8 ) ( 9 ) ( 10 ) ( 11 ) ( 12 ) ( 13 ) ( 14 ) ( 15 ) - Level: 3 Size: 20 MB Cache groups:  ( 0 1 2 3 4 5 6 7 ) ( 8 9 10 11 12 13 14 15 ) - ************************************************************* NUMA Topology ************************************************************* NUMA domains: 2 - Domain 0: Processors: 0 1 2 3 4 5 6 7 Relative distance to nodes: 10 21 Memory: 14390.6 MB free of total 16354.6 MB - Domain 1: Processors: 8 9 10 11 12 13 14 15 Relative distance to nodes: 21 10 Memory: 15838 MB free of total 16384 MB - ************************************************************* Graphical: ************************************************************* Socket 0: +-+ | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | |  0   | |   1   | |   2   | |   3   | |   4   | |   5   | |   6   | |   7   | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +-+ | | |                                    20MB                                    | | | +-+ | +-+ Socket 1: +-+ | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | |  8   | |   9   | |   10  | |   11  | |   12  | |   13  | |   14  | |   15  | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | |  32kB | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | 256kB | | | +---+ +---+ +---+ +---+ +---+ +---+ +---+ +---+ | | +-+ | | |                                    20MB                                    | | | +-+ | +-+