Install the platform » Historique » Révision 25
Révision 24 (pierre kestener, 20/01/2014 11:35) → Révision 25/62 (pierre kestener, 20/01/2014 11:59)
{{toc}}
h1. Install the platform without MAGMA
The COMPASS platform is distributed as a single bundle of CArMA and SuTrA libraries and YoGA and its AO extension for Yorick.
h2. Hardware requirements
The system must contain at least an x86 CPU and a CUDA capable GPU. list of compatible GPUs can be found here http://www.nvidia.com/object/cuda_gpus.html. Specific requirements apply to clusters (to be updated).
h2. Environment requirements
The system must be running a 64 bit distribution of Linux or Mac OS with the latest NVIDIA drivers and "CUDA toolkit":https://developer.nvidia.com/cuda-downloads. The installation of the corresponding version of the "CULA tools":http://www.culatools.com/downloads/dense/ is also required. The following installation instructions are valid if the default installation paths have been selected for these components.
Additionally, to benefit from the user-oriented features of the platform, Yorick should be installed as well as the latest version of Python and the associated pygtk module.
To install Yorick, download the latest version from the github repository:
<pre>
git clone https://github.com/dhmunro/yorick.git yorick.git
</pre>
then cd onto the created directory and install:
<pre>
./configure && make && make install
</pre>
once Yorick is locally installed, you will have to add this directory : yorick.git/relocate/bin to your PATH to have an easy access to the yorick executable. You may want to add support for command history by using rlwrap and alias the yorick executable as :
<pre>
alias yorick='rlwrap path_to_yorick_executable/yorick'
</pre>
h2. Installation process
First check out the latest version from the svn repository :
<pre>
svn co https://version-lesia.obspm.fr/repos/compass compass
</pre>
then go in the newly created directory and then trunk:
<pre>
cd compass/trunk
</pre>
once there, you need to modify system variables in the define_var.sh executable :
<pre>
emacs define_var.sh
</pre>
in this file define properly CUDA_ROOT, CULA_ROOT and YoGA path. Note that for the latter, as YoGA is distributed with SUTrA you should just point to the newly created trunk directory. On a Linux system you should normally have:
<pre>
export CUDA_ROOT=/usr/local/cuda
export CULA_ROOT=/usr/local/cula
export YOGA_DIR=/home/MyUserName/path2compass/trunk
</pre>
in this file, you also have to indicate the proper architecture of your GPU so as the compiler will generate the appropriate code. Modify the following line:
<pre>
export GENCODE="arch=compute_12,code=sm_12"
</pre>
and change both 12 to your architecture : for instance a Tesla Fermi will have 2.0 computing capabilities so change 12 to 20, a Kepler GPU will have 3.0 or 3.5 (K20) computing capabilities, change 12 to 30 (or 35).
Once this is done, you're ready to compile the whole library. First run define_var.sh to define the system variables that will be used during the compilation process:
<pre>
./define_var.sh
</pre>
then identify the absolute path to your Yorick executable using:
<pre>
which yorick
</pre>
and run the compilation script:
<pre>
./reinstall absolute_path_to_yorick
</pre>
If you did not get any error, CArMA, SuTrA and YoGA are now installed on your machine. You can check that everything is working by launching a GUI to test a simulation:
<pre>
yorick -i yoga_ao/ywidgets/widget_ao.i
</pre>
h1. Install the platform with MAGMA
h2. Why MAGMA ?
The MAGMA project aims to develop a dense linear algebra library similar to LAPACK but for heterogeneous/hybrid architectures, starting with current "Multicore+GPU" systems.
Unlike CULA, MAGMA propose a dense linear algebra library handling double for free.
But MAGMA needs a LAPACK and a BLAS implementation. Actually, we try two options : ATLAS BLAS (free, easy to install) and MKL (free, need a registration but more powerful)
h2. Dependencies : gfortran
Use your package manager to install dependencies:
* on scientific linux : yum install gcc-gfortran libgfortran
* on debian : apt-get install gfortran gfortran-multilib
h2. Configure MAGMA with ATLAS
h3. Dependencies : blas, lapack, atlas
Use your package manager to install dependencies:
* on scientific linux : yum install blas-devel lapack-devel atlas-devel
* on debian : apt-get install libblas-dev liblapack-dev libatlas-base-dev libatlas-dev
The binary packages of ATLAS (and also OpenBLAS / GotoBLAS2) distributed by your Linux distribution (SL, Fedora, Debian,...) are generic packages, which are not optimized for a specific machine.
It is strongly advised to recompile ATLAS on your local machine to get best performances.
IMPORTANT NOTE: when building ATLAS, you must ensure that cpu throtling is disabled (if not timing measurement are erroneous, which may lead to an unoptimized build of ATLAS); see page
http://math-atlas.sourceforge.net/atlas_install/node5.html
h3. extraction
MAGMA is available here : http://icl.cs.utk.edu/magma/software/index.html
extract the tgz file and go into the new directory
> ~$ tar xf magma-1.4.1-beta.tar.gz
> ~$ cd magma-1.4.1
h3. configuration
You have to create your own make.inc :
* example on a scientific linux : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# -- MAGMA (version 1.4.1) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# November 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET specifies for which GPU you want to compile MAGMA:
# "Tesla" (NVIDIA compute capability 1.x cards)
# "Fermi" (NVIDIA compute capability 2.x cards)
# "Kepler" (NVIDIA compute capability 3.x cards)
# See http://developer.nvidia.com/cuda-gpus
GPU_TARGET ?= Fermi
CC = gcc
NVCC = nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -fPIC -O3 -DADD_ -fopenmp -DMAGMA_SETAFFINITY
F77OPTS = -fPIC -O3 -DADD_
FOPTS = -fPIC -O3 -DADD_ -x f95-cpp-input
NVOPTS = -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"
LDOPTS = -fPIC -fopenmp
# Depending on how ATLAS and LAPACK were compiled, you may need one or more of:
LIB = -llapack -lf77blas -latlas -lcblas -lcublas -lcudart -lstdc++ -lm -lgfortran
# define library directories here or in your environment
LAPACKDIR = /usr/lib64
ATLASDIR = /usr/lib64/atlas
CUDADIR = /usr/local/cuda
LIBDIR = -L$(LAPACKDIR) \
-L$(ATLASDIR) \
-L$(CUDADIR)/lib64
INC = -I$(CUDADIR)/include
</code></pre>
* example on debian : *please verify GPU_TARGET, LAPACKDIR, ATLASDIR, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# -- MAGMA (version 1.4.1) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# November 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET specifies for which GPU you want to compile MAGMA:
# "Tesla" (NVIDIA compute capability 1.x cards)
# "Fermi" (NVIDIA compute capability 2.x cards)
# "Kepler" (NVIDIA compute capability 3.x cards)
# See http://developer.nvidia.com/cuda-gpus
GPU_TARGET ?= Fermi
CC = gcc
NVCC = nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -fPIC -O3 -DADD_ -fopenmp -DMAGMA_SETAFFINITY
F77OPTS = -fPIC -O3 -DADD_
FOPTS = -fPIC -O3 -DADD_ -x f95-cpp-input
NVOPTS = -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"
LDOPTS = -fPIC -fopenmp
# Depending on how ATLAS and LAPACK were compiled, you may need one or more of:
LIB = -llapack -lf77blas -latlas -lcblas -lcublas -lcudart -lstdc++ -lm -lgfortran
# define library directories here or in your environment
LAPACKDIR = /usr/lib
ATLASDIR = /usr/lib
CUDADIR = /usr/local/cuda
LIBDIR = -L$(LAPACKDIR) \
-L$(ATLASDIR) \
-L$(CUDADIR)/lib64 \
-L/usr/lib/x86_64-linux-gnu
INC = -I$(CUDADIR)/include
</code></pre>
h2. Configure MAGMA with MKL
h3. extraction
To download MKL, you have to create a account here : https://registrationcenter.intel.com/RegCenter/NComForm.aspx?ProductID=1517
extract l_ccompxe_2013_sp1.1.106.tgz and go into l_ccompxe_2013_sp1.1.106
install it with ./install_GUI.sh and add IPP stuff to default choices
h3. configuration
* example on debian : *please verify GPU_TARGET, MKLROOT, CUDADIR*
<pre><code class="Makefile">
#//////////////////////////////////////////////////////////////////////////////
# -- MAGMA (version 1.4.1-beta2) --
# Univ. of Tennessee, Knoxville
# Univ. of California, Berkeley
# Univ. of Colorado, Denver
# December 2013
#//////////////////////////////////////////////////////////////////////////////
# GPU_TARGET contains one or more of Tesla, Fermi, or Kepler,
# to specify for which GPUs you want to compile MAGMA:
# Tesla - NVIDIA compute capability 1.x cards
# Fermi - NVIDIA compute capability 2.x cards
# Kepler - NVIDIA compute capability 3.x cards
# The default is all, "Tesla Fermi Kepler".
# See http://developer.nvidia.com/cuda-gpus
#
GPU_TARGET ?= Fermi
CC = gcc
NVCC = nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
OPTS = -fPIC -O3 -DADD_ -Wall -fno-strict-aliasing -fopenmp -DMAGMA_WITH_MKL -DMAGMA_SETAFFINITY
F77OPTS = -fPIC -O3 -DADD_ -Wall
FOPTS = -fPIC -O3 -DADD_ -Wall -x f95-cpp-input
NVOPTS = -O3 -DADD_ -Xcompiler "-fno-strict-aliasing -fPIC"
LDOPTS = -fPIC -fopenmp
# gcc with MKL 10.3, Intel threads
LIB = -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_core -lpthread -lcublas -lcudart -lstdc++ -lm -liomp5 -lgfortran
# define library directories preferably in your environment, or here.
# for MKL run, e.g.: source /opt/intel/composerxe/mkl/bin/mklvars.sh intel64
MKLROOT ?= /opt/intel/composerxe/mkl
CUDADIR ?= /usr/local/cuda
-include make.check-mkl
-include make.check-cuda
LIBDIR = -L$(MKLROOT)/lib/intel64 \
-L$(CUDADIR)/lib64
INC = -I$(CUDADIR)/include -I$(MKLROOT)/include
</code></pre>
In this example, I use gcc but with MKL, you can use icc instead of gcc. In this case, you have to compile yorick with icc. For this, you have to change the CC flag in Make.cfg
h2. compilation and installation
h3. compilation
just compile the shared target (and test if you want)
> ~$ make -j 8 shared
h3. installation
To install libraries and include files in a given prefix, run:
> ~$ make install prefix=/usr/local/magma
The default prefix is /usr/local/magma. You can also set prefix in make.inc.
h3. tune (not tested)
For multi-GPU functions, set $MAGMA_NUM_GPUS to set the number of GPUs to use.
For multi-core BLAS libraries, set $OMP_NUM_THREADS or $MKL_NUM_THREADS or $VECLIB_MAXIMUM_THREADS to set the number of CPU threads, depending on your BLAS library.
h2. Platform installation
Just just define $MAGMA_PATH and use the standard procedure