PEXSI requires an external parallel \(LU\) factorization or \(LDL^T\) factorization routine, and an external parallel matrix reordering routine to reduce the fill-in of the factorization routine.

Currently we use SuperLU_DIST for the parallel \(LU\) factorization, and ParMETIS for the parallel fill-in reducing reordering. It is also possible to use PT-Scotch for the reordering. But we recommend to first download ParMETIS.

Attention: **The installation procedure and dependencies of every version of the PEXSI package may be different. Please follow the documentation of the version of the PEXSI package you are working with (provided in the Download page)**

Build ParMETIS

Download ParMETIS (latest version 4.0.3) from

http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis/parmetis-4.0.3.tar.gz

Follow the installation step to install ParMETIS.

Attention

After untar the ParMETIS package, in Install.txt

Edit the file metis/include/metis.h and specify the width (32 or
64 bits) of the elementary data type used in ParMetis (and
METIS). This is controled by the IDXTYPEWIDTH constant.

For now, on a 32 bit architecture you can only specify a width
of 32, whereas for a 64 bit architecture you can specify a width
of either 32 or 64 bits.

In our experience for most cases, the following setup work fine.

#define IDXTYPEWIDTH 32

Build SuperLU_DIST

SuperLU_DIST v5.1.2 starting from PEXSI v0.10.0

Download SuperLU_DIST (latest version 5.1.2) from

http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_5.1.2.tar.gz

Follow the installation step to install SuperLU_DIST.

Attention: Our experience shows that on some machines it may be better to build SuperLU_DIST with -O2 option than the more aggresive optimization options provided by vendors.; In SuperLU_DIST v5.1.2, some functions conflict when both real and complex arithmetic factorization is needed. This can be temporarily solved by adding -Wl,--allow-multiple-definition in the linking option.; In SuperLU_DIST v5.1.2, there could be some excessive outputs. This can be removed by going to the SRC/ directory of superlu, and comment out the line starting with printf(".. dQuery_Space in dmemory_dist.c. Do the same thing for the line starting with printf(".. zQuery_Space in zmemory_dist.c.; Please note that the number of processors for symbolic factorization cannot be too large when PARMETIS is used together with SuperLU. The exact number of processors for symbolic factorization is unfortunately a magic parameter. See Frequently asked questions.

(Optional) Build PT-Scotch

On some machines, ParMETIS may only allow to use a relatively small number of processors for the matrix permutation. In such circumstance, a workaround can be to use PT-Scotch, which can be downloaded from (latest version 6.0.0)

https://gforge.inria.fr/frs/download.php/31831/scotch_6.0.0.tar.gz

Follow the installation step to install PT-Scotch.

Attention

In INSTALL.TXT, pay special attention to the following sections in order to compile PT-Scotch correctly.

2.3) Integer size issues
2.5) Threads issues

PT-Scotch is also METIS-Compatible. See the following section in INSTALL.TXT for more information.

2.9) MeTiS compatibility library

(Optional) Build symPACK

symPACK is a sparse symmetric matrix direct linear solver which can be optionally used with PEXSI. More information can be found at http://www.sympack.org/.

To use symPACK, first, download the package as follows:

git clone git@bitbucket.org:berkeleylab/sympack.git  /path/to/sympack

Several environment variables can be optionally set before configuring the build:

METIS_DIR = Installation directory for MeTiS
PARMETIS_DIR = Installation directory for ParMETIS
SCOTCH_DIR = Installation directory for SCOTCH and PT-SCOTCH

Then, create a build directory, enter that directory and type:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/path/to/install/sympack
...OPTIONS... /path/to/sympack

The ...OPTIONS... can be one of the following:

-DENABLE_METIS=ON|OFF to make MeTiS ordering available in symPACK (METIS_DIR must be set in the environment)
-DENABLE_PARMETIS=ON|OFF to make ParMETIS ordering available in symPACK (PARMETIS_DIR must be set in the environment, METIS_DIR is required as well)
-DENABLE_SCOTCH=ON|OFF to make SCOTCH / PT-SCOTCH orderings available in symPACK (SCOTCH_DIR must be set in the environment)

Some platforms have preconfigured toolchain files which can be used by adding the following option to the cmake command:

-DCMAKE_TOOLCHAIN_FILE=/path/to/sympack/toolchains/edison.cmake     
(To build on NERSC Edison for instance)

A sample toolchain file can be found in /path/to/sympack/toolchains/build_config.cmake and customized for the target platform.

The cmake command will configure the build process, which can now start by typing:

make
make install

Additionally, a standalone driver for symPACK can be built by typing make examples