Build & I/O Interception

In this section, we describe how to build UnifyFS with I/O interception.


The current version of UnifyFS adopts the mdhim key-value store, which strictly requires:

“An MPI distribution that supports MPI_THREAD_MULTIPLE and per-object locking of critical sections (this excludes OpenMPI up to version 3.0.1, the current version as of this writing)”

as specified in the project github.

How to Build UnifyFS

To install all dependencies and set up your build environment, we recommend using the Spack package manager. If you already have Spack, make sure you have the latest release or if using a clone of their develop branch, ensure you have pulled the latest changes.

Building with Spack

These instructions assume that you do not already have a module system installed such as LMod, Dotkit, or Environment Modules. If your system already has Dotkit or LMod installed then installing the environment-modules package with Spack is unnecessary (so you can safely skip that step).

If you use Dotkit then replace spack load with spack use. First, install Spack if you don’t already have it:

$ git clone
$ ./spack/bin/spack install environment-modules
$ . spack/share/spack/

Make use of Spack’s shell support to automatically add Spack to your PATH and allow the use of the spack command.

Then install UnifyFS:

$ spack install unifyfs
$ spack load unifyfs

Include or remove variants with Spack when installing UnifyFS when a custom build is desired. Type spack info unifyfs for more info.

UnifyFS Build Variants
Variant Command Description

spack install unifyfs+hdf5

spack install unifyfs+hdf5 ^hdf5~mpi

Build with parallel HDF5

Build with serial HDF5

Fortran spack install unifyfs+fortran Build with gfortran
NUMA spack install unifyfs+numa Build with NUMA
pmpi spack install unifyfs+pmpi Transparent mount/unmount
PMI spack install unifyfs+pmi Enable PMI2 build options
PMIx spack install unifyfs+pmix Enable PMIx build options


The initial install could take a while as Spack will install build dependencies (autoconf, automake, m4, libtool, and pkg-config) as well as any dependencies of dependencies (cmake, perl, etc.) if you don’t already have these dependencies installed through Spack or haven’t told Spack where they are locally installed on your system (i.e., through a custom packages.yaml). Type spack spec -I unifyfs before installing to see what Spack is going to do.

Building with Autotools

Download the latest UnifyFS release from the Releases page.

Building the Dependencies

UnifyFS requires MPI, LevelDB, GOTCHA(version 0.0.2), FlatCC, and Margo. References to these dependencies can be found here.

Build the Dependencies with Spack

Once Spack is installed on your system (see above), you can install just the dependencies for an easier manual installation of UnifyFS.

If you use Dotkit then replace spack load with spack use.

$ spack install leveldb
$ spack install gotcha@0.0.2
$ spack install flatcc
$ spack install margo


You can use spack install --only=dependencies unifyfs to install all of UnifyFS’s dependencies without installing UnifyFS.

Keep in mind this will also install all the build dependencies and dependencies of dependencies if you haven’t already installed them through Spack or told Spack where they are locally installed on your system.

Then to manually build UnifyFS:

$ spack load leveldb
$ spack load gotcha@0.0.2
$ spack load flatcc
$ spack load mercury
$ spack load argobots
$ spack load margo
$ ./
$ ./configure --prefix=/path/to/install
$ make
$ make install


Fortran Compatibility

To build with gfortran compatibility, include the --enable-fortran configure option:

./configure --prefix=/path/to/install/ --enable-fortran

There is a known ifort_issue with the Intel Fortran compiler as well as an xlf_issue with the IBM Fortran compiler. Other Fortran compilers are currently unknown.

To see all available build configuration options, type ./configure --help after ./ has been run.

Build the Dependencies without Spack

For users who cannot use Spack, a script has been provided in order to make manual build and installation of dependencies easier. Simply run the script in the top level directory of the source code.

$ ./

References to the UnifyFS dependencies can be found here.

After is finished building the dependencies, it will print out the commands you need to run to build UnifyFS. The commands look something like this:

$ export PKG_CONFIG_PATH=path/to/mercury/lib/pkgconfig:path/to/argobots/lib/pkgconfig:path/to/margo/lib/pkgconfig
$ ./
$ ./configure --prefix=/path/to/install --with-gotcha=/path/to/gotcha --with-leveldb=/path/to/leveldb  --with-flatcc=/path/to/flatcc
$ make
$ make install


You may need to add the following to your configure line if it is not in your default path on a linux machine:


This is needed to enable NUMA-aware memory allocation on Linux machines. Set the NUMA policy at runtime with UNIFYFS_NUMA_POLICY = local | interleaved, or set NUMA nodes explicitly with UNIFYFS_USE_NUMA_BANK = <node no.>

I/O Interception

POSIX calls can be intercepted via the methods described below.


Steps for static linking using –wrap:

To intercept I/O calls using a static link, you must add flags to your link line. UnifyFS installs a unifyfs-config script that returns those flags, e.g.,

$ mpicc -o test_write \
      `<unifyfs>/bin/unifyfs-config --pre-ld-flags` \
      test_write.c \
      `<unifyfs>/bin/unifyfs-config --post-ld-flags`


Steps for dynamic linking using gotcha:

To intercept I/O calls using gotcha, use the following syntax to link an application.


$ mpicc -o test_write test_write.c \
    -I<unifyfs>/include -L<unifycy>/lib -lunifyfs_gotcha \
    -L<gotcha>/lib64 -lgotcha


$ mpif90 -o test_write test_write.F \
    -I<unifyfs>/include -L<unifycy>/lib -lunifyfsf -lunifyfs_gotcha