UnifyFS Configuration

Here, we explain how users can customize the runtime behavior of UnifyFS. In particular, UnifyFS provides the following ways to configure:

  • Configuration file: $INSTALL_PREFIX/etc/unifyfs/unifyfs.conf

  • Environment variables

  • Command line options to unifyfsd

All configuration settings have corresponding environment variables, but only certain settings have command line options. When defined via multiple methods, the command line options have the highest priority, followed by environment variables, and finally config file options from unifyfs.conf.

The system-wide configuration file is used by default when available. However, users can specify a custom location for the configuration file using the -f command-line option to unifyfsd (see below). There is a sample unifyfs.conf file in the installation directory under etc/unifyfs/. This file is also available in the extras directory in the source repository.

The unified method for providing configuration control is adapted from CONFIGURATOR. Configuration settings are grouped within named sections, and each setting consists of a key-value pair with one of the following types:

  • BOOL: 0|1, y|n, Y|N, yes|no, true|false, on|off

  • FLOAT: scalars convertible to C double, or compatible expression

  • INT: scalars convertible to C long, or compatible expression

  • STRING: quoted character string

System Configuration File (unifyfs.conf)

unifyfs.conf specifies the system-wide configuration options. The file is written in INI language format, as supported by the inih parser.

The config file has several sections, each with a few key-value settings. In this description, we use section.key as shorthand for the name of a given section and key.


[unifyfs] section - main configuration settings

Key

Type

Description

cleanup

BOOL

cleanup storage on server exit (default: off)

configfile

STRING

path to custom configuration file

daemonize

BOOL

enable server daemonization (default: off)

mountpoint

STRING

mountpoint path prefix (default: /unifyfs)


[client] section - client settings

Key

Type

Description

cwd

STRING

effective starting current working directory

excl_private

BOOL

create node-local private files when given O_EXCL (default: on)

fsync_persist

BOOL

persist data to storage on fsync() (default: on)

local_extents

BOOL

service reads from local data (default: off)

max_files

INT

maximum number of open files per client process (default: 128)

node_local_extents

BOOL

service reads from node local data for laminated files (default: off)

super_magic

BOOL

whether to return UNIFYFS (on) or TMPFS (off) statfs magic (default: on)

unlink_usecs

INT

number of microseconds to sleep after initiating unlink rpc (default: 0)

write_index_size

INT

maximum size (B) of memory buffer for storing write log metadata

write_sync

BOOL

sync data to server after every write (default: off)

The client.cwd setting is used to emulate the behavior one expects when changing into a working directory before starting a job and then using relative file names within the application. If set, the application changes its working directory to the value specified in client.cwd when unifyfs_mount() is called. The value specified in client.cwd must be within the directory space of the UnifyFS mount point.

Enabling the client.local_extents optimization may significantly improve read performance for extents written by the same process. However, it should not be used by applications in which different processes write to the same byte offset within a file, nor should it be used with applications that truncate files.


[log] section - logging settings

Key

Type

Description

dir

STRING

path to directory to contain server log file

file

STRING

log file base name (rank will be appended)

on_error

BOOL

increase log verbosity upon encountering an error (default: off)

verbosity

INT

logging verbosity level [0-5] (default: 0)


[logio] section - log-based write data storage settings

Key

Type

Description

chunk_size

INT

data chunk size (B) (default: 4 MiB)

shmem_size

INT

maximum size (B) of data in shared memory (default: 256 MiB)

spill_size

INT

maximum size (B) of data in spillover file (default: 4 GiB)

spill_dir

STRING

path to spillover data directory


[margo] section - margo server NA settings

Key

Type

Description

tcp

BOOL

Use TCP for server-to-server rpcs (default: on, turn off to enable libfabric RMA)

client_timeout

INT

timeout in milliseconds for rpcs between client and server (default: 5000)

server_timeout

INT

timeout in milliseconds for rpcs between servers (default: 15000)


[runstate] section - server runstate settings

Key

Type

Description

dir

STRING

path to directory to contain server-local state


[server] section - server settings

Key

Type

Description

hostfile

STRING

path to server hostfile

init_timeout

INT

timeout in seconds to wait for servers to be ready for clients (default: 120)

local_extents

BOOL

use server extents to service local reads without consulting file owner


[sharedfs] section - server shared files settings

Key

Type

Description

dir

STRING

path to directory to contain server shared files


Environment Variables

All environment variables take the form UNIFYFS_SECTION_KEY, except for the [unifyfs] section, which uses UNIFYFS_KEY. For example, the setting log.verbosity has a corresponding environment variable named UNIFYFS_LOG_VERBOSITY, while unifyfs.mountpoint corresponds to UNIFYFS_MOUNTPOINT.

Command Line Options

For server command line options, we use getopt_long() format. Thus, all command line options have long and short forms. The long form uses --section-key=value, while the short form -<optchar> value, where the short option character is given in the below table.

Note that for configuration options of type BOOL, the value is optional. When not provided, the true value is assumed. If the short form option is used, the value must immediately follow the option character (e.g., -Cyes).

unifyfsd command line options

LongOpt

ShortOpt

--unifyfs-cleanup

-C

--unifyfs-configfile

-f

--unifyfs-daemonize

-D

--log-verbosity

-v

--log-file

-l

--log-dir

-L

--runstate-dir

-R

--server-hostfile

-H

--sharedfs-dir

-S

--server-init_timeout

-t