1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
|
Mercury
=======
[![Build status][travis-ci-svg]][travis-ci-link]
[![Latest version][mercury-release-svg]][mercury-release-link]
Mercury is an RPC framework specifically designed for use in HPC systems
that allows asynchronous transfer of parameters and execution requests,
as well as direct support of large data arguments. The network implementation
is abstracted, allowing easy porting to future systems and efficient use
of existing native transport mechanisms. Mercury's interface is generic
and allows any function call to be serialized.
Please see the accompanying COPYING file for license details.
Contributions and patches are welcomed but require a Contributor License
Agreement (CLA) to be filled out. Please contact us if you are interested
in contributing to Mercury by subscribing to the [mailing lists][mailing-lists].
Architectures supported
=======================
Architectures supported by MPI implementations are generally supported by the
network abstraction layer. The OFI libfabric plugin as well as the SM plugin
are stable and provide the best performance in most workloads. Libfabric
providers currently supported are: `tcp`, `verbs`, `psm2`, `gni`.
MPI and BMI (tcp) plugins are still supported but gradually being moved as
deprecated, therefore should only be used as fallback methods.
The CCI plugin is deprecated and underlying CCI transport plugins
(`tcp`, `sm`, `verbs`, `gni`) are no longer supported.
See the [plugin requirements](#plugin-requirements) section for
plugin requirement details.
Documentation
=============
Please see the documentation available on the mercury [website][documentation]
for a quick introduction to Mercury.
Software requirements
=====================
Compiling and running Mercury requires up-to-date versions of various
software packages. Beware that using excessively old versions of these
packages can cause indirect errors that are very difficult to track down.
Plugin requirements
-------------------
To make use of the libfabric/OFI plugin, please refer to the libfabric build
instructions available on this [page][libfabric].
To make use of the native NA SM (shared-memory) plugin on Linux,
the cross-memory attach (CMA) feature introduced in kernel v3.2 is required.
The yama security module must also be configured to allow remote process memory
to be accessed (see this [page][yama]). On MacOS, code signing with inclusion of
the na_sm.plist file into the binary is currently required to allow process
memory to be accessed.
To make use of the BMI plugin, the most convenient way is to install it through
spack or one can also do:
git clone https://xgitlab.cels.anl.gov/sds/bmi.git && cd bmi
./prepare && ./configure --enable-shared --enable-bmi-only
make && make install
To make use of the MPI plugin, Mercury requires a _well-configured_ MPI
implementation (MPICH2 v1.4.1 or higher / OpenMPI v1.6 or higher) with
`MPI_THREAD_MULTIPLE` available on targets that will accept remote
connections. Processes that are _not_ accepting incoming connections are
_not_ required to have a multithreaded level of execution.
To make use of the CCI plugin, please refer to the CCI build instructions
available on this [page][cci].
Optional requirements
---------------------
For optional automatic code generation features (which are used for generating
serialization and deserialization routines), the preprocessor subset of the
BOOST library must be included (Boost v1.48 or higher is recommended).
The library itself is therefore not necessary since only the header is used.
Mercury includes those headers if one does not have BOOST installed and
wants to make use of this feature.
On Linux OpenPA v1.0.3 or higher is required (the version that is included
with MPICH can also be used) for systems that do not have `stdatomic.h`
(GCC version less than 4.9).
Building
========
If you install the full sources, put the tarball in a directory where you
have permissions (e.g., your home directory) and unpack it:
gzip -cd mercury-X.tar.gz | tar xvf -
or
bzip2 -dc mercury-X.tar.bz2 | tar xvf -
Replace `'X'` with the version number of the package.
(Optional) If you checked out the sources using git (without the `--recursive`
option) and want to build the testing suite (which requires the kwsys
submodule) or use checksums (which requires the mchecksum submodule), you need
to issue from the root of the source directory the following command:
git submodule update --init
Mercury makes use of the CMake build-system and requires that you do an
out-of-source build. In order to do that, you must create a new build
directory and run the `ccmake` command from it:
cd mercury-X
mkdir build
cd build
ccmake .. (where ".." is the relative path to the mercury-X directory)
Type `'c'` multiple times and choose suitable options. Recommended options are:
BUILD_SHARED_LIBS ON (or OFF if the library you link
against requires static libraries)
BUILD_TESTING ON
Boost_INCLUDE_DIR /path/to/include/directory
CMAKE_INSTALL_PREFIX /path/to/install/directory
MERCURY_ENABLE_DEBUG ON/OFF
MERCURY_ENABLE_PARALLEL_TESTING ON/OFF
MERCURY_USE_BOOST_PP ON
MERCURY_USE_CHECKSUMS ON
MERCURY_USE_SYSTEM_BOOST ON/OFF
MERCURY_USE_SYSTEM_MCHECKSUM ON/OFF
MERCURY_USE_XDR OFF
NA_USE_BMI ON/OFF
NA_USE_MPI ON/OFF
NA_USE_CCI ON/OFF
NA_USE_OFI ON/OFF
NA_USE_SM ON/OFF
Setting include directory and library paths may require you to toggle to
the advanced mode by typing `'t'`. Once you are done and do not see any
errors, type `'g'` to generate makefiles. Once you exit the CMake
configuration screen and are ready to build the targets, do:
make
(Optional) Verbose compile/build output:
This is done by inserting `VERBOSE=1` in the `make` command. E.g.:
make VERBOSE=1
Installing
==========
Assuming that the `CMAKE_INSTALL_PREFIX` has been set (see previous step)
and that you have write permissions to the destination directory, do
from the build directory:
make install
Testing
=======
Tests can be run to check that basic RPC functionality (requests and bulk
data transfers) is properly working. CTest is used to run the tests,
simply run from the build directory:
ctest .
(Optional) Verbose testing:
This is done by inserting `-V` in the `ctest` command. E.g.:
ctest -V .
Extra verbose information can be displayed by inserting `-VV`. E.g.:
ctest -VV .
Some tests run with one server process and X client processes. To change the
number of client processes that are being used, the `MPIEXEC_MAX_NUMPROCS`
variable needs to be modified (toggle to advanced mode if you do not see
it). The default value is automatically detected by CMake based on the number
of cores that are available.
Note that you need to run `make` again after the makefile generation
to use the new value.
FAQ
===
Below is a list of the most common questions.
- _Q: Why am I getting undefined references to libfabric symbols?_
A: In rare occasions, multiple copies of the libfabric library are installed
on the same system. To make sure that you are using the correct copy of the
libfabric library, do:
ldconfig -p | grep libfabric
If the library returned is not the one that you would expect, make sure to
either set `LD_LIBRARY_PATH` or add an entry in your `/etc/ld.so.conf.d`
directory.
- _Q: Is there any logging mechanism?_
A: To turn on error/warning/debug logs, the `HG_LOG_LEVEL` or
`HG_NA_LOG_LEVEL` environment variables can be set to either `error`,
`warning` or `debug` values. Note that for debugging output to be printed,
the CMake variable `MERCURY_ENABLE_DEBUG` must also be set at compile time.
[mailing-lists]: http://mercury-hpc.github.io/help#mailing-lists
[documentation]: http://mercury-hpc.github.io/documentation/
[cci]: http://cci-forum.com/?page_id=46
[libfabric]: https://github.com/ofiwg/libfabric
[travis-ci-svg]: https://travis-ci.org/mercury-hpc/mercury.svg
[travis-ci-link]: https://travis-ci.org/mercury-hpc/mercury
[mercury-release-svg]: https://img.shields.io/github/release/mercury-hpc/mercury.svg
[mercury-release-link]: https://github.com/mercury-hpc/mercury/releases/latest
[yama]: https://www.kernel.org/doc/Documentation/security/Yama.txt
|
