Skip to content

Latest commit

 

History

History
176 lines (142 loc) · 9.1 KB

README.md

File metadata and controls

176 lines (142 loc) · 9.1 KB
Raw

FairMQ

license DOI OpenSSF Best Practices fair-software.eu Spack package

C++ Message Queuing Library and Framework

Docs: Book

Find all FairMQ releases here.

Introduction

FairMQ is designed to help implementing large-scale data processing workflows needed in next-generation Particle Physics experiments. FairMQ is written in C++ and aims to

  • provide an asynchronous message passing abstraction of different data transport technologies,
  • provide a reasonably efficient data transport service (zero-copy, high throughput),
  • be data format agnostic, and
  • provide basic building blocks that can be used to implement higher level data processing workflows.

The core of FairMQ provides an abstract asynchronous message passing API with scalability protocols inspired by ZeroMQ (e.g. PUSH/PULL, PUB/SUB). FairMQ provides multiple implementations for its API (so-called "transports", e.g. zeromq and shmem (latest release of the ofi transport in v1.4.56, removed since v1.5+)) to cover a variety of use cases (e.g. inter-thread, inter-process, inter-node communication) and machines (e.g. Ethernet, Infiniband). In addition to this core functionality FairMQ provides a framework for creating "devices" - actors which are communicating through message passing. FairMQ does not only allow the user to use different transport but also to mix them; i.e: A Device can communicate using different transport on different channels at the same time. Device execution is modelled as a simple state machine that shapes the integration points for the user task. Devices also incorporate a plugin system for runtime configuration and control. Next to the provided devices and plugins the user can extend FairMQ by developing his own plugins to integrate his devices with external configuration and control services.

FairMQ has been developed in the context of its mother project FairRoot - a simulation, reconstruction and analysis framework.

Installation from Source

Recommended:

git clone https:/FairRootGroup/FairMQ fairmq_source
cmake -S fairmq_source -B fairmq_build -GNinja -DCMAKE_BUILD_TYPE=Release [-DBUILD_TESTING=ON]
cmake --build fairmq_build
[ctest --test-dir fairmq_build --output-on-failure --schedule-random -j<ncpus>] # needs -DBUILD_TESTING=ON
cmake --install fairmq_build --prefix $(pwd)/fairmq_install

Please consult the manpages of your CMake version for more options.

If dependencies are not installed in standard system directories, you can hint the installation location via -DCMAKE_PREFIX_PATH=... or per dependency via -D{DEPENDENCY}_ROOT=... (*_ROOT variables can also be environment variables).

Installation via Spack

Prerequisite: Spack

spack info fairmq # inspect build options
spack install fairmq # build latest packaged version with default options

Build FairMQ's dependencies via Spack for development:

git clone -b dev https:/FairRootGroup/FairMQ fairmq_source
spack --env fairmq_source install # installs deps declared in fairmq_source/spack.yaml
spack env activate fairmq_source # sets $CMAKE_PREFIX_PATH which is used by CMake to find FairMQ's deps
cmake -S fairmq_source -B fairmq_build -GNinja -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON
# develop, compile, test
spack env deactivate # at end of dev session, or simply close the shell

Usage

FairMQ ships as a CMake package, so in your CMakeLists.txt you can discover it like this:

find_package(FairCMakeModules 1.0 REQUIRED)
include(FairFindPackage2)
find_package2(FairMQ)
find_package2_implicit_dependencies()

The FairFindPackage2 module is part of the FairCMakeModules package.

If FairMQ is not installed in system directories, you can hint the installation:

list(PREPEND CMAKE_PREFIX_PATH /path/to/fairmq_install)

Dependencies

Which dependencies are required depends on which components are built.

Supported platform is Linux. macOS is supported on a best-effort basis.

CMake options

On command line:

  • -DDISABLE_COLOR=ON disables coloured console output.
  • -DBUILD_TESTING=OFF disables building of tests.
  • -DBUILD_EXAMPLES=OFF disables building of examples.
  • -DBUILD_DOCS=ON enables building of API docs.
  • -DFAIRMQ_CHANNEL_DEFAULT_AUTOBIND=OFF disable channel autoBind by default
  • You can hint non-system installations for dependent packages, see the #installation-from-source section above

After the find_package(FairMQ) call the following CMake variables are defined:

Variable Info
${FairMQ_PACKAGE_DEPENDENCIES} the list of public package dependencies
${FairMQ_<dep>_VERSION} the minimum <dep> version FairMQ requires
${FairMQ_<dep>_COMPONENTS} the list of <dep> components FairMQ depends on
${FairMQ_PACKAGE_COMPONENTS} the list of components FairMQ consists of
${FairMQ_#COMPONENT#_FOUND} TRUE if this component was built
${FairMQ_VERSION} the version in format MAJOR.MINOR.PATCH
${FairMQ_GIT_VERSION} the version in the format returned by git describe --tags --dirty --match "v*"
${FairMQ_PREFIX} the actual installation prefix
${FairMQ_BINDIR} the installation bin directory
${FairMQ_INCDIR} the installation include directory
${FairMQ_LIBDIR} the installation lib directory
${FairMQ_DATADIR} the installation data directory (../share/fairmq)
${FairMQ_CMAKEMODDIR} the installation directory of shipped CMake find modules
${FairMQ_BUILD_TYPE} the value of CMAKE_BUILD_TYPE at build-time
${FairMQ_CXX_FLAGS} the values of CMAKE_CXX_FLAGS and CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE} at build-time

Documentation

  1. Device
    1. Topology
    2. Communication Patterns
    3. State Machine
    4. Multiple devices in the same process
  2. Transport Interface
    1. Message
      1. Ownership
    2. Channel
    3. Poller
  3. Configuration
    1. Device Configuration
    2. Communication Channels Configuration
      1. JSON Parser
      2. SuboptParser
    3. Introspection
  4. Development
    1. Testing
    2. Static Analysis
      1. CMake Integration
      2. Extra Compiler Arguments
  5. Logging
    1. Log severity
    2. Log verbosity
    3. Color for console output
    4. File output
    5. Custom sinks
  6. Examples
  7. Plugins
    1. Usage
    2. Development
    3. Provided Plugins
      1. PMIx