An extensible Java framework for benchmarking InfiniBand verbs implementations
Observatory is a benchmark aiming at comparing different InfiniBand solutions for Java. A native version (written C++) is also provided to measure the difference between Java-based and native libraries.
It is intended to benchmark fundamental point-to-point connections by measuring uni- and bidirectional throughput, one-sided latency and full round-trip time. It also uses the jDetector library to determine the data overhead, imposed by any software defined protocol.
Observatory provides a single lean interface, that combines messaging, as well as RDMA operations, so that implementations for different libaries can easily be developed. To configure a set of benchmark runs, a JSON file can be used.
The source code is available at GitHub.
The benchmark itself is made up of six phases that are executed consecutively:
- Initialization: During this phase, the client should allocate any needed resources (e.g open an IB context and allocate a Protection Domain). Client-specific configuration parameters, that are defined in the configuration file (JSON), are passed to the client as key-value tuples. This can be used to pass IB related parameters to the client (e.g. the device and port number).
- Connection: A connection is setup, after IB connection information has been exchanged (e.g. via TCP sockets).
- Preparation: The operation size, which dictates the size of the messages being sent, respectively the size of RDMA writes/reads being performed, is passed to the client, allowing it to allocate matching buffers to use in the benchmark. It is also reasonable to preallocate reusable Work Requests during this phase.
- Warmup: A configurable amount of operations are executed as a warmup, allowing the JVM and its JIT to optimize the benchmark code.
- Operation: This is the main phase of the benchmark, executing the configured amount of operations. If a bidirectional benchmark run is performed, dedicated threads for sending and receiving are started. If a throughput benchmark is being performed, two timestamps will be taken right before the first operation starts and right after the last one has finished. Otherwise, if a latency measurement is performed, the time needed for each operation is measured and stored in an array. This allows calculating percentiles afterwards.
Furthermore, the benchmark utilizes the performance counters of the IB HCA to determine the raw amount of data being sent/received. Thus, it is possible to calculate the overhead added by any software defined protocol which is especially relevant for the socket-based libraries.
- Cleanup: The benchmark is finished, resources shall be freed and all connections shall be closed.
During our research, we have used Observatory to compare different InfiniBand solutions for Java and published the acquired results as a paper to the IEEE International Symposium on Parallel and Distributed Computing (ISPDC) 2020.
In the future, we plan on enhancing Observatory to support multiple threads and connections to evaluate more communication patterns, as well as supporting different InfiniBand connection types besides RC (e.g. UD, DCT).