Most relevant MPI/HPC tunable parameters

[shefty]

There are several environment variables that can be modified that will impact MPI performance and scalability. This is a list of the most relevant. For details see fi_info -g <search_string>. E.g. "fi_info -g tcp" or "fi_info -g msg_tx_size".

For rxm (over verbs or tcp):

• FI_OFI_RXM_MSG_TX_SIZE - max sends at the msg endpoint level
• FI_OFI_RXM_MSG_RX_SIZE - max receives at the msg endpoint or shared receive context
• FI_OFI_RXM_USE_SRX - determines if shared receive contexts should be used
• FI_OFI_RXM_TX_SIZE - max sends per RDM endpoint
• FI_OFI_RXM_RX_SIZE - max receives per RDM endpoint
• FI_OFI_RXM_BUFFER_SIZE - size of pre-allocated bounce buffers
• FI_OFI_RXM_EAGER_LIMIT - maximum size for eager protocol
• FI_OFI_RXM_SAR_LIMIT - maximum size for segmentation and reassembly protocol
• FI_OFI_RXM_ENABLE_DYN_RBUF - if enabled, reduces receive side data copies
• FI_OFI_RXM_ENABLE_DIRECT_SEND - if enabled, reduces send side data copies

For tcp:

• FI_TCP_STAGING_SBUF_SIZE - size of buffer used to coalesce sends when network is busy
• FI_TCP_PREFETCH_RBUF_SIZE - amount of data to prefetch from kernel on receives

[shefty]

Updated list for TCP, relative to v1.13:

FI_TCP_TX_SIZE= default 256
FI_TCP_RX_SIZE= default 256, but higher (1k-2k) if SRX is enabled

FI_OFI_RXM_BUFFER_SIZE=256 (default 16k), smaller size recommended to reduce memory footprint
FI_OFI_RXM_EAGER_LIMIT=128k or 256k (default 16k), larger size recommended to improve medium sized message bw
FI_OFI_RXM_USE_SRX=1 (should be default - recommended set to 1 to reduce memory footprint)
FI_OFI_RXM_TX_SIZE=default 2k, may be reduced (recommended >= 1k)
FI_OFI_RXM_RX_SIZE=default 2k, may be reduced (recommended >= 1k)
FI_OFI_RXM_MSG_TX_SIZE=align with FI_TCP_TX_SIZE
FI_OFI_RXM_MSG_RX_SIZE=2048 (align with FI_TCP_RX_SIZE and SRX)
FI_OFI_RXM_DEF_TCP_WAIT_OBJ=POLLFD (this needs to be documented)

The memory footprint will be most dependent on FI_OFI_RXM_BUFFER_SIZE x FI_OFI_RXM_MSG_RX_SIZE. If SRX is enabled (set to 1), footprint is buffer size x msg rx size. If SRX is disabled (set to 0), multiple the previous footprint by the number of peers/ranks.

[roblatham00]

This is a great list, though tcp and sockets aren't that common in HPC environments, right?

[shefty]

Currently about half of systems in the top 500 supercomputers are running ethernet. Search for gigabit ethernet as the interconnect here:

https://www.top500.org/statistics/sublist/

Some of those are slingshot or RoCE, but most are either 10 or 100 Gbit Ethernet. The transport protocol is not listed. HPC applications running in the cloud frequently use standard sockets; even though, there are lower latency options like EFA on AWS. I think it depends on whether the app is latency or bandwidth limited and the number of nodes as to whether a higher performing interconnect is beneficial.

FWIW, a significant amount of my recent development has been directly related to improving TCP performance for HPC applications, which is customer driven.

[roblatham00]

Good point about wider perspective! the 'H' for me is shifted way to the left where infiniband, slingshot (and the bizarre gni) live.

[mmubarak]

@roblatham00 FWIW this is a good resource comparing the transport protocols for HPC on the cloud-- https://ieeexplore.ieee.org/document/9167399

[roblatham00]

Other environment variables I've experimented with:

• In an ideal world FI_VERBS_TX_IOV_LIMIT and FI_VERBS_RX_IOV_LIMIT could be cranked up to thousands of elements, but underlying hardware rarely if ever supports more than 4. sadface.
• FI_MR_CACHE_MAX_COUNT : we often set this to zero, which unintuitively helps performance in a lot of our cases. Coordination cost of serializing the MR_CACHE outweigh the benefits for a lot of our workloads.
• FI_MR_CACHE_MONITOR: userfaultfd seems to give us a 10% boost to latency. Uh, of course if you set FI_MR_CACHE_MAX_COUNT to zero as suggested immediately above this does nothing.

[shefty]

Interesting. Btw, the verbs IOV limits increase the size of each command written to the HW. Some HW support variable sized commands, so the impact there can be minimal. For HW that uses a fixed sized command, increasing the IOV size can negatively impact performance. The other side effect of the IOV limit is the size of the memory allocated for the QP increases, and that memory is pinned. That can impact scaling.