TCP Multistream and Multipool

TCP Multistream

Using multiple TCP connections for a single tunnel can be beneficial in specific scenarios:

• UTP3 can't be applied: because UDP is blocked; because UDP performance is low due to firewall, VPN or NIC settings; when machine has no enough processing power to handle UTP3.
• Network has high RTT (10+ms) and experiences some losses, either due to medium issues or because of congestion.

Configuration

To enable multistream, add the following custom parameters to your Job profile:

• net.tcp.streams - The maximum number of TCP connections that can be open within a single multistream connection (default: 1).
• net.tcp.force_max_streams (optional) - If enabled, the system will create the maximum allowed number of TCP connections right away instead of increasing the number of connections gradually depending on the current TCP sockets load (default: false).

Internals

Multistream replaces the traditional TCP socket wrapper in tunnel connections with a multiplexing "socket" that directly manages multiple TCP connections. Network quota, both license and speed limit, applies to a multistream connection as to a single connection, preventing imbalances. 

A Multistream Connection (MSC) begins with a single socket inherited from the existing TCP wrapper. Conversion to multistream occurs after a tunnel is established between agents, meaning it is not used during peer discovery. 

Data transmission in MSC is structured using a list of socket IDs (SIDs), which define the order and size of data blocks. Default block size is 128Kb.
Each MSC is assigned a unique MSCID, which is globally scoped within an Agent process.

Dynamic increase in TCP connections may be requested if there’re no connecting TCP sockets for the MSC (i.e. all current connection attempts either succeeded or failed) and if at least 60% of sockets on send list were busy for the last 2 seconds. If net.tcp.force_max_streams:true, Agent creates enough connections to reach the configured number of net.tcp.streams; it increases connections by 10% of net.tcp.streams, but no less than 1. To open additional TCP connections Agent requires peer listen address, therefore additional connections are made from peer that opened the tunnel.

Dynamic reduce of  TCP connections is not reasonable. Instead tunnel is rebuilt if there’re no app-level connections (folder tree merges, downloads, etc.) on the side that opened the tunnel originally. To do that we start out of order recheck that creates new tunnel that takes over existing one if succeeds.

Multipool

Multipool enables increasing transfer speeds by adding extra tunnels to the primary TCP tunnel.

Enable multipool in case you're experiencing either of the following:

• The total network throughput of an endpoint exceeds the capability of a single TCP thread (12+ Gbps)
  Note: Depending on your CPU's performance, this case may require additional adjustments in your configuration (described below).
• A limit in total transfer speed is reached at approximately 10 TCP streams and adding more streams doesn't result in meaningful gains, while you know the system is capable of delivering twice the bandwidth you're observing.
• The system has several IP addresses assigned (possibly on multiple network interfaces) without bonding enabled, or in case the bonding underperforms.

Configuration
To enable multipool, add the following custom parameters to your Job profile:

• net.tcp.threads - Defines the number of TCP threads, which should correspond to the capabilities of your system. A single TCP thread is capable of handling 7-8 Gbps of network throughput. If your system is capable of delivering 20 Gbps, set the net.tcp.threads parameter value to 3 (default: 1).
• net.tcp.tunnels - Defines the number of tunnels established for a data transfer. The value should be as low as possible while ensuring high per-tunnel performance. In our testing, this was between 10 and 20 (default: 1).
• net.tcp.multiport.bind (optional) - Defines additional IP address and port number combinations, where TCP threads should listen for connections. You can assign specific IP addresses to specific threads. For example addr1:port1,addr2:port2; addr3:port3 will result in the extra thread #1 listening on addr1:port1 and addr2:port2, and the extra thread#2 listening on addr3:port3.

In case the CPU is unable to cope with computing hash values of the transferred data, you may also need to apply the following changes to your Job profile:

• Disable rolling hash by setting the torrent_rolling_checksum parameter value to 0.
• Use xxhash3_128 to create files' metadata (instead of sha1) by setting the transfer_job_hashing_algo parameter value to 3.