One million concurrent connections

I’ve heard erroneous claims that a server can only accept 65,000 connections, or that a server always uses one port per accepted connection. Here’s what they look like:

A TCP/IP address only supports 65,000 connections, so you would need to assign approximately 30,000 IP addresses to this server.

There are 65535 TCP port numbers, does this mean that no more than 65535 clients can connect to a TCP server? You might think that this imposes a strict limit on the number of clients a single computer/application can support.

If there is a limit on the number of ports a single machine can have, and a socket can only be bound to an unused port number, how do servers that have an extremely high number of requests (greater than the maximum number of ports) deal with this? Is this problem solved by the distribution of the system, that is, a bunch of servers on many machines?

Therefore, I wrote this article to dispel this myth from three sides:

  1. The WhatsApp messenger and the Elixir-based web framework Phoenix have already demonstrated millions of connections listening on a single port.
  2. Theoretical capabilities based on TCP/IP protocol
  3. A simple experiment with Java that anyone can do on their machine if they are still not convinced by my words.

If you do not want to study the details, then go to the “Results” section at the end of the article.

Phoenix Framework

reached 2000000 concurrent websocket connections

. In the article, the developers demonstrate a chat application that simulates 2 million users and takes 1 second to send messages to all users. They also go into detail about the technical challenges they faced with the framework while trying to achieve this record. Some of the ideas in their article I used to write my post, such as assigning multiple IPs to overcome the 65k client connection limit.

whatsapp too reached 2,000,000 connections. Unfortunately, the developers almost do not share the details. They only talked about the hardware and the operating system.

Someone thinks the limit is 2

sixteen

=65536 because these are all ports available according to the TCP specification. This limit is valid for a single client making outgoing connections with a single IP and port pair. For example, my laptop will only be able to make 65536 connections to 172.217.13.174:443 (google.com:443), but Google will probably block me before I even make 65,000 connections. So, if you need a connection between two machines with more than 65 thousand simultaneous connections, then the client will need to connect from the second IP address, or the server must make the second port available.

The server listening on the port has each incoming connection NOT takes the server port. The server can only use one port that it is listening on. In addition, connections will come from multiple IP addresses. In the best case, the server will be able to listen on all IP addresses coming from all ports.

Each TCP connection is uniquely defined by the following parameters:

  1. 32-bit source IP (the IP address from which the connection is coming)
  2. 16-bit source port (the port of the source IP address from which the connection is coming)
  3. 32-bit destination IP (the IP address to which you are connecting)
  4. 16-bit destination port (the port of the IP address of the destination to which you are connecting)

This means that the theoretical limit that a server can support on one port is 2

48

which is about 1 quadrillion, because:

  1. The server distinguishes connections from client IP addresses and source ports
  2. [количество исходных IP-адресов]x[количество исходных портов]
  3. 32 bits per address and 16 bits per port
  4. Putting it all together: 232 x2sixteen = 248.
  5. This is roughly equal to a quadrillion (log(248)/log(10)=14,449)!

To determine an optimistic practical limit, I ran experiments trying to open as many TCP connections as possible and have the server send and receive a message on each connection. Compared to load

Phoenix

or

whatsapp

this load is completely impractical, but it is easier to implement if you want to try it yourself. To conduct an experiment, you need to deal with three difficulties: the operating system, the JVM, and the TCP/IP protocol.

Experiment

If you are interested in the source code, you can study it

here

.

The pseudocode looks like this:

Поток 1:
  открыть сокет сервера
  for i from 1 to 1 000 000:
    принять входящее подключение
  for i from 1 to 1 000 000
    отправить число i на сокет i
  for i from 1 to 1 000 000
    получить число j на сокете i
    assert i == j

Поток 2:
  for i from 1 to 1 000 000:
    открыть сокет клиента серверу
  for i from 1 to 1 000 000:
    получить число j на сокете i
    assert i == j
  for i from 1 to 1 000 000
    отправить число i на сокет i

Cars

As machines I used my Mac:

2.5 GHz Quad-Core Intel Core i7
16 GB 1600 MHz DDR3

and your Linux desktop:

AMD FX(tm)-6300 Six-Core Processor
8GiB 1600 MHz

File descriptors

First of all, we have to fight with the operating system. The default options severely restrict file descriptors. You will see an error like this:

Exception in thread "main" java.lang.ExceptionInInitializerError
  at java.base/sun.nio.ch.SocketDispatcher.close(SocketDispatcher.java:70)
  at java.base/sun.nio.ch.NioSocketImpl.lambda$closerFor$0(NioSocketImpl.java:1203)
  at java.base/jdk.internal.ref.CleanerImpl$PhantomCleanableRef.performCleanup(CleanerImpl.java:178)
  at java.base/jdk.internal.ref.PhantomCleanable.clean(PhantomCleanable.java:133)
  at java.base/sun.nio.ch.NioSocketImpl.tryClose(NioSocketImpl.java:854)
  at java.base/sun.nio.ch.NioSocketImpl.close(NioSocketImpl.java:906)
  at java.base/java.net.SocksSocketImpl.close(SocksSocketImpl.java:562)
  at java.base/java.net.Socket.close(Socket.java:1585)
  at Main.main(Main.java:123)
Caused by: java.io.IOException: Too many open files
  at java.base/sun.nio.ch.FileDispatcherImpl.init(Native Method)
  at java.base/sun.nio.ch.FileDispatcherImpl.<clinit>(FileDispatcherImpl.java:38)
  ... 9 more

Each server socket needs two file descriptors:

  1. Send Buffer
  2. Receive Buffer

The same applies to client connections. Therefore, to run this experiment on one machine, you will need:

  • 1000000 connections per client
  • 1000000 connections for the server
  • 2 file descriptors per connection
  • = 4000000 file descriptors

On a Mac with bigSur 11.4, you can increase the file descriptor limit like this:

sudo sysctl kern.maxfiles=2000000 kern.maxfilesperproc=2000000
kern.maxfiles: 49152 -> 2000000
kern.maxfilesperproc: 24576 -> 2000000
sysctl -a | grep maxfiles
kern.maxfiles: 2000000
kern.maxfilesperproc: 1000000

ulimit -Hn 2000000
ulimit -Sn 2000000

as recommended in this Answer on StackOverflow.

On Ubuntu 20.04, the fastest way is to do this:

sudo su
# 2^25 должно быть более чем достаточно
sysctl -w fs.nr_open=33554432
fs.nr_open = 33554432
ulimit -Hn 33554432
ulimit -Sn 33554432

Java file descriptor limits

We’ve dealt with the operating system, but the JVM won’t like what we’ll be doing in this experiment either. When running it, we will get the same or similar stack trace.

In that Answer on StackOverflow the solution is specified in the form of a JVM flag:

-XX:-MaxFDLimit: Disables attempts to set a software limit to a hardware limit on the number of open file descriptors. This option is enabled by default on all platforms, but is ignored on Windows. Disabling it is only worth it in Mac OS, where its use imposes a limit of 10240, which is less than the actual maximum of the system.

java -XX:-MaxFDLimit Main 6000

As written in this quote from the Java documentation, you only need to disable the flag on a Mac.

On Ubuntu, I was able to experiment without this flag.

Source ports

But the experiment still doesn’t work. I found the following stack trace:

Exception in thread "main" java.net.BindException: Can't assign requested
address
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:555)
        at java.base/sun.nio.ch.Net.bind(Net.java:544)
        at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:643)
        at
java.base/java.net.DelegatingSocketImpl.bind(DelegatingSocketImpl.java:94)
        at java.base/java.net.Socket.bind(Socket.java:682)
        at java.base/java.net.Socket.<init>(Socket.java:506)
        at java.base/java.net.Socket.<init>(Socket.java:403)
        at Main.main(Main.java:137)

The final battle is with the TCP/IP specification. At this point, we have fixed the server address, server port and client IP address. At the same time, we have only 16 bits of freedom left, that is, we can only open 65 thousand connections.

This is completely insufficient for our experiment. We can’t change either the server’s IP or the server’s port, because that’s the problem we’re investigating in this experiment. It remains possible to change the client IP, which gives us access to another 32 bits. As a result, we get around the limitation by conservatively assigning a client IP address for every 5000 client connections. The same technique was used in experiment with Phoenix.

In bigSur 11.4, you can add a series of fake loopback addresses with the following command:

for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i/8 up  ; done 

To test the operation of IP addresses, you can ping them:

for i in `seq 0 200`; do ping -c 1 10.0.0.$i  ; done 

To remove, use the following command:

for i in `seq 0 200`; do sudo ifconfig lo0 alias 10.0.0.$i  ; done 

On Ubuntu 20.04, you will need to use the tool instead ip:

for i in `seq 0 200`; do sudo ip addr add 10.0.0.$i/8 dev lo; done 

To remove, use the command:

for i in `seq 0 200`; do sudo ip addr del 10.0.0.$i/8 dev lo; done 

results

On Mac

I managed to reach 80000 connections. However, a few minutes after the experiment was completed, my poor Mac mysteriously crashed every time without crash reports in

/Library/Logs/DiagnosticReports

so I wasn’t able to diagnose what was wrong.

The TCP send and receive buffers on my Mac are 131072 bytes:

sysctl net | grep tcp | grep -E '(recv)|(send)'
net.inet.tcp.sendspace: 131072
net.inet.tcp.recvspace: 131072

So maybe it happened because of what I used 80000 подключений *131072 байт на буфер * 2 буфера ввода и вывода * 2 клиентских и серверных подключения bytes, which is approximately 39 GB of virtual memory. Or maybe Mac OS doesn’t like what I’m using 80000*2*2=320000 file descriptors. Unfortunately, I’m not familiar with debugging on a Mac without crash reporting, so if anyone has any information on the subject, please email me.

On Linux I managed to reach 840000 connections! However, during the experiment, it took several seconds to register the movement of the mouse across the screen. As the number of connections increased, Linux began to hang and stop responding.

To understand which resource is causing problems, I used sysstat. You can look at the graphs generated by sysstat here.

To have sysstat capture statistics for all hardware and then generate graphs, I used the following command:

sar -o out.840000.sar -A 1 3600 2>&1 > /dev/null  &
sadf -g  out.840000.sar -- -w -r -u -n SOCK -n TCP -B -S -W > out.840000.svg

Curious facts:

  • MBmemfree showed the least memory, 96 MB
  • MBavail showed 1587 MB
  • MBmemused showed only 1602 MB (19.6% of my 8 GB)
  • MBswpused at the peak it showed 1086 MB (despite the fact that there was still free memory)
  • 1680483 sockets (840k server sockets and 840k client connections plus what worked on my desktop)
  • A few seconds after the start of the experiment, the operating system decided to use swap, although I still had memory

To determine the standard send and receive buffer sizes on Linux,

you can use this command:

# минимальное, стандартное и максимальное значения размера памяти (в байтах)
cat /proc/sys/net/ipv4/tcp_rmem
4096    131072  6291456
cat /proc/sys/net/ipv4/tcp_wmem
4096    16384   4194304

sysctl net.ipv4.tcp_rmem
net.ipv4.tcp_rmem = 4096        131072  6291456
sysctl net.ipv4.tcp_wmem
net.ipv4.tcp_wmem = 4096        16384   4194304

I would need 247 GB of virtual memory to support all connections!

131072 байта для получения
16384 для записи
(131072+16384)*2*840000
=247 ГБ виртуальной памяти

I suspect that buffers were requested, but since only 4 bytes of each are needed, only a small fraction of the buffers were used. Even if I loaded 1 page of memory, because I only need to write 4 bytes to write an integer to the buffer:

getconf PAGESIZE
4096

Размер страницы 4096 байт
(4096+4096)*2*840000
=13 ГБ


then 13 GB would be used, using 2*840000 memory pages. I have no idea how it all works without fail! However, 840,000 simultaneous connections are enough for me.

You can improve my result if you have more memory, or if you further optimize the operating system parameters, for example, by reducing the sizes of TCP buffers.

  1. Phoenix framework reached 2,000,000 connections
  2. WhatsApp reached 2,000,000 connections
  3. The theoretical limit is approximately 1 quadrillion (1,000,000,000,000,000)
  4. You will run out of source ports (total 2sixteen)
  5. This can be fixed by adding client loopback IPs
  6. You will run out of file descriptors
  7. This can be fixed by changing the limits on operating system file descriptors
  8. Java will also limit the number of file descriptors
  9. This can be fixed by adding a JVM argument -XX:MaxFDLimit
  10. On my 16 GB Mac, the practical limit was 80,000 connections
  11. On my 8 GB Linux desktop, the practical limit was 840,000 connections

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *