Tuning your Linux kernel and HAProxy instance for high loads

How to Tune HAProxy Maxconn for High Loads in 2023

If you‘re using HAProxy to load balance high traffic websites and applications, you know how critical performance is. One of the most important settings that affects HAProxy performance and scalability is maxconn, which sets the maximum number of concurrent connections. Tuning maxconn properly is essential to ensuring HAProxy can handle heavy traffic loads without crashing or dropping connections.

In this guide, you‘ll learn everything you need to know about configuring maxconn for optimal performance based on the latest HAProxy best practices for 2023. I‘ll explain what maxconn does, how to determine the right values for your use case, and how maxconn interacts with other key system and kernel settings to affect performance under heavy load. Whether you‘re new to HAProxy or an experienced user, this guide will help you get the most out of this powerful load balancer.

What is HAProxy maxconn?

The maxconn setting in HAProxy specifies the maximum number of concurrent connections the load balancer will accept. This includes both connections from clients and connections to backend servers. Once the maxconn limit is reached, any additional incoming connections will be queued in the kernel socket queue until an existing connection closes and a slot becomes available.

It‘s important to set maxconn high enough to handle your expected peak traffic levels, but not so high that it overwhelms your available server resources. When the number of connections exceeds what the server can handle, you may see issues like connections timing out, servers becoming unresponsive, or even HAProxy crashing.

Determining maxconn values

So how do you determine the right maxconn values for your environment? Here are some key factors to consider:

1. Server resources – The amount of memory and CPU available on your HAProxy server will determine how many concurrent connections it can realistically handle. A good rule of thumb is that each connection requires about 50KB of memory, so if you have a server with 32GB RAM, a reasonable maxconn would be around 600,000. However, this is just a rough guide – you‘ll need to test and monitor to determine the true viable connection limit for your server.

2. Expected traffic – Consider your application‘s expected traffic patterns, both average and peak. Look at metrics like requests per second, concurrent users, and new SSL/TLS connections per second. Your maxconn should be set such that HAProxy and your backend servers can handle your peak traffic levels without getting overwhelmed.

3. Use case – Is your HAProxy primarily doing Layer 4 TCP load balancing or Layer 7 HTTP load balancing? HTTP load balancing is more resource intensive so you may need a lower maxconn than pure TCP, especially if you‘re doing SSL/TLS offloading. Are your backend servers handling long-lived connections like MQTT or WebSockets or short-lived connections like HTTP? Long-lived connections will obviously consume slots for longer.

4. Number of backends – The more backend servers you have, the more concurrent connections you can potentially handle and the higher you can set your maxconn. However, there are diminishing returns due to the increased overhead of managing connections to more backends.

As a starting point, I recommend setting your maxconn to handle 5-10X your expected peak concurrent connections, and then adjusting up or down based on testing and monitoring. So if you expect to handle 10,000 concurrent users at peak, start with a maxconn of 50,000-100,000.

Configuring maxconn settings

In an HAProxy configuration, there are actually four levels where you can specify maxconn:

1. Global maxconn – Sets the default maxconn for the entire HAProxy instance. This can be overridden at a frontend or backend level.

2. Frontend maxconn – Sets the maxconn for a particular frontend (port/IP). Incoming client connections will be limited to this value.

3. Backend maxconn – Sets the maxconn for a particular backend (group of servers). Outgoing connections to the backend servers will be limited to this value.

4. Server maxconn – Sets the maxconn for an individual server within a backend. This limits the number of concurrent connections HAProxy will make to that specific server.

Here‘s an example of how you might configure maxconn at each level:

global
    maxconn 100000

frontend www
    bind :80
    maxconn 50000
    default_backend web-backend

backend web-backend
    balance roundrobin
    maxconn 25000
    server web1 10.0.0.1:80 maxconn 2500
    server web2 10.0.0.2:80 maxconn 2500

In this example, the global maxconn default is 100,000 concurrent connections. The "www" frontend will accept up to 50,000 concurrent connections on port 80. The "web-backend" backend will allow up to 25,000 concurrent connections to be distributed among its servers. And each individual web server will only be sent a maximum of 2,500 concurrent connections.

When setting maxconn values, remember:

1. The frontend maxconn cannot exceed the global maxconn
2. The sum of the backend maxconns cannot exceed the global maxconn
3. The server maxconns cannot exceed the backend maxconn

I recommend starting with a global maxconn that matches what your server can handle, and then adjusting the frontend, backend, and server maxconns to control how the connections get distributed based on what the respective frontends and backends can handle.

It‘s better to queue connections at the frontend before they are accepted than overwhelm the backends. And generally you want the sum of the backend maxconns to be less than the global maxconn, otherwise the global maxconn will be the bottleneck.

Related system settings

HAProxy maxconn works in conjunction with the Linux kernel TCP stack and other system settings to affect connection handling and performance. Here are a few related sysctl settings you should tune for high connection count scenarios:

net.ipv4.ip_local_port_range – This specifies the range of ephemeral ports HAProxy will use for outbound connections to backend servers. The default range of 32768-60999 provides 28,232 ports, which limits how many concurrent connections a single HAProxy process can have to a particular backend server IP/port. Increase this range to 1024-65535 for a pool of 64,512 ephemeral ports.

net.core.somaxconn – This specifies the maximum number of inbound connections that can be queued in the kernel socket backlog before being passed to HAProxy. The default is 128, which is far too low – set this to 2-3X your frontend maxconn.

net.ipv4.tcp_max_syn_backlog – Similar to somaxconn but for the SYN backlog – the queue of inbound SYN packets that haven‘t been acknowledged. Also set this to 2-3X your frontend maxconn.

net.core.netdev_max_backlog – The maximum number of packets queued on the network interface card before being handed to the CPU. The default of 1000 should be increased if you‘re using 10G or higher NICs – up to 16,000 for 10G.

fs.file-max – The system-wide limit on open file handles. Since each connection requires a file handle, this needs to be greater than your global maxconn. A good guideline is to set fs.file-max to 1024 * maxconn.

Putting it all together, if your HAProxy global maxconn is 100,000, a reasonable set of sysctl settings would be:

net.ipv4.ip_local_port_range = 1024 65535
net.core.somaxconn = 200000  
net.ipv4.tcp_max_syn_backlog = 200000
net.core.netdev_max_backlog = 16000
fs.file-max = 102400000

Remember to make these sysctl settings persistent by adding them to /etc/sysctl.conf and reloading with sysctl -p. And if using Systemd, also set the open file limit in the haproxy.service override:

[Service]
LimitNOFILE=102400000

Monitoring and troubleshooting maxconn issues

To ensure your maxconn settings are working as expected and not causing issues, it‘s important to monitor key HAProxy metrics like:

• CurrConns (current number of established connections)
• PipesUsed/PipesFree (pipes are preallocated connection slots – you want some PipesFree remaining)
• ConnRate (number of new connections per second)
• Errors like ETIMEDOUT, EADDRNOTAVAIL, EMFILE (indicate issues accepting new connections)

You should also monitor backend server metrics like active/waiting connections, request queueing/buffering, and error rates.

If you see CurrConns approaching your configured maxconns, connections building up in the socket backlog, pipes getting exhausted, or backend servers overwhelmed with connections, those are all signs you may need to adjust your maxconns and/or related system settings.

Other issues to watch out for:

• Connections being dropped or timing out before reaching HAProxy due to full socket or SYN backlogs
• Ephemeral port exhaustion (backend server IP/port combo running out of ports due to high connection rate)
• SSL/TLS errors due to too many new SSL connections overwhelming the cryptographic resources

You can often spot these issues by looking at the HAProxy logs. For example:

Nov  3 14:32:51 localhost haproxy[16989]: Server nodes/node1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms.

This error indicates a backend server refusing connections, which could be due to hitting its own maxconn limit.

Advanced techniques

Here are a few advanced techniques for dynamically managing maxconn to respond to changing load conditions:

1. Use the HAProxy Runtime API to adjust frontend/backend/server maxconns on the fly based on load. For example, if you see a backend server‘s active connections approaching its maxconn, you could increase its maxconn. Or if you see the global CurrConns approaching the maxconn, you could lower the frontend maxconns to reduce the inflow of new connections.

2. Implement connection queuing and buffering to gracefully handle brief spikes that exceed the steady state maxconn. HAProxy supports both SYN flood protection (queuing connections in the kernel backlog before accepting) and request buffering (queuing requests in HAProxy memory before sending to backend servers).

3. Use the "backup" and "check" server options to specify reserve backend servers that are only sent traffic when the primary servers become overloaded or unreachable.

4. Utilize HAProxy connection multiplexing to support more concurrent client connections. Enable the "reuse" server option and set "http-reuse" to allow multiple requests from the same client to be sent over a single backend connection. This allows you to support more concurrent clients with fewer backend connections.

5. Shard your traffic across multiple independent HAProxy instances, each with its own maxconn setting. This allows you to scale your total connection handling capability linearly by adding more HAProxy instances.

Conclusion

In this comprehensive guide, we‘ve covered everything you need to know to effectively tune HAProxy maxconn for high traffic loads, including:

• How maxconn affects HAProxy performance and how to determine reasonable values
• The different levels of maxconn settings and how to configure them
• Related Linux kernel and system settings to adjust for high connection count scenarios
• How to monitor and troubleshoot issues with maxconns being reached or exceeded
• Advanced techniques for dynamically managing maxconns to handle spiky, variable traffic

By understanding and properly configuring HAProxy maxconn, you can ensure your load balancer is able to efficiently handle high volumes of traffic, even in the face of sudden spikes. The best maxconn settings will depend on your specific environment and use case, but the guidelines in this article provide a framework for optimizing this critical setting.

Remember, maxconn is just one piece of the performance puzzle. Be sure to also tune your other HAProxy settings, backend server resources, and system kernel parameters holistically. And always test your configuration under realistic load before deploying to production.

Hopefully this guide has armed you with the knowledge and best practices to get the most out of HAProxy as a high performance load balancer. If you have any other tips and techniques for tuning maxconn, please share them in the comments!