Sticky session load balancing — MQTT broker clustering part 2

MQTT session

In order to continuously receive messages, MQTT clients usually subscribe to MQTT broker with a long-living connection. It is not unusual that the connection might be broken for a while due to network issues or client software maintenance reasons, but the clients often wish to receive messages published during the time when the connection was broken.

Session takeover

Things get a bit more complicated when MQTT brokers form a cluster. From the client’s perspective, there are more than one brokers to connect to, and it’s hard to know which broker is the best to connect to. We need another critical component in the network: the load balancer. The load balancer becomes the access point for the entire cluster, and routes the connections from clients to one of the brokers in the cluster.

Sticky session to the rescue

The word ‘sticky’ here is referring to the ability of the load balancer being able to route the client to the old broker at reconnect, which can avoid session take over. This is an especially useful feature when there are many clients reconnecting around the same time, or in case of a problematic client repeatedly disconnecting and connecting again.

Sticky session with HAProxy 2.4

To minimise the prerequisites, in this demo cluster, we’ll start two EMQ X nodes and an HAProxy 2.4 in docker containers.

Create a docker network

In order for the containers to connect to each other, we create a docker network for them.

docker network create test.net

Start two EMQ X 4.3 nodes

In order for the nodes to connect to each other, the container name and the EMQX node name should be assigned within the network namespace (test.net).

docker run -d \
--name n1.test.net \
--net test.net \
-e EMQX_NODE_NAME=emqx@n1.test.net \
-e EMQX_LISTENER__TCP__EXTERNAL__PROXY_PROTOCOL=on \
emqx/emqx:4.3.7
docker run -d \
--name n2.test.net \
--net test.net \
-e EMQX_NODE_NAME=emqx@n2.test.net \
-e EMQX_LISTENER__TCP__EXTERNAL__PROXY_PROTOCOL=on \
emqx/emqx:4.3.7

Make EMQ X nodes join a cluster

docker exec -it n2.test.net emqx_ctl cluster join emqx@n1.test.net
[EMQ X] emqx shutdown for join
Join the cluster successfully.
Cluster status: #{running_nodes => ['emqx@n1.test.net','emqx@n2.test.net'], stopped_nodes => []}

Start HAProxy 2.4

Create a file /tmp/haproxy.configwith below content

global
log stdout format raw daemon debug
nbproc 1
nbthread 2
cpu-map auto:1/1-2 0-1
# Enable the HAProxy Runtime API
# e.g. echo "show table emqx_tcp_back" | sudo socat stdio tcp4-connect:172.100.239.4:9999
stats socket :9999 level admin expose-fd listeners

defaults
log global
mode tcp
option tcplog
maxconn 1024000
timeout connect 30000
timeout client 600s
timeout server 600s

frontend emqx_tcp
mode tcp
option tcplog
bind *:1883
default_backend emqx_tcp_back

backend emqx_tcp_back
mode tcp

# Create a stick table for session persistence
stick-table type string len 32 size 100k expire 30m

# Use ClientID / client_identifier as persistence key
stick on req.payload(0,0),mqtt_field_value(connect,client_identifier)

# send proxy-protocol v2 headers
server emqx1 n1.test.net:1883 check-send-proxy send-proxy-v2
server emqx2 n2.test.net:1883 check-send-proxy send-proxy-v2
docker run -d \
--net test.net \
--name proxy.test.net \
-p 9999:9999 \
-v /tmp/haproxy.cfg:/haproxy.cfg \
haproxy:2.4 haproxy -f /haproxy.cfg

Test it out

Now we use the popular mosquitto MQTT client (also in docker) to test it out.

docker run --rm -it --net test.net eclipse-mosquitto \
mosquitto_sub -h proxy.test.net -t 't/#' -I subscriber1
docker run --rm -it --net test.net eclipse-mosquitto \
mosquitto_pub -h proxy.test.net -t 't/xyz' -m 'hello'

Inspect the sticky table in HAProxy

We can also inspect the sticky table created in HAProxy with this command. It requires socat command, so I am running it from the docker host.

show table emqx_tcp_back" | sudo socat stdio tcp4-connect:127.0.0.1:9999
# table: emqx_external_tcp_listners, type: string, size:102400, used:1
0x7f930c033d90: key=subscriber1 use=0 exp=1793903 server_id=2 server_key=emqx2

Other articles in this series

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
EMQ Technologies

EMQ Technologies

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.