Highly Reliable MQTT Data Persistence Based on RocksDB

Introduction: Native MQTT Session Persistence Support

Understanding MQTT Sessions

QoS 1 message flow diagram
QoS 2 message flow diagram
  • QoS 1 and QoS 2 messages that have been sent to the server but not yet fully acknowledged
  • QoS 2 messages that have been received from the server but not yet fully acknowledged
  • The existence status of the session, even if the session is empty
  • Client subscription messages
  • QoS 1 and QoS 2 messages that have been sent to the client but have not been fully acknowledged
  • Waiting for QoS 0 (optional), QoS 1, and QoS 2 messages to be transmitted to the client
  • QoS 2 messages, Will Message and Will Delay Interval that have been received from the client but not yet fully acknowledged

Session life Cycle and Session Storage

Relationship between Session lifecycle and Clean Session in MQTT 3.1.1
The Relationship Between Session lifecycle and Session Expiry Interval in MQTT 5.0

Session Persistence Design in Previous Versions of EMQX

  1. EMQX solves the core problem of connectivity and routing. In rare cases, messages need to be stored persistently, and reserved messages as a special case are supported to be stored on disk.
  2. EMQX is a cloud service, and the server stability is reliable enough in this kind of environment. Even if the messages are in the memory, there is not much risk of loss.
  3. The built-in persistence design requires a trade-off between memory and disk usage in high-throughput scenarios, and data storage and replication design in a multi-server distributed cluster architecture, which makes it difficult to ensure the persistence design is in place in a single step in a fast-growing project.

Why RocksDB: A New Session Layer Selection

RocksDB Introduction

Selection Basis

  • Mnesia: Mnesia is a distributed real-time database system built in Erlang/OTP. All the nodes of the Mnesia cluster are equal. Each of these nodes can store a copy of the data and can also start a transaction or perform a read or write operation. Mnesia can support extremely high read due to its replication feature, but this also limits its write throughput as it means MQTT messages are largely broadcast within the cluster and the broadcast cannot scale out.
  • LevelDB: RocksDB is an improved branch of LevelDB, and they are mostly functionally equivalent, but LevelDB lacks an actively maintained driver in Erlang (Erlang NIF). So, it has not been adopted.
Mnesia mesh topology
  • Extremely high write throughput: RocksDB is based on an LSM-Tree structure optimized for data writes, capable of supporting EMQX massive message throughput and high-frequency data writes during fast subscriptions
  • Iterators and fast range queries: RocksDB supports iteration over sorted keys. Based on this feature, EMQX can be extended with more features
  • Support for Erlang: The NIF library for RocksDB is mature and actively supported

EMQX Session Persistence Design Based on RocksDB

What Data can be Persisted with RocksDB

  1. Session records for clients connected with Clean Start = 0
  2. Subscriptions are written to RocksDB when Subscriptions are made and deleted from RocksDB when Subscriptions are canceled
  3. Every time the client publishes a message QoS 1, QoS 2 message, the data will be written to RocksDB and reserved until it is confirmed and then deleted
  4. Serves as a Storage for other scenarios with high throughput and low latency, such as message retention and data bridging cache queues

Extension of Persistence Capability

  1. The publisher publishes a persistent message
  2. EMQX stores the message in the replay queue, without caring whether the subscriber is online or not
  3. The subscriber initiates a subscription
  4. EMQX reads the message from the specified location
  5. The replay message is published to the subscriber

Epilogue

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
EMQ Technologies

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.