Design of Million-Level Message Throughput Architecture for an IoV Platform

Related Factors of Message Throughput Design in IoV Scenarios

Message Frequency

Message Packet Size

Message Latency

Number and Level of Topic

Payload Codec

EMQX+Kafka Building Million-Level Throughput IoV Platform

Architecture Design

  1. The connection and message of front-end vehicle machines can be used as domain name forwarding through load balancing products provided by public cloud vendors. If TLS/DTLS security authentication is adopted, four HAProxy/Nginx servers can be established on the cloud as certificate unloading and load balancing.
  2. Ten EMQX nodes are used to form a large cluster, which divides 1 million message throughput into 100,000 message throughput per node on average, while meeting the requirements of high-availability scenarios.
  3. Redis can be used as the storage database if there is a need for offline/message caching.
  4. As the overall message queue, EMQX forwards the full message to the back-end Kafka cluster via the rule engine.
  5. Applications such as the back-end TSP platform/OTA receive corresponding messages by subscribing to the topic of Kafka. The control instructions and push messages of the service platform can be sent to EMQX by means of Kafka/API.
  • It supports ten- million-level vehicle connection and million-level message throughput capability.
  • The distributed cluster architecture is stable and reliable, which can support dynamic horizontal expansion.
  • Strong rule engine and data bridging, persistence capability, which can support million-level message throughput processing.
  • Rich API and certification system, which achieves smooth connection.

Verification of Million Throughput Scenarios

EMQX Cluster Dashboard Statistics
EMQX Rule Engine Statistics
Statistics of Kafka management interface

EMQX+InfluxDB Building Million-Level Throughput IoV Platform

Architecture Design

  1. The front-end device messages are used as domain name forwarding and load balancing via cloud vendor’s load balancing products.
  2. One EMQX node is used for testing this time. If necessary, multi-node mode can be adopted to form a corresponding cluster scheme (10 EMQX nodes can be deployed in the test of 1 million nodes).
  3. Redis can be used as the storage database if there is a need for offline/message caching.
  4. EMQX forwards total messages through the rule engine to the back-end InfluxDB for data persistence storage.
  5. The back-end big data platform receives corresponding messages through InfluxDB, analyzes the big data, and then transmits the desired information to EMQX through APIs.

Scenario Verification

EMQX Dashboard Statistics
EMQX Rule Engine Statistics
Data received in InfluxDB database
Statistics of EMQX Dashboard Messages

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
EMQ Technologies

EMQ Technologies

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.