Bridge MQTT Data from EMQX Cloud to Confluent Cloud on GCP

Where to go after Google IoT Core?

EMQX Cloud

Confluent Cloud

Create Deployments

Create EMQX Cluster

Create Kafka Cluster

# Install Confluent CLI
sudo apt-get update -y && sudo apt-get dist-upgrade -y && sudo apt autoremove -y
curl -sL --http1.1 https://cnfl.io/cli | sh -s -- -b /usr/local/bin
confluent update
# Environment preparation
confluent login --save
confluent environment use env-xxx
confluent kafka cluster use lkc-xxx
# Create an API key and secret for kafka authentication
confluent api-key create --resource lkc-xxx
confluent api-key use <API Key> --resource lkc-xxx
# Create a topic named emqx
confluent kafka topic create emqx
confluent kafka topic list
# Go to the confluent instance and view the emqx topic
confluent kafka topic consume -b emqx

VPC Peering Setup

  1. Go to the Networking section of the Cluster settings page and click on the Add Peering button.
  2. Fill in the vpc information, you could get the information from VPC Peering section of the deployment console.

Data Integrations

SELECT 
timestamp as up_timestamp,
clientid as client_id,
payload.temp as temp,
payload.hum as hum
FROM
"temp_hum/emqx"
# kafka topic
emqx
# kafka message template
{"up_timestamp": ${up_timestamp}, "client_id":
${client_id}, "temp": ${temp}, "hum": ${hum}}

Verification

  1. Use MQTTX to simulate temperature and humidity data reporting
confluent kafka topic consume -b emqx

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
EMQ Technologies

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.