Open in app

Sign in

Write

Sign in

Building K8S cluster of EMQ X MQTT broker starting from scratch

EMQ Technologies
15 min readJun 30, 2020

EMQ X Team provides Helm chart which facilitates users to one-click deploy EMQ X MQTT broker in the Kubernetes cluster. EMQ X Team mostly recommends This method for deploying EMQ X MQTT broker in the Kubernetes or k3s cluster. This article will start from scratch using the handwriting YAML file method to deploy a K8S cluster of EMQ X MQTT broker, and analyze details and techniques of the deployment. It will facilitate users to flexibly use during real deployment.

Reading this article needs users to know the basic concept of Kubernetes and having an operational Kubernetes cluster.

Deploy single EMQ X MQTT broker node in K8S

Use pod directly deploy EMQ X Broker

In Kubernetes, the smallest management element is Pod than individual containers. Pod is the basic executing unit of Kubernetes applications, which means that it is the smallest and simplest unit be created or deployed in the Kubernetes object model. Pod represents the processes running on the cluster.

EMQ X Broker has provided mirroring in docker hub, so users can easily deploy EMQ X Broker in the single pod. Using the command kubectl run to create a pod running EMQ X Broker.

$ kubectl run emqx --image=emqx/emqx:v4.1-rc.1  --generator=run-pod/v1
pod/emqx created

View the status of EMQ X Broker:

$ kubectl get pods -o wide
NAME   READY   STATUS    RESTARTS   AGE
emqx   1/1     Running   0          3m13s$ kubectl exec emqx -- emqx_ctl status
Node 'emqx@192.168.77.108' is started
emqx 4.1-rc.1 is running

Delete pod:

$ kubectl delete pods emqx
pod "emqx" deleted

Pod is not designed as a persistence resource, it will not survive in the scheduling failing, node crashing or other recovering(such as because of lack of resource or other maintenances). Therefore, a controller is needed to manage Pod.

Use Deployment deploy Pod

Deployment provides a declarative definition(declarative) method for pod and ReplicaSet to replace the previous ReplicationController for easily managing applications. The typical applying scenarios including:

  • Define Deployment to create Pod and ReplicaSet
  • Scroll upgrade and rollback applications
  • Expand and shrink
  • Pause and continue Deployment

Use Deployment deploy an EMQ X Broker Pod：

  • Define Deployment：
$ cat deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
         containerPort: 1883
        - name: mqttssl
         containerPort: 8883
        - name: mgmt
         containerPort: 8081
        - name: ws
         containerPort: 8083
        - name: wss
         containerPort: 8084
        - name: dashboard
         containerPort: 18083
  • Deploy Deployment：
$  kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created
  • View Deployment situation:
$ kubectl get deployment
NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/emqx-deployment   3/3     3            3           74s$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
pod/emqx-deployment-7c44dbd68-8j77l   1/1     Running   0          74s$ kubectl exec pod/emqx-deployment-7c44dbd68-8j77l -- emqx_ctl status
Node 'emqx-deployment-7c44dbd68-8j77l@192.168.77.117' is started
  • Trying manually delete Pod
$ kubectl delete pods emqx-deployment-7c44dbd68-8j77l
pod "emqx-deployment-7c44dbd68-8j77l" deleted$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-68fcb4bfd6-2nhh6   1/1     Running   0          59s

The output result represents that successfully using Deployment deploy EMQ X Broker Pod. Even if this pod is accidentally stopped, Deployment will recreate a new pod.

Use services exposing EMQ X Broker Pod service

Kubernetes Pods has a life cycle. They can be created, and will not run if they are destroyed. It can dynamically create and destroy pod, if use Deployment to run applications.

Every pod has its IP address, but in Deployment, the pod collection running at the same moment may differ from that runs this application later.

This will cause a problem: if use EMQ X Broker Pod to provide service to the MQTT client, how does the client find and track the IP address that will be connected, to facilitate users to use EMQ X Broker service.

The answer is: Service

Service is an abstract method for exposing the application which is running on a set of Pods as network service.

Use Service to expose EMQ X Broker Pod as network service.

  • Define Service：
$cat service.yamlapiVersion: v1
kind: Service
metadata:
  name: emqx-service
spec:
  selector:
    app: emqx
  ports:
    - name: mqtt
      port: 1883
      protocol: TCP
      targetPort: mqtt
    - name: mqttssl
      port: 8883
      protocol: TCP
      targetPort: mqttssl
    - name: mgmt
      port: 8081
      protocol: TCP
      targetPort: mgmt
    - name: ws
      port: 8083
      protocol: TCP
      targetPort: ws
    - name: wss
      port: 8084
      protocol: TCP
      targetPort: wss
    - name: dashboard
      port: 18083
      protocol: TCP
      targetPort: dashboard
  • Deploy Service：
$ kubectl apply -f service.yaml
service/emqx-service created

View deployment situation

$ kubectl get svc
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                                        AGE
emqx-service   ClusterIP   10.96.54.205   <none>        1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP   58s
  • Use the IP provided by Service to view API of EMQ X Broker
$ curl 10.96.54.205:8081/status
Node emqx-deployment-68fcb4bfd6–2nhh6@192.168.77.120 is started
emqx is running

So far, the deployment of a single EMQ X Broker node is done. Manage EMQ X Broker Pod through Deployment, and expose EMQ X Broker service through Service.

Automatically cluster EMQ X MQTT broker through Kubernetes

The article above introduced how to deploy single EMQ X Broker Pod through Deployment. It is extremely convenient to expand the number of Pod through Deployment, execute the command kubectl scale deployment ${deployment_name} --replicas ${numer} can expand the number of Pod. Expanding EMQ X Broker Pod to three is as follows:

$ kubectl scale deployment emqx-deployment --replicas 3
deployment.apps/emqx-deployment scaled$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-68fcb4bfd6-2nhh6   1/1     Running   0          18m
emqx-deployment-68fcb4bfd6-mpvch   1/1     Running   0          6s
emqx-deployment-68fcb4bfd6-mx55q   1/1     Running   0          6s$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl status
Node 'emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120' is started
emqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120'],
                  stopped_nodes => []}

You can see that the number of EMQ X Broker Pod is expanded to three, but each pod is separated, and there are no clusters. Next try to automatically cluster EMQ X Broker Pod through Kubernetes.

Modify EMQ X Broker configuration

View the content related to automatically cluster in the EMQ X Broker documentation, you can see that we need to modify the configuration of EMQ X Broker.

cluster.discovery = kubernetes
cluster.kubernetes.apiserver = http://10.110.111.204:8080
cluster.kubernetes.service_name = ekka
cluster.kubernetes.address_type = ip
cluster.kubernetes.app_name = ekka

cluster.kubernetes.apiserver is the address of Kubernetes apiserver, we can get it through this command kubectl cluster-info. cluster.kubernetes.service_name is the Service name we mentioned above. cluster.kubernetes.app_name is the part before the @ symbol in node.name of EMQ X Broker, so you also need to set the EMQ X Broker in the cluster as a uniform prefix of node.name.

The docker mirroring of EMQ X Broker provides the function of modifying configurations through environment variables. For more details, you can check docker hub or Github.

  • Modify the yaml file of Deployment and add environment variables:
$ cat deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        env:
        - name: EMQX_NAME
         value: emqx
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx-service
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: "https://kubernetes.default.svc:443"
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default

Because this command `kubectl scale deployment ${deployment_name} --replicas ${numer} will not modify file yaml, needs to set spec.replicas: 3 when modify yaml.

https://kubernetes.default.svc:443 will be parsed as the address of Kubernetes apiserver because of the build-in DNS rules of Kubernetes.

  • Delete the previous Deployment and then redeploy:
$ kubectl delete deployment emqx-deployment
deployment.apps “emqx-deployment” deleted
​
$ kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created

Give authority to pod for accessing Kubernetes apiserver

After successfully deploy Deployment, check the status of EMQ X Broker, then you can see that although EMQ X Broker runs successfully, the cluster is still not successful. Check the log of EMQ X Broker Pod:

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-5c8cfc4d75-67lmt   1/1     Running   0          5s
emqx-deployment-5c8cfc4d75-r6jgb   1/1     Running   0          5s
emqx-deployment-5c8cfc4d75-wv2hj   1/1     Running   0          5s$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl status
Node 'emqx@192.168.87.150' is started
emqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl cluster status
Cluster status: #{running_nodes => ['emqx@192.168.87.150'],
                  stopped_nodes => []}
                  
$ kubectl logs emqx-deployment-76f6895c46-4684f···
(emqx@192.168.87.150)1> 2020-05-20 01:48:39.726 [error] Ekka(AutoCluster): Discovery error: {403,
                                     "{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"endpoints \\\"emqx-service\\\" is forbidden: User \\\"system:serviceaccount:default:default\\\" cannot get resource \\\"endpoints\\\" in API group \\\"\\\" in the namespace \\\"default\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"emqx-service\",\"kind\":\"endpoints\"},\"code\":403}\n"}
···Pod is refused because of authority, when it accesses Kubernetes apiserver and return HTTP 403, the cluster failed.

The normal pod can not access Kubernetes apiserver. There are two methods that can figure out this problem. The first one is to open the HTTP interface of Kubernetes apiserver, but some security risks are exiting. Another one is through ServiceAccount, Role and RoleBinding to configure RBAC authentication.

  • Define ServiceAccount, Role and RoleBinding:
$ cat rbac.yamlapiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: emqx
---
kind: Role
apiVersion: rbac.authorization.kubernetes.io/v1beta1
metadata:
  namespace: default
  name: emqx
rules:
- apiGroups:
  - ""
  resources:
  - endpoints 
  verbs: 
  - get
  - watch
  - list
---
kind: RoleBinding
apiVersion: rbac.authorization.kubernetes.io/v1beta1
metadata:
  namespace: default
  name: emqx
subjects:
  - kind: ServiceAccount
    name: emqx
    namespace: default
roleRef:
  kind: Role
  name: emqx
  apiGroup: rbac.authorization.kubernetes.io
  • Deploy the corresponding resources:
$ kubectl apply -f rbac.yaml
serviceaccount/emqx created
role.rbac.authorization.kubernetes.io/emqx created
rolebinding.rbac.authorization.kubernetes.io/emqx created
  • Modify file yaml of Deployment, add spec.template.spec.serviceAccountName, and redeploy:
$cat deployment.yaml
​
apiVersion: apps/v1
kind: Deployment
metadata:
 name: emqx-deployment
 labels:
 app: emqx
spec:
 replicas: 3
 selector:
 matchLabels:
 app: emqx
 template:
 metadata:
  • View status:
$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-6b854486c-dhd7p   1/1     Running   0          10s
emqx-deployment-6b854486c-psv2r   1/1     Running   0          10s
emqx-deployment-6b854486c-tdzld   1/1     Running   0          10s$ kubectl exec emqx-deployment-6b854486c-dhd7p  -- emqx_ctl status
Node 'emqx@192.168.77.92' is started
emqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-6b854486c-dhd7p  -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.115','emqx@192.168.77.92',
                       'emqx@192.168.87.157'],
                  stopped_nodes => []}
  • abort a Pod:
$ kubectl delete pods emqx-deployment-6b854486c-dhd7p
pod "emqx-deployment-6b854486c-dhd7p" deleted$ kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
emqx-deployment-6b854486c-846v7   1/1     Running   0          56s
emqx-deployment-6b854486c-psv2r   1/1     Running   0          3m50s
emqx-deployment-6b854486c-tdzld   1/1     Running   0          3m50s$ kubectl exec emqx-deployment-6b854486c-846v7 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.115','emqx@192.168.77.84',
                       'emqx@192.168.87.157'],
                  stopped_nodes => ['emqx@192.168.77.92']}

The output result represents that EMQ X Broker will correctly display the stopped pod, and add the new created Pod to the cluster.

So far, EMQ X Broker has successfully created cluster in kubernetes.

Persistence EMQ X Broker cluster

The Deployment used above to manage Pod, but the network of Pod is constantly changeable. When the pod is destroyed and rebuild, the data and configuration stored in EMQ X Broker also disappeared, this is can not be accepted during production. Next try to persistence EMQ X Broker cluster. Even if the pod is destroyed and rebuild, the data of the EMQ X Broker can be retained.

ConfigMap

ConfigMap is an API object which is used to store non-confidential data to key-value pairs. It will be used as an environment variables, command-line arguments, or as configuration files in a volume.

ConfigMap decouples your environment configuration information from container mirroring, so that you can easily modify application configuration.

ConfigMap does not provide secrecy or encryption. If the data you want to store are confidential, use a Secret rather than a ConfigMap, or use third-party tools to keep your data private.

Next use ConfigMap to record the configuration of EMQ X Broker, and import them as environment variables to the Deployment.

  • Define Configmap and deploy:
$cat configmap.yamlapiVersion: v1
kind: ConfigMap
metadata:
  name: emqx-config
data:
  EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
  EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443"
  EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"
  
$ kubectl apply -f configmap.yaml
configmap/emqx-config created
  • Configure Deployment for using Configmap
$cat deployment.yamlapiVersion: apps/v1
kind: Deployment
metadata:
  name: emqx-deployment
  labels:
    app: emqx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        envFrom:
          - configMapRef:
              name: emqx-config
  • Redeploy Deployment and view status
$ kubectl delete -f deployment.yaml
deployment.apps "emqx-deployment" deleted$ kubectl apply -f deployment.yaml
deployment.apps/emqx-deployment created$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-deployment-5c7696b5d7-k9lzj   1/1     Running   0          3s
emqx-deployment-5c7696b5d7-mdwkt   1/1     Running   0          3s
emqx-deployment-5c7696b5d7-z57z7   1/1     Running   0          3s$ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl status
Node 'emqx@192.168.87.149' is started
emqx 4.1-rc.1 is running$ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.106','emqx@192.168.77.107',
                       'emqx@192.168.87.149'],
                  stopped_nodes => []}

The configuration files of EMQ X Broker have decoupled to Configmap. If necessary, you can freely configure one or more Configmap, and import them as environment variables or files to the pod.

StatefulSet

StatefulSet is used for figuring out the problem that is stateful service(Deployments and ReplicaSets are designed for stateless service). The scenarios it is applied including:

  • Stable persistence storage, that is, after the pod is re-dispatched, it can also access the same persistence data. Based on PVC to implement.
  • Stable network sign, that is, after the pod is re-dispatched, it’s PodName and HostName have no changes. Based on Headless Service(the Service without Cluster IP) to implement.
  • Orderly deployment, orderly expands, that is, the pod is sequential. So that it must be carried out in sequence according to the defined order when deploying or expanding(from 0 to N-1, all the previous pods must are Running and Ready before running the next pod). Based on init containers to implement.
  • Orderly shrink, orderly delete(from 0 to N-1)

From the above scenarios, we can find that StatefulSet consists of the following parts:

  • Headless Service for defining network sign(DNS domain)
  • VolumeClaimTemplates for creating PersistentVolumes
  • StatefulSet for defining specific applications

The DNS format of every pod in StatefulSet is statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local:

  • serviceName is the name of Headless Service
  • 0..N-1 is the number of Pod, from 0 to N-1
  • statefulSetName is the name of StatefulSet
  • namespace is the namespace where service locate in, Headless Service and StatefulSet must locate in the same namespace
  • .cluster.local is Cluster Domain

Then, using StatefulSet replace Deployment to manage pod.

  • Delete Deployment:
$ kubectl delete deployment emqx-deployment
deployment.apps “emqx-deployment” deleted
  • Define StatefulSet:
$cat statefulset.yamlapiVersion: apps/v1
kind: StatefulSet
metadata:
  name: emqx-statefulset
  labels:
    app: emqx
spec:
 serviceName: emqx-headless
  updateStrategy:
    type: RollingUpdate
  replicas: 3
  selector:
    matchLabels:
      app: emqx
  template:
    metadata:
      labels:
        app: emqx
    spec:
      serviceAccountName: emqx
      containers:
      - name: emqx
        image: emqx/emqx:v4.1-rc.1
        ports:
        - name: mqtt
          containerPort: 1883
        - name: mqttssl
          containerPort: 8883
        - name: mgmt
          containerPort: 8081
        - name: ws
          containerPort: 8083
        - name: wss
          containerPort: 8084
        - name: dashboard
          containerPort: 18083
        envFrom:
          - configMapRef:
              name: emqx-config
  • It should be noted that StatefulSet needs Headless Service to implement a stable network sign, so we need to define one Service more.
$cat headless.yamlapiVersion: v1
kind: Service
metadata:
  name: emqx-headless
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: emqx
  ports:
  - name: mqtt
    port: 1883
    protocol: TCP
    targetPort: 1883
  - name: mqttssl
    port: 8883
    protocol: TCP
    targetPort: 8883
  - name: mgmt
    port: 8081
    protocol: TCP
    targetPort: 8081
  - name: websocket
    port: 8083
    protocol: TCP
    targetPort: 8083
  - name: wss
    port: 8084
    protocol: TCP
    targetPort: 8084
  - name: dashboard
    port: 18083
    protocol: TCP
    targetPort: 18083

Because Headless Service does not require an IP, configured clusterIP: None.

  • Deploy the corresponding configurations:
$ kubectl apply -f headless-service.yaml
service/emqx-headless created$ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-deployment created$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
emqx-statefulset-0                 1/1     Running   0          2m59s
emqx-statefulset-1                 1/1     Running   0          2m57s
emqx-statefulset-2                 1/1     Running   0          2m54s$ kubectl exec emqx-statefulset-0 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@192.168.77.105','emqx@192.168.87.153',
                       'emqx@192.168.87.155'],
                  stopped_nodes => []}
  • Update Configmap:

StatefulSet provides a stable network sign. EMQ X Broker supports using hostname and DNS rules to replace IP for implementing cluster. Taking hostname as an example, need to modify emqx.conf:

cluster.kubernetes.address_type = hostname
cluster.kubernetes.suffix = “svc.cluster.local”

The DNS rules of Pod in the cluster Kubernetes can be customized by users. EMQ X Broker provides cluster.kubernetes.suffix for facilitating users to match custom DNS rules. This article use the default DNS rules: statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local. serviceName in the DNS rules is Headless Service used by StatefulSet, so we also need to modify cluster.kubernetes.service_name to Headless Service Name.

Convert configuration items to environment variables, we need to configure the following items in the Configmap:

EMQX_CLUSTER__K8S__ADDRESS_TYPE: “hostname”
EMQX_CLUSTER__K8S__SUFFIX: “svc.cluster.local”
EMQX_CLUSTER__K8S__SERVICE_NAME: emqx-headless

Configmap provides hot update. Execute $ kubectl edit configmap emqx-config to a hot update Configmap.

  • Redeploy StatefulSet:

Pod will not be re-enabled after updating Configmap, so we need to manually update StatefulSet

$ kubectl delete statefulset emqx-statefulset
statefulset.apps "emqx-statefulset" deleted$ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-statefulset created$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          115s
emqx-statefulset-1   1/1     Running   0          112s
emqx-statefulset-2   1/1     Running   0          110s$ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                  stopped_nodes => []}

We can see that the new EMQ X Broker cluster has successfully built.

  • Abort a Pod:

The PodName and HostName of Pod in the StatefulSet will not change after rescheduling. Let’s try:

$ kubectl get pods
kuNAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          6m20s
emqx-statefulset-1   1/1     Running   0          6m17s
emqx-statefulset-2   1/1     Running   0          6m15s$ kubectl delete pod emqx-statefulset-0
pod "emqx-statefulset-0" deleted$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          27s
emqx-statefulset-1   1/1     Running   0          9m45s
emqx-statefulset-2   1/1     Running   0          9m43s$ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
                      ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local',
                       'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'],
                  stopped_nodes => []}
  • As expected, StatefulSet rescheduled a pod with the same network symbol. EMQ X Broker in the pod has also successfully joined the cluster.

StorageClasses, PersistentVolume and PersistentVolumeClaim

PersistentVolume(PV) is the storage set by the administrator, and it is part of the cluster. It like the node is the resource of the cluster, PV is also the resource od cluster. PV is volume plugin like Volume, but it has a life cycle which is independent of the pod using PV. This API object includes the details of the implementation of storage, that is, NFS, iSCSI or storage systems specific to cloud vendors.

PersistentVolumeClaim(PVC) is the requirements stored by users. It is similar to Pod. Pod consumes the node resource, and PVC consumes PV resources. Pod can request resources at a specific level(CPU and RAM). The declaration can request the specific size and accessing mode(for example, can read/write once or mount in read-only multiple)

StorageClass describes the method used to store “class” for administrators. Different class may be mapped to different quality of service levels or backup strategies, or the arbitrary strategy confirmed by the cluster administrator. Kubernetes itself is not clear what the various class represent. This concept is sometimes referred to as “configuration file” in other storage systems.

You can create PV or StorageClass in advance when deploy EMQ X Broker, and then using PVC mount the directory /opt/emqx/data/mnesia of EMQ X Broker. After rescheduling Pods, EMQ X will recover the data from directory /opt/emqx/data/mnesia for implementing the persistence of EMQ X Broker.

  • Define StatefulSet
$cat statefulset.yaml
​
apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: emqx-statefulset
 labels:
 app: emqx
spec:
 replicas: 3
 serviceName: emqx-headless
 updateStrategy:
 type: RollingUpdate
 selector:
 matchLabels:
 app: emqx
 template:
 metadata:
 labels:
 app: emqx
 spec:
 volumes:
 — name: emqx-data
 persistentVolumeClaim:
 claimName: emqx-pvc
 serviceAccountName: emqx
 containers:
 — name: emqx
 image: emqx/emqx:v4.1-rc.1
 ports:
 — name: mqtt
 containerPort: 1883
 — name: mqttssl
 containerPort: 8883
 — name: mgmt
 containerPort: 8081
 — name: ws
 containerPort: 8083
 — name: wss
 containerPort: 8084
 — name: dashboard
 containerPort: 18083
 envFrom:
 — configMapRef:
 name: emqx-config
 volumeMounts:
 — name: emqx-data
 mountPath: “/opt/emqx/data/mnesia”
 volumeClaimTemplates:
 — metadata:
 name: emqx-pvc
 annotations:
 volume.alpha.kubernetes.io/storage-class: manual
 spec:
 accessModes: [ “ReadWriteOnce” ]
 resources:
 requests:
 storage: 1Gi

First, this file specifies using the storage class named manual through volumeClaimTemplates to create the PVC resources named emqx-pvc. The read and write mode of PVC resources is ReadWriteOnce, which need 1Gi space. Next, define this PVC as a volume named emqx-data, and mount this volume on the directory /opt/emqx/data/mnesia of Pod.

  • Deploy resources:

Users or cluster Kubernetes administrators need to create storage class before deploying StatefulSet.

$ kubectl apply -f statefulset.yaml
statefulset.apps/emqx-statefulset created$ kubectl get pods
NAME                 READY   STATUS    RESTARTS   AGE
emqx-statefulset-0   1/1     Running   0          27s
emqx-statefulset-1   1/1     Running   0          9m45s
emqx-statefulset-2   1/1     Running   0          9m43s$ kubectl get pvc
NAME                        STATUS    VOLUME                                 CAPACITY   ACCESS MODES   STORAGECLASS   AGE
emqx-data-emqx-statefulset-0   Bound     pvc-8094cd75-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            2m11s
emqx-data-emqx-statefulset-0   Bound     pvc-9325441d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            99s
emqx-data-emqx-statefulset-0   Bound     pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064   1Gi        RWO            gp2            56s
  • The output result represents that the status of this PVC is Bound, and PVC storage has been successfully established. EMQ X Broker will read the data is mounted in PVC for implementing persistence, when rescheduling pod.

So far, the process that EMQ X Broker builds persistence cluster in the Kubernetes has been completed. This article omits some details, and the process of deployment is also for the simple Demo. Users can read kubernetes documentation and Helm chart source code provided by the EMQ X Team for more in-depth research. Of course, contributing issues, pulling requests and star on Github are welcome.

Copyright notice: this article was originally written by EMQ. If you want to reprint, please indicate the source clearly. The link of original article:https://www.emqx.io/blog/emqx-mqtt-broker-k8s-cluster

K8s
Emq
Mqtt
IoT

--

--

EMQ Technologies

Written by EMQ Technologies

732 Followers

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams