Open in app

Sign in

Write

Sign in

EMQ X Auto-cluster in Kubernetes

EMQ Technologies
6 min readNov 5, 2018

--

EMQ X supports automatic clustering through the Kubernetes service. You need to use kubeadm to create a 3 nodes cluster in Ubuntu before starting to build an EMQ X cluster as introduced in below.

Notice: This article requires you have the basic understanding for kubernetes cluster resources, such as pods, services, etc. You can read the documentation to learn it.

Lab Environment

  • Public cloud environment: AWS EC2
  • Operating system: Ubuntu 16.04
  • kubeadm version: v1.12.1
  • Docker version: 18.6.1
  • Cluster node:
| hostname   | Node role | IP address    |
| ---------- | --------- | ------------- |
| kube-node1 | master    | 172.31.18.155 |
| kube-node2 | worker    | 172.31.21.171 |
| kube-node3 | worker    | 172.31.20.189 |

Ready to Work

View kubernetes Cluster Status

$ kubectl get node -o wide
NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
kube-node1   Ready    master   28h   v1.12.1   172.31.18.155   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node2   Ready    <none>   28h   v1.12.1   172.31.21.171   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1
kube-node3   Ready    <none>   28h   v1.12.1   172.31.20.189   <none>        Ubuntu 18.04.1 LTS   4.15.0-1021-aws   docker://18.6.1$ kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE     IP          NODE         NOMINATED NODE
kube-system   coredns-576cbf47c7-b5xbb             1/1     Running   0          28h   10.244.0.5      kube-node1   <none>
kube-system   coredns-576cbf47c7-g5s9j             1/1     Running   0          28h   10.244.0.4      kube-node1   <none>
kube-system   etcd-kube-node1                      1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-apiserver-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-controller-manager-kube-node1   1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-flannel-ds-amd64-fscrm          1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-flannel-ds-amd64-hhj8b          1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-flannel-ds-amd64-l6ccn          1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-proxy-79thv                     1/1     Running   0          28h   172.31.20.189   kube-node3   <none>
kube-system   kube-proxy-ckg9t                     1/1     Running   0          28h   172.31.21.171   kube-node2   <none>
kube-system   kube-proxy-skq8m                     1/1     Running   0          28h   172.31.18.155   kube-node1   <none>
kube-system   kube-scheduler-kube-node1            1/1     Running   0          28h   172.31.18.155   kube-node1   <none>

View apiserver Configuration

EMQ X automatic clustering function need to use kubernetes apiserver. Let’s firstly take a look at configuration of apiserver.

$ kubectl describe pods kube-apiserver-kube-node1 -n kube-system  
Name:               kube-apiserver-kube-node1
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               kube-node1/172.31.18.155
Start Time:         Tue, 23 Oct 2018 02:29:37 +0000
Labels:             component=kube-apiserver
                    tier=control-plane
Annotations:        kubernetes.io/config.hash: c5ac975e628056601100307026359ba8
                    kubernetes.io/config.mirror: c5ac975e628056601100307026359ba8
                    kubernetes.io/config.seen: 2018-10-17T02:00:07.757483468Z
                    kubernetes.io/config.source: file
                    scheduler.alpha.kubernetes.io/critical-pod: 
Status:             Running
IP:                 172.31.18.155
Containers:
  kube-apiserver:
    Container ID:  docker://d753a932b41ff8a99b6a11767d463f13af7e9de7526567df6eb5bd29a101f17b
    Image:         k8s.gcr.io/kube-apiserver:v1.12.1
    Image ID:      docker-pullable://k8s.gcr.io/kube-apiserver@sha256:52b9dae126b5a99675afb56416e9ae69239e012028668f7274e30ae16112bb1f
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --authorization-mode=Node,RBAC
      --advertise-address=172.31.18.155
      --allow-privileged=true
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Tue, 23 Oct 2018 02:29:39 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Oct 2018 02:00:09 +0000
      Finished:     Thu, 18 Oct 2018 12:43:31 +0000
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:        250m
    Liveness:     http-get https://172.31.18.155:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:            <none>

Obtain the IP address of the apiserver by —-advertis-address=172.31.18.155 (this address can be specified by —-apiserver-advertise-address=172.31.18.155 parameter when using kubeadm init command to create a cluster).

Access apiserver

Type kubectl proxy --help command to see that the kubectl proxy command can create a proxy server between local machine and apiserver. It allows http access to the specified rules.

$ kubectl proxy --accept-hosts="^.*$" --address='172.31.18.155' -p=8080 &
$ curl http://172.31.18.155:8080
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1",
    "/apis/apiregistration.k8s.io/v1beta1",
    ......

Create a Resource

kubectl can create various resources through kubectl create -f xxx.yml, then you only need to write one or more valid yml files to create an EMQ X cluster. This article writes all the resources within one yml file. With actual practice, you can create multiple yml files in one directory and deploy them using kubectl create -f directory name.

| Resource Type | Name |
| ------------- | ---- |
| service       | emqx |
| deployment    | emqx |
| pod           | emqx |

In general, services, deployment, and pods should have different names to distiguish them, but EMQ X’s automatic clustering feature needs have them the same name.

Deployment && Pod

Refer to the configuration of the Kubernetes auto-cluster in documentation of EMQ X, you need to modify configuration of etc/emqx.conf as following.

cluster.discovery = k8s##--------------------------------------------------------------------
## Cluster with k8scluster.k8s.apiserver = http://10.110.111.204:8080cluster.k8s.service_name = emqx## Address Type: ip | dns
cluster.k8s.address_type = ip## The Erlang application name
cluster.k8s.app_name = emqx## Kubernates Namespace
cluster.k8s.namespace = default

Refer to documentation in the EMQ X image on Github, you can edit etc/emqx.conf to change the environment variables. According to the configuration required as previous, we can specify the following environment variables.

- name: EMQX_CLUSTER__DISCOVERY
    value: k8s
- name: EMQX_NAME
    value: emqx
- name: EMQX_CLUSTER__K8S__APISERVER
    value: http://172.31.19.161:8080
- name: EMQX_CLUSTER__K8S__NAMESPACE
    value: default
- name: EMQX_CLUSTER__K8S__SERVICE_NAME
    value: emqx
- name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
    value: ip
- name: EMQX_CLUSTER__K8S__APP_NAME
    value: emqx

Create deployment and pod: pod can only use the docker hub image, because kubernetes can’t directly use Dockerfile to compile the image and you’ll have to pull from the docker’s mirror repository. Please specify 2 images for deploy pod, and port, environment variables as well.

kind: Deployment
metadata:
  name: emqx
  labels:
        app: emqx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:latest
        ports:
        - name: emqx-dashboard
          containerPort: 18083
        env:
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: http://172.31.19.161:8080
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
          value: ip
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        tty: true

Services

Create a service and use the NodePort method to expose the port of emqx-dashboard for external access, so that user can access the dashboard by host address.

apiVersion: v1
kind: Service
metadata:
  name: emqx
spec:
  ports:
  - port: 32333
    nodePort: 32333
    targetPort:  emqx-dashboard
    protocol: TCP
  selector:
    app: emqx
  type: NodePort

Deploy Service

Check out the emqx.yml file.

cat emqx.ymlapiVersion: v1
kind: Service
metadata:
  name: emqx
spec:
  ports:
  - port: 32333
    nodePort: 32333
    targetPort:  emqx-dashboard
    protocol: TCP
  selector:
    app: emqx
  type: NodePort---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: emqx
  labels:
        app: emqx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: emqx
    spec:
      containers:
      - name: emqx
        image: emqx/emqx:latest
        ports:
        - name: emqx-dashboard
          containerPort: 18083
        env:
        - name: EMQX_CLUSTER__DISCOVERY
          value: k8s
        - name: EMQX_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__APISERVER
          value: http://172.31.19.161:8080
        - name: EMQX_CLUSTER__K8S__NAMESPACE
          value: default
        - name: EMQX_CLUSTER__K8S__SERVICE_NAME
          value: emqx
        - name: EMQX_CLUSTER__K8S__ADDRESS_TYPE
          value: ip
        - name: EMQX_CLUSTER__K8S__APP_NAME
          value: emqx
        tty: true

Deploy EMQ X.

$ kubectl create -f emqx.yml 
service/emqx created
deployment.extensions/emqx created

Take a look at status of deployment, you can see that all resources have been successfully deployed.

$ kubectl get all -o wide
NAME                                  READY   STATUS        RESTARTS   AGE     IP            NODE         NOMINATED NODE
pod/emqx-7685fc45cd-qmxlq             1/1     Running       0          3m22s   10.244.2.14   kube-node3   <none>
pod/emqx-7685fc45cd-zq2cj             1/1     Running       0          3m22s   10.244.1.18   kube-node2   <none>NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE     SELECTOR
service/emqx         NodePort    10.98.146.60   <none>        32333:32333/TCP   2m49s   app=emqx
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP           6d5h    <none>NAME                   DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES             SELECTOR
deployment.apps/emqx   2         2         2            2           2m49s   emqx         emqx/emqx:latest   app=emqxNAME                              DESIRED   CURRENT   READY   AGE     CONTAINERS   IMAGES             SELECTOR
replicaset.apps/emqx-7685fc45cd   2         2         2       2m49s   emqx         emqx/emqx:latest   app=emqx,pod-template-hash=7685fc45cd

Using the command to view the cluster status, you can see that EMQ X has been automatically clustered.

$ kubectl exec -it emqx-7685fc45cd-qmxlq /opt/emqx/bin/emqx_ctl cluster status
Cluster status: [{running_nodes,['emqx@10.244.1.18','emqx@10.244.2.14']}]

Even if you delete a pod, kubernetes will automatically create a new pod to auto-cluster.

$ kubectl delete pod/emqx-7685fc45cd-zq2cj
pod "emqx-7685fc45cd-zq2cj" deleted$ kubectl get pods 
NAME                    READY   STATUS    RESTARTS   AGE
emqx-7685fc45cd-nt54v   1/1     Running   0          56s
emqx-7685fc45cd-qmxlq   1/1     Running   0          6m25$ kubectl exec -it emqx-7685fc45cd-qmxlq /opt/emqx/bin/emqx_ctl cluster status
Cluster status: [{running_nodes,['emqx@10.244.1.19','emqx@10.244.2.14']},
                 {stopped_nodes,['emqx@10.244.1.18']}]

Open a browser and enter `http://nodeIP: 32333`, EMQ X dashboard page can be dislayed successfully.

Welcome to our open source project github.com/emqx/emqx. Please visit the documentation for details.

Kubernetes
Emqx
Emqx Cluster

--

--

EMQ Technologies
EMQ Technologies

Written by EMQ Technologies

806 Followers
17 Following

EMQ is an open-source IoT data infrastructure software provider, delivering the world’s leading open-source MQTT message broker and stream processing database.

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams