Cluster setup using Microsoft Azure infrastructure

Prerequisites
- Install and configure tools
Step 1. Open TBMQ K8S scripts repository
Step 2. Define environment variables
Step 3. Configure and create AKS cluster
Step 4. Update the context of kubectl
Step 5. Provision PostgreSQL DB
Step 6. Azure Cache for Redis (Optional)
Step 7. Installation
Step 8. Provision Kafka
Step 9. Starting
Step 10. Configure Load Balancers
- 10.1 Configure HTTP(S) Load Balancer
  - HTTP Load Balancer
  - HTTPS Load Balancer
- 10.2 Configure MQTT Load Balancer
  - MQTT over SSL
Step 11. Validate the setup
- Validate MQTT access
- Troubleshooting
Upgrading
Cluster deletion
Next steps

This guide will help you to set up TBMQ in Azure AKS.

Prerequisites

Install and configure tools

To deploy TBMQ on the AKS cluster you will need to install kubectl, helm, and az tools.

After installation is done you need to log in to the cli using the next command.

az login

Step 1. Open TBMQ K8S scripts repository

git clone -b release-1.4.0 https://github.com/thingsboard/tbmq.git
cd tbmq/k8s/azure

Step 2. Define environment variables

Define environment variables that you will use in various commands later in this guide.

We assume you are using Linux. Execute the following command:

export AKS_RESOURCE_GROUP=TBMQResources
export AKS_LOCATION=eastus
export AKS_GATEWAY=tbmq-gateway
export TB_CLUSTER_NAME=tbmq-cluster
export TB_DATABASE_NAME=tbmq-db
export TB_REDIS_NAME=tbmq-redis
echo "You variables ready to create resource group $AKS_RESOURCE_GROUP in location $AKS_LOCATION 
and cluster in it $TB_CLUSTER_NAME with database $TB_DATABASE_NAME"

where:

TBMQResources - a logical group in which Azure resources are deployed and managed. We will refer to it later in this guide using AKS_RESOURCE_GROUP;
eastus - is the location where you want to create resource group. We will refer to it later in this guide using AKS_LOCATION. You can see all locations list by executing az account list-locations;
tbmq-gateway - the name of Azure application gateway;
tbmq-cluster - cluster name. We will refer to it later in this guide using TB_CLUSTER_NAME;
tbmq-db is the name of your database server. You may input a different name. We will refer to it later in this guide using TB_DATABASE_NAME.

Step 3. Configure and create AKS cluster

Before creating the AKS cluster we need to create Azure Resource Group. We will use Azure CLI for this:

az group create --name $AKS_RESOURCE_GROUP --location $AKS_LOCATION

To see more info about az group please follow the next link.

After the Resource group is created we can create AKS cluster by using the next command:

az aks create --resource-group $AKS_RESOURCE_GROUP \
    --name $TB_CLUSTER_NAME \
    --generate-ssh-keys \
    --enable-addons ingress-appgw \
    --appgw-name $AKS_GATEWAY \
    --appgw-subnet-cidr "10.2.0.0/16" \
    --node-vm-size Standard_DS3_v2 \
    --node-count 3

az aks create has two required parameters - name and resource-group (we use variables that we have set earlier), and a lot of not required parameters (defaults values will be used if not set). A few of them are:

node-count - Number of nodes in the Kubernetes node pool. After creating a cluster, you can change the size of its node pool with az aks scale (default value is 3);
enable-addons - Enable the Kubernetes addons in a comma-separated list (use az aks addon list to get available addons list);
node-osdisk-size - OS disk type to be used for machines in a given agent pool: Ephemeral or Managed. Defaults to ‘Ephemeral’ when possible in conjunction with VM size and OS disk size. May not be changed for this pool after creation;
node-vm-size (or -s) - Size of Virtual Machines to create as Kubernetes nodes (default value is Standard_DS2_v2);
generate-ssh-keys - Generate SSH public and private key files if missing. The keys will be stored in the ~/.ssh directory.

From the command above we add AKS addon for ApplicationGateway. We will use this gateway as Path-Based Load Balancer for the TBMQ.

Full list af az aks create options can be found here.

Alternatively, you may use this guide for custom cluster setup.

Step 4. Update the context of kubectl

When the cluster is created we can connect kubectl to it using the next command:

az aks get-credentials --resource-group $AKS_RESOURCE_GROUP --name $TB_CLUSTER_NAME

For validation, you can execute the following command:

kubectl get nodes

You should see cluster`s nodes list.

Step 5. Provision PostgreSQL DB

You’ll need to set up PostgreSQL on Azure. You may follow this guide, but take into account the following requirements:

Keep your postgresql password in a safe place. We will refer to it later in this guide using YOUR_AZURE_POSTGRES_PASSWORD;
Make sure your Azure Database for PostgreSQL version is 15.x;
Make sure your Azure Database for PostgreSQL instance is accessible from the TBMQ cluster;
Make sure you use “thingsboard_mqtt_broker” as the initial database name.

Note: Use “High availability” enabled. It enables a lot of useful settings by default.

Another way by which you can create Azure Database for PostgreSQL is using az tool (don’t forget to replace ‘POSTGRESS_USER’ and ‘POSTGRESS_PASS’ with your username and password):

az postgres flexible-server create --location $AKS_LOCATION --resource-group $AKS_RESOURCE_GROUP \
  --name $TB_DATABASE_NAME --admin-user POSTGRESS_USER --admin-password POSTGRESS_PASS \
  --public-access 0.0.0.0 --storage-size 32 \
  --version 15 -d thingsboard_mqtt_broker

az postgres flexible-server create has a lot of parameters, a few of them are:

location - Location. Values from: az account list-locations;
resource-group (or -g) - Name of resource group;
name - Name of the server. The name can contain only lowercase letters, numbers, and the hyphen (-) character. Minimum 3 characters and maximum 63 characters;
admin-user - Administrator username for the server. Once set, it cannot be changed;
admin-password - The password of the administrator. Minimum 8 characters and maximum 128 characters. Password must contain characters from three of the following categories: English uppercase letters, English lowercase letters, numbers, and non-alphanumeric characters;
public-access - Determines the public access. Enter single or range of IP addresses to be included in the allowed list of IPs. IP address ranges must be dash-separated and not contain any spaces. Specifying 0.0.0.0 allows public access from any resources deployed within Azure to access your server. Setting it to “None” sets the server in public access mode but does not create a firewall rule;
storage-size - The storage capacity of the server. Minimum is 32 GiB and maximum is 16 TiB;
version - Server major version;
high-availability - enable or disable high availability feature. High availability can only be set during flexible server creation (accepted values: Disabled, Enabled. Default value: Disabled);
database-name (or -d) - The name of the database to be created when provisioning the database server.

You can see full parameters list here.

Example of response:

{
  "connectionString": "postgresql://postgres:postgres@$tbmq-db.postgres.database.azure.com/postgres?sslmode=require",
  "databaseName": "thingsboard_mqtt_broker",
  "firewallName": "AllowAllAzureServicesAndResourcesWithinAzureIps_2021-11-17_15-45-6",
  "host": "tbmq-db.postgres.database.azure.com",
  "id": "/subscriptions/daff3288-1d5d-47c7-abf0-bfb7b738a18c/resourceGroups/myResourceGroup/providers/Microsoft.DBforPostgreSQL/flexibleServers/thingsboard_mqtt_broker",
  "location": "East US",
  "password": "postgres",
  "resourceGroup": "TBMQResources",
  "skuname": "Standard_D2s_v3",
  "username": "postgres",
  "version": "15"
}

Note the value of host from the command output (tbmq-db.postgres.database.azure.com in our case). Also note username and password (postgres) from the command.

Edit the database settings file and replace YOUR_AZURE_POSTGRES_ENDPOINT_URL with the host value, YOUR_AZURE_POSTGRES_USER and YOUR_AZURE_POSTGRES_PASSWORD with the correct values:

nano tb-broker-db-configmap.yml

Step 6. Azure Cache for Redis (Optional)

Optionally, you can set up Azure Cache for Redis. TBMQ uses cache to improve performance and avoid frequent DB reads.

We recommend enabling this in case you have several thousand MQTT clients (devices) connected to TBMQ. It is useful when clients connect to TBMQ with the authentication enabled. For every connection, the request is made to find MQTT client credentials that can authenticate the client. Thus, there could be an excessive amount of requests to be processed for a large number of connecting clients at once.

In order to set up the Redis, follow this guide.

Another way to do this is by using az tool:

az redis create --name $TB_REDIS_NAME --location $AKS_LOCATION --resource-group $AKS_RESOURCE_GROUP --sku basic --vm-size C0 --enable-non-ssl-port

az redis create has a lot of options and a few of them are required:

name (or -n) - name of the Redis cache;
resource-group (or -g) - name of resource group;
sku - type of Redis cache (accepted values: Basic, Premium, Standard);
vm-size - size of Redis cache to deploy. Basic and Standard Cache sizes start with C. Premium Cache sizes start with P (accepted values: c0, c1, c2, c3, c4, c5, c6, p1, p2, p3, p4, p5);
location (or -l) - location. Values from: az account list-locations.

To see the full list of parameters go to the following page.

Example of response:

{
  "accessKeys": null,
  "enableNonSslPort": true,
  "hostName": "tbmq-redis.redis.cache.windows.net",
  "id": "/subscriptions/daff3288-1d5d-47c7-abf0-bfb7b738a18c/resourceGroups/myResourceGroup/providers/Microsoft.Cache/Redis/tbmq-redis",
  "instances": [
    {
      "isMaster": false,
      "isPrimary": false,
      "nonSslPort": 13000,
      "shardId": null,
      "sslPort": 15000,
      "zone": null
    }
  ],
  "linkedServers": [],
  "location": "East US",
  "minimumTlsVersion": null,
  "name": "tbmq-redis",
  "port": 6379,
  "privateEndpointConnections": null,
  "provisioningState": "Creating",
  "publicNetworkAccess": "Enabled",
  "redisConfiguration": {
    "maxclients": "256",
    "maxfragmentationmemory-reserved": "12",
    "maxmemory-delta": "2",
    "maxmemory-reserved": "2"
  },
  "redisVersion": "6.0.20",
  "replicasPerMaster": null,
  "replicasPerPrimary": null,
  "resourceGroup": "myResourceGroup",
  "shardCount": null,
  "sku": {
    "capacity": 0,
    "family": "C",
    "name": "Basic"
  },
  "sslPort": 6380,
  "staticIp": null,
  "subnetId": null,
  "tags": {},
  "tenantSettings": {},
  "type": "Microsoft.Cache/Redis",
  "zones": null
}

We need to take hostName parameter and replace YOUR_REDIS_ENDPOINT_URL_WITHOUT_PORT in the file tb-broker-cache-configmap.yml.

After this we need to get redis keys for connection, for this we need to execute:

az redis list-keys --name $TB_REDIS_NAME --resource-group $AKS_RESOURCE_GROUP

Take “primary” and paste into tb-broker-cache-configmap.yml file replacing YOUR_REDIS_PASSWORD.

For more information, see the following script.

Step 7. Installation

Execute the following command to run the initial setup of the database. This command will launch short-living TBMQ pod to provision necessary DB tables, indexes, etc.

./k8s-install-tbmq.sh

After this command finish you should see the next line in the console:

INFO  o.t.m.b.i.ThingsboardMqttBrokerInstallService - Installation finished successfully!

Step 8. Provision Kafka

We recommend deploying Bitnami Kafka from Helm. For that, review the kafka folder.

ls kafka/

You can find there default-values-kafka.yml file - default values downloaded from Bitnami artifactHub. And values-kafka.yml file with modified values. We recommend keeping the first file untouched and making changes to the second one only. This way the upgrade process to the next version will go more smoothly as it will be possible to see diff.

To add the Bitnami helm repo:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

To install Bitnami Kafka execute the following command:

helm install kafka -f kafka/values-kafka.yml bitnami/kafka --version 25.3.3

Wait up to several minutes until Kafka and Zookeeper pods are up and running.

Step 9. Starting

Execute the following command to deploy the broker:

./k8s-deploy-tbmq.sh

After a few minutes, you may execute the next command to check the state of all pods.

kubectl get pods

If everything went fine, you should be able to see tb-broker-0 and tb-broker-1 pods. Every pod should be in the READY state.

Step 10. Configure Load Balancers

10.1 Configure HTTP(S) Load Balancer

Configure HTTP(S) Load Balancer to access web interface of your TBMQ instance. Basically you have 2 possible options of configuration:

http - Load Balancer without HTTPS support. Recommended for development. The only advantage is simple configuration and minimum costs. May be good option for development server but definitely not suitable for production.
https - Load Balancer with HTTPS support. Recommended for production. Acts as an SSL termination point. You may easily configure it to issue and maintain a valid SSL certificate. Automatically redirects all non-secure (HTTP) traffic to secure (HTTPS) port.

See links/instructions below on how to configure each of the suggested options.

HTTP Load Balancer

Execute the following command to deploy plain http load balancer:

kubectl apply -f receipts/http-load-balancer.yml

The process of load balancer provisioning may take some time. You may periodically check the status of the load balancer using the following command:

kubectl get ingress

Once provisioned, you should see similar output:

NAME                          CLASS    HOSTS   ADDRESS         PORTS   AGE
tb-broker-http-loadbalancer   <none>   *       34.111.24.134   80      3d1h

HTTPS Load Balancer

For using ssl certificates we can add our certificate directly in Azure ApplicationGateway using command:

az network application-gateway ssl-cert create \
   --resource-group $(az aks show --name $TB_CLUSTER_NAME --resource-group $AKS_RESOURCE_GROUP --query nodeResourceGroup | tr -d '"') \
   --gateway-name $AKS_GATEWAY\
   --name TBMQHTTPSCert \
   --cert-file YOUR_CERT \
   --cert-password YOUR_CERT_PASS

Execute the following command to deploy plain https load balancer:

kubectl apply -f receipts/https-load-balancer.yml

10.2 Configure MQTT Load Balancer

Configure MQTT load balancer to be able to use MQTT protocol to connect devices.

Create TCP load balancer using following command:

kubectl apply -f receipts/mqtt-load-balancer.yml

The load balancer will forward all TCP traffic for ports 1883 and 8883.

MQTT over SSL

Follow this guide to create a .pem file with the SSL certificate. Store the file as server.pem in the working directory.

You’ll need to create a config-map with your PEM file, you can do it by calling command:

kubectl create configmap tbmq-mqtts-config \
 --from-file=server.pem=YOUR_PEM_FILENAME \
 --from-file=mqttserver_key.pem=YOUR_PEM_KEY_FILENAME \
 -o yaml --dry-run=client | kubectl apply -f -

where YOUR_PEM_FILENAME is the name of your server certificate file.
where YOUR_PEM_KEY_FILENAME is the name of your server certificate private key file.

Then, uncomment all sections in the ‘tb-broker.yml’ file that are marked with “Uncomment the following lines to enable two-way MQTTS”.

Execute command to apply changes:

kubectl apply -f tb-broker.yml

Step 11. Validate the setup

Now you can open TBMQ web interface in your browser using DNS name of the load balancer.

You can get DNS name of the load-balancers using the next command:

kubectl get ingress

You should see the similar picture:

NAME                          CLASS    HOSTS   ADDRESS         PORTS   AGE
tb-broker-http-loadbalancer   <none>   *       34.111.24.134   80      3d1h

Use ADDRESS field of the tb-broker-http-loadbalancer to connect to the cluster.

You should see TBMQ login page. Use the following default credentials for System Administrator:

Username:

sysadmin@thingsboard.org

Password:

sysadmin

On the first user log-in you will be asked to change the default password to the preferred one and then re-login using the new credentials.

Validate MQTT access

To connect to the cluster via MQTT you will need to get corresponding service IP. You can do this with the command:

kubectl get services

You should see the similar picture:

NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP              PORT(S)                         AGE
tb-broker-mqtt-loadbalancer   LoadBalancer   10.100.119.170   *******                  1883:30308/TCP,8883:31609/TCP   6m58s

Use EXTERNAL-IP field of the load-balancer to connect to the cluster via MQTT protocol.

Troubleshooting

In case of any issues you can examine service logs for errors. For example to see TBMQ logs execute the following command:

kubectl logs -f tb-broker-0

Use the next command to see the state of all statefulsets.

kubectl get statefulsets

See kubectl Cheat Sheet command reference for more details.

Upgrading

Please note:
For the TBMQ 1.3.0 version, the installation scripts were updated to contain a new 8084 port for MQTT over WebSockets. This is needed for the correct work with the WebSocket client page.

Please pull the latest configuration files or modify your existing ones to include a new port entry. To find more details please visit the following link.

Once the required changes are made, you should be able to connect the MQTT client on the WebSocket client page. Otherwise, please contact us, so we can answer any questions and provide our help if needed.

In case you would like to upgrade, please pull the recent changes from the latest release branch:

git pull origin release-1.4.0

Note: Make sure custom changes of yours if available are not lost during the merge process.

If you encounter conflicts during the merge process that are not related to your changes, we recommend accepting all the new changes from the remote branch.

You could revert the merge process by executing the following:

git merge --abort

And repeat the merge by accepting theirs changes.

git pull origin release-1.4.0 -X theirs

There are several useful options for the default merge strategy:

-X ours - this option forces conflicting hunks to be auto-resolved cleanly by favoring our version.
-X theirs - this is the opposite of ours. See more details here.

After that execute the following commands:

./k8s-delete-tbmq.sh
./k8s-upgrade-tbmq.sh --fromVersion=FROM_VERSION
./k8s-deploy-tbmq.sh

Where FROM_VERSION - from which version upgrade should be started. See Upgrade Instructions for valid fromVersion values.

Cluster deletion

Execute the following command to delete TBMQ nodes:

./k8s-delete-tbmq.sh

Execute the following command to delete all TBMQ nodes and configmaps, load balancers, etc.:

./k8s-delete-all.sh

Execute the following command to delete the AKS cluster:

az aks delete --resource-group $AKS_RESOURCE_GROUP --name $TB_CLUSTER_NAME

Next steps

Getting started guide - This guide provide quick overview of TBMQ.
Security guide - Learn how to enable authentication and authorization of MQTT clients.
Configuration guide - Learn about TBMQ configuration files and parameters.
MQTT client type guide - Learn about TBMQ client types.
Integration with ThingsBoard - Learn about how to integrate TBMQ with ThingsBoard.