Integration Executor configuration properties

How to change configuration parameters?
- Docker-Based deployment
- K8S-Based deployment
Configuration parameters

This guide will help you become familiar with TBMQ Integration Executor (IE) configuration files and parameters. We strongly recommend configuring TBMQ IE using environment variables. This way, you won’t need to merge the configuration files when a new platform release arrives.

The list of available configuration parameters and corresponding environment variables can be found here.

How to change configuration parameters?

Docker-Based deployment

If TBMQ IE is installed in a Docker Compose environment, you can edit the scripts and add environment variables for the corresponding containers. For more details, refer to the Docker documentation.

K8S-Based deployment

If TBMQ IE is installed in a K8S environment, you can edit the scripts and add environment variables for the corresponding deployments/stateful sets. For more details, refer to the K8S documentation.

Configuration parameters

The configuration file is written in YAML format. All configuration parameters have corresponding environment variable names and default values. To change a configuration parameter, simply modify its default value. For example:

server:
  address: "${HTTP_BIND_ADDRESS:0.0.0.0}"

In this case, ‘HTTP_BIND_ADDRESS’ is the environment variable name and ‘0.0.0.0’ is the default value.

You can use the simple example below to add a new environment variable ‘HTTP_BIND_PORT’ with value ‘8084’.

...
export HTTP_BIND_PORT=8084

The parameters are grouped by system components. The list contains the name (address in tbmq-integration-executor.yml file), environment variable, default value, and description.

HTTP server parameters

Parameter	Environment Variable	Default Value	Description
server.address	HTTP_BIND_ADDRESS	0.0.0.0	HTTP Server bind address
server.port	HTTP_BIND_PORT	8082	HTTP Server bind port

Kafka parameters

Parameter	Environment Variable	Default Value	Description
queue.integration-downlink.poll-interval	TB_IE_DOWNLINK_POLL_INTERVAL	1000	Interval in milliseconds to poll messages from 'tbmq.ie.downlink' topics
queue.integration-msg.poll-interval	TB_IE_MSG_POLL_INTERVAL	1000	Interval in milliseconds to poll messages from 'tbmq.msg.ie' topics
queue.integration-msg.pack-processing-timeout	TB_IE_MSG_PACK_PROCESSING_TIMEOUT	30000	Timeout in milliseconds for processing the pack of messages
queue.integration-msg.ack-strategy.type	TB_IE_MSG_ACK_STRATEGY_TYPE	SKIP_ALL	Processing strategy for 'tbmq.msg.ie' topics. Can be: SKIP_ALL, RETRY_ALL
queue.integration-msg.ack-strategy.retries	TB_IE_MSG_ACK_STRATEGY_RETRIES	5	Number of retries, 0 is unlimited. Use for RETRY_ALL processing strategy
queue.integration-msg.ack-strategy.pause-between-retries	TB_IE_MSG_ACK_STRATEGY_PAUSE_BETWEEN_RETRIES	1	Time in seconds to wait in consumer thread before retries
queue.kafka.bootstrap.servers	TB_KAFKA_SERVERS	localhost:9092	List of kafka bootstrap servers used to establish connection
queue.kafka.enable-topic-deletion	TB_KAFKA_ENABLE_TOPIC_DELETION	true	Enable/disable deletion of topics for Application MQTT Clients
queue.kafka.default.consumer.partition-assignment-strategy	TB_KAFKA_DEFAULT_CONSUMER_PARTITION_ASSIGNMENT_STRATEGY	org.apache.kafka.clients.consumer.StickyAssignor	A list of class names or class types, ordered by preference, of supported partition assignment strategies that the client will use to distribute partition ownership amongst consumer instances when group management is used
queue.kafka.default.consumer.session-timeout-ms	TB_KAFKA_DEFAULT_CONSUMER_SESSION_TIMEOUT_MS	300000	The timeout in milliseconds used to detect client failures when using Kafka's group management facility
queue.kafka.default.consumer.max-poll-interval-ms	TB_KAFKA_DEFAULT_CONSUMER_MAX_POLL_INTERVAL_MS	300000	The maximum delay in milliseconds between invocations of poll() when using consumer group management
queue.kafka.default.consumer.max-poll-records	TB_KAFKA_DEFAULT_CONSUMER_MAX_POLL_RECORDS	2000	The maximum number of records returned in a single call to poll()
queue.kafka.default.consumer.max-partition-fetch-bytes	TB_KAFKA_DEFAULT_CONSUMER_MAX_PARTITION_FETCH_BYTES	16777216	The maximum amount of data in bytes per-partition the server will return
queue.kafka.default.consumer.fetch-max-bytes	TB_KAFKA_DEFAULT_CONSUMER_FETCH_MAX_BYTES	134217728	The maximum amount of data in bytes the server should return for a fetch request
queue.kafka.default.producer.acks	TB_KAFKA_DEFAULT_PRODUCER_ACKS	1	The number of acknowledgments the producer requires the leader to have received before considering a request complete
queue.kafka.default.producer.retries	TB_KAFKA_DEFAULT_PRODUCER_RETRIES	1	Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error
queue.kafka.default.producer.batch-size	TB_KAFKA_DEFAULT_PRODUCER_BATCH_SIZE	16384	The producer will attempt to batch records together into fewer requests whenever multiple records are being sent to the same partition. Size in bytes
queue.kafka.default.producer.linger-ms	TB_KAFKA_DEFAULT_PRODUCER_LINGER_MS	5	The producer groups together any records that arrive in between request transmissions into a single batched request, set in milliseconds
queue.kafka.default.producer.buffer-memory	TB_KAFKA_DEFAULT_PRODUCER_BUFFER_MEMORY	33554432	The total bytes of memory the producer can use to buffer records waiting to be sent to the server
queue.kafka.default.producer.compression-type	TB_KAFKA_DEFAULT_COMPRESSION_TYPE	none	The compression type for all data generated by the producer. Valid values are `none`, `gzip`, `snappy`, `lz4`, or `zstd`
queue.kafka.admin.config	TB_KAFKA_ADMIN_CONFIG	retries:1	List of configs separated by semicolon used for admin kafka client creation
queue.kafka.consumer-stats.enabled	TB_KAFKA_CONSUMER_STATS_ENABLED	true	Prints lag if enabled between consumer group offset and last messages offset in Kafka topics
queue.kafka.consumer-stats.print-interval-ms	TB_KAFKA_CONSUMER_STATS_PRINT_INTERVAL_MS	60000	Statistics printing interval in milliseconds for Kafka's consumer-groups stats
queue.kafka.consumer-stats.kafka-response-timeout-ms	TB_KAFKA_CONSUMER_STATS_RESPONSE_TIMEOUT_MS	1000	Time to wait in milliseconds for the stats-loading requests to Kafka to finish
queue.kafka.consumer-stats.consumer-config	TB_KAFKA_CONSUMER_STATS_CONSUMER_CONFIG		List of configs separated by semicolon used for kafka stats consumer
queue.kafka.integration-downlink.topic-prefix	TB_KAFKA_IE_DOWNLINK_TOPIC_PREFIX	tbmq.ie.downlink	Prefix for topics for sending integration configurations and validation requests from tbmq to integration executors
queue.kafka.integration-downlink.http.topic-properties	TB_KAFKA_IE_DOWNLINK_HTTP_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;partitions:6;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.ie.downlink.http` topic
queue.kafka.integration-downlink.http.additional-consumer-config	TB_KAFKA_IE_DOWNLINK_HTTP_ADDITIONAL_CONSUMER_CONFIG		Additional Kafka consumer configs separated by semicolon for `tbmq.ie.downlink.http` topic
queue.kafka.integration-downlink.http.additional-producer-config	TB_KAFKA_IE_DOWNLINK_HTTP_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.ie.downlink.http` topic
queue.kafka.integration-downlink.kafka.topic-properties	TB_KAFKA_IE_DOWNLINK_KAFKA_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;partitions:6;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.ie.downlink.kafka` topic
queue.kafka.integration-downlink.kafka.additional-consumer-config	TB_KAFKA_IE_DOWNLINK_KAFKA_ADDITIONAL_CONSUMER_CONFIG		Additional Kafka consumer configs separated by semicolon for `tbmq.ie.downlink.kafka` topic
queue.kafka.integration-downlink.kafka.additional-producer-config	TB_KAFKA_IE_DOWNLINK_KAFKA_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.ie.downlink.kafka` topic
queue.kafka.integration-downlink.mqtt.topic-properties	TB_KAFKA_IE_DOWNLINK_MQTT_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;partitions:6;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.ie.downlink.mqtt` topic
queue.kafka.integration-downlink.mqtt.additional-consumer-config	TB_KAFKA_IE_DOWNLINK_MQTT_ADDITIONAL_CONSUMER_CONFIG		Additional Kafka consumer configs separated by semicolon for `tbmq.ie.downlink.mqtt` topic
queue.kafka.integration-downlink.mqtt.additional-producer-config	TB_KAFKA_IE_DOWNLINK_MQTT_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.ie.downlink.mqtt` topic
queue.kafka.integration-uplink.topic	TB_KAFKA_IE_UPLINK_TOPIC	tbmq.ie.uplink	Topic for sending messages/events from integration executors to tbmq
queue.kafka.integration-uplink.topic-properties	TB_KAFKA_IE_UPLINK_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;partitions:6;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.ie.uplink` topic
queue.kafka.integration-uplink.additional-consumer-config	TB_KAFKA_IE_UPLINK_ADDITIONAL_CONSUMER_CONFIG		Additional Kafka consumer configs separated by semicolon for `tbmq.ie.uplink` topic
queue.kafka.integration-uplink.additional-producer-config	TB_KAFKA_IE_UPLINK_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.ie.uplink` topic
queue.kafka.integration-uplink-notifications.topic-prefix	TB_KAFKA_IE_UPLINK_NOTIF_TOPIC_PREFIX	tbmq.ie.uplink.notifications	Prefix for topics for sending notifications or replies from integration executors to specific tbmq node
queue.kafka.integration-uplink-notifications.topic-properties	TB_KAFKA_IE_UPLINK_NOTIF_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;partitions:1;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.ie.uplink.notifications` topics
queue.kafka.integration-uplink-notifications.additional-consumer-config	TB_KAFKA_IE_UPLINK_NOTIF_ADDITIONAL_CONSUMER_CONFIG		Additional Kafka consumer configs separated by semicolon for `tbmq.ie.uplink.notifications` topic
queue.kafka.integration-uplink-notifications.additional-producer-config	TB_KAFKA_IE_UPLINK_NOTIF_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.ie.uplink.notifications` topic
queue.kafka.integration-msg.topic-properties	TB_KAFKA_IE_MSG_TOPIC_PROPERTIES	retention.ms:604800000;segment.bytes:26214400;retention.bytes:1048576000;replication.factor:1	Kafka topic properties separated by semicolon for `tbmq.msg.ie` topics
queue.kafka.integration-msg.additional-consumer-config	TB_KAFKA_IE_MSG_ADDITIONAL_CONSUMER_CONFIG	max.poll.records:50	Additional Kafka consumer configs separated by semicolon for `tbmq.msg.ie` topics
queue.kafka.integration-msg.additional-producer-config	TB_KAFKA_IE_MSG_ADDITIONAL_PRODUCER_CONFIG		Additional Kafka producer configs separated by semicolon for `tbmq.msg.ie` topics
queue.kafka.kafka-prefix	TB_KAFKA_PREFIX		The common prefix for all Kafka topics, producers, consumer groups, and consumers. Defaults to empty string meaning no prefix is added

Service parameters

Parameter	Environment Variable	Default Value	Description
service.type	TB_SERVICE_TYPE	tbmq-integration-executor	Microservice type. Allowed value: tbmq-integration-executor
service.id	TB_SERVICE_ID		Unique id for this service (autogenerated if empty)
service.integrations.supported	TB_SERVICE_INTEGRATIONS_SUPPORTED	ALL	Allow to enable integration on microservice integration executor. Allowed values: HTTP, KAFKA, MQTT. By default, ALL
service.integrations.excluded	TB_SERVICE_INTEGRATIONS_EXCLUDED	NONE	List of integrations to exclude from processing on service/microservice integration executor. Allowed values: HTTP, KAFKA, MQTT. By default, NONE

Integration common parameters

Parameter	Environment Variable	Default Value	Description
integrations.statistics.enabled	INTEGRATIONS_STATISTICS_ENABLED	true	Enable/disable integrations statistics
integrations.statistics.persist-frequency	INTEGRATIONS_STATISTICS_PERSIST_FREQUENCY	3600000	Integration statistic persistence frequency in milliseconds
integrations.init.connection-timeout-sec	INTEGRATIONS_INIT_CONNECTION_TIMEOUT_SEC	15	Maximum connection timeout allowed for integrations in seconds. Any greater user defined timeout will be reduced down to this limit
integrations.init.connection-check-api-request-timeout-sec	INTEGRATIONS_INIT_CONNECTION_CHECK_API_REQUEST_TIMEOUT_SEC	20	Connection check timeout for API request in seconds
integrations.reinit.enabled	INTEGRATIONS_REINIT_ENABLED	true	Enable/disable integrations hot reinitialization. This process is done for integrations with state 'FAILED'
integrations.reinit.frequency	INTEGRATIONS_REINIT_FREQUENCY_MS	300000	Checking interval in milliseconds for reinit integrations. Defaults to 5 minutes
integrations.destroy.graceful-timeout-ms	INTEGRATIONS_DESTROY_TIMEOUT_MS	1000	The duration (in milliseconds) to wait during each iteration of the graceful shutdown process for integrations to terminate properly. Default value is set to 1 seconds
integrations.destroy.count	INTEGRATIONS_DESTROY_COUNT	10	The number of iterations to attempt a graceful shutdown before forcefully stopping the process
integrations.destroy.forced-shutdown-timeout-ms	INTEGRATIONS_DESTROY_FORCED_SHUTDOWN_TIMEOUT_MS	15000	The maximum duration (in milliseconds) to wait before forcefully stopping the application if the graceful shutdown process has not started or exceeds the allowed time
integrations.allow-local-network-hosts	INTEGRATIONS_ALLOW_LOCAL_NETWORK_HOSTS	true	Enable/disable integrations local network hosts
integrations.uplink.callback-threads-count	INTEGRATIONS_UPLINK_THREADS	4	Number of threads in the pool to process callbacks of uplink events to tbmq nodes
integrations.manage.lifecycle-threads-count	INTEGRATIONS_MANAGE_LIFECYCLE_THREADS	4	Number of threads in the pool to process lifecycle events (CREATE/UPDATE/DELETE) of integrations
integrations.manage.command-threads-count	INTEGRATIONS_MANAGE_COMMAND_THREADS	4	Number of threads in the pool to process integration validation requests
integrations.external.threads-count	INTEGRATIONS_EXTERNAL_THREADS	10	Number of threads in the pool dedicated to handling external operations, such as producing messages to Kafka topics
integrations.netty.threads-count	INTEGRATIONS_NETTY_SHARED_GROUP_THREADS	0	Netty shared worker group threads count. Defaults to 0 meaning the threads count is number of availableProcessors * 2. Used to send messages to external MQTT brokers using MQTT bridge (integration)

Management parameters

Parameter	Environment Variable	Default Value	Description
management.endpoints.web.exposure.include	METRICS_ENDPOINTS_EXPOSE	prometheus	Expose metrics endpoint (use value 'info' to disable prometheus metrics)

Statistics parameters

Parameter	Environment Variable	Default Value	Description
stats.ie.enabled	STATS_IE_ENABLED	true	Enable/disable stats printing to the logs
stats.ie.print-interval-ms	STATS_IE_PRINT_INTERVAL_MS	60000	Period in milliseconds to print stats. Default value corresponds to 1 minute
stats.timer.percentiles	STATS_TIMER_PERCENTILES	0.5	Metrics percentiles returned by actuator for timer metrics. List of comma-separated (,) double values

Event configuration parameters

Parameter	Environment Variable	Default Value	Description
event.error.rate-limits.enabled	EVENT_ERROR_RATE_LIMITS_ENABLED	true	If true rate limits will be active
event.error.rate-limits.integration	EVENT_ERROR_RATE_LIMITS_INTEGRATION	5000:3600,100:60	No more than 5000 messages per hour or 100 messages per minute for all integrations
event.error.rate-limits.ttl-minutes	EVENT_ERROR_RATE_LIMITS_TTL	60	Time (in minutes) to prevent duplicate persistence of rate limit events once the error event rate limit is reached