Module: Kafka

• 1: Quick Start
• 2: Configuration
• 3: Parameters
• 4: Administration
• 5: Playbook
• 6: Monitoring
• 7: Metrics
• 8: FAQ

flowchart LR
    admin["Pigsty admin node"] -->|"kafka.yml / exact cluster"| kafka["Kafka 4.1+ / dynamic KRaft"]
    kafka --> jmx["Each Kafka JVM / JMX :9404"]
    kafka --> exporter["Up to two brokers / kafka_exporter :9308"]
    kafka --> journal["Journald"]
    jmx --> vm["VictoriaMetrics"]
    exporter --> vm
    journal --> vector["Vector"] --> vl["VictoriaLogs"]
    vm --> grafana["Grafana"]
    vl --> grafana
    vm --> alert["Alertmanager"]

    style kafka fill:#70C1B3,stroke:#4f968b,color:#fff
    style vm fill:#E66B7A,stroke:#b84e5c,color:#fff
    style vl fill:#C98367,stroke:#9e634e,color:#fff
    style grafana fill:#F29C64,stroke:#c77845,color:#fff

1 - Quick Start

This tutorial starts from a minimal single-node cluster, walking through topic creation and message read/write; it then deploys a separate, secured three-node cluster with application users, ACLs, quotas, and production topics; finally, it demonstrates core parameter changes, client access, monitoring verification, and pre-launch checks.

Learning Path

Before You Begin

Unless noted otherwise, the following commands are run from the project directory on the Pigsty admin node:

cd ~/pigsty

Before you start, confirm that:

• pigsty.yml is the source of configuration for the current environment — back it up and review its existing contents first;
• each Kafka node’s inventory_hostname can be resolved and routed directly by every Kafka member and client;
• the admin node and the Kafka nodes have synchronized clocks;
• ports 9092, 9093, 9308, and 9404 are free of conflicts;
• /data/kafka maps to a dedicated data disk or directory that holds nothing else;
• every kafka.yml run uses -l to select exactly all members of one Kafka cluster;
• before any real change, you run --check first, review the output, and obtain approval for the change.

The inventory must keep the all.children hierarchy. Merge the groups below into your existing pigsty.yml; do not let the examples overwrite your existing all.vars, infra, etcd, pgsql, or other configuration.

Part 1: Deploy a Single-Node Kafka

1. Define the Cluster

Add the following kf-dev group to all.children. This node omits kafka_role, so it uses the default combined and serves as both broker and controller:

all:
  children:
    # keep your existing infra, etcd, pgsql, and other groups

    kf-dev:
      hosts:
        10.10.10.10: { kafka_seq: 1 }
      vars:
        kafka_cluster: kf-dev
        kafka_data: /data/kafka
        kafka_security: plaintext
        kafka_topics:
          - name: quickstart.events
            partitions: 1
            replication_factor: 1
            config:
              retention.ms: 86400000   # 1 day, tutorial only

This configuration yields:

• a random cluster ID;
• a single dynamic KRaft combined node;
• default RF=1 and minISR=1;
• a single-partition topic named quickstart.events;
• a JMX exporter on :9404 and a protocol exporter on :9308.

plaintext has no transport encryption, authentication, or ACLs, so use it only for local development or a trusted, isolated network.

2. Bring the Node Under Management

If this host has not yet been initialized by NODE, run check mode first:

./node.yml --check -l kf-dev

After reviewing the results and getting approval, bring the node under management:

./node.yml -l kf-dev

You can skip this step for a node that Pigsty already manages and whose package repository and time synchronization are healthy. For full NODE preparation and day-to-day management, see Node Administration.

3. Deploy Kafka

Run a check against the full cluster first:

./kafka.yml --check -l kf-dev

Confirm the target is exactly the full membership of kf-dev, review the data path, packages, ports, and configuration changes, then run:

./kafka.yml -l kf-dev

The role installs Java and kafka-stack, generates a random identity and bootstrap manifest, formats the KRaft storage, starts the service, creates the topics, and registers the monitoring targets.

4. Verify the Service and Quorum

Log in to the Kafka node and check the services:

systemctl is-active kafka kafka_exporter
journalctl -u kafka --since '-10 min' --no-pager

Run the role’s own health check:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties

The returned JSON should contain "healthy": true. Continue by checking the dynamic quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

You should see a valid LeaderId, a CurrentVoters list that includes this node, and quickstart.events with RF=1 and ISR=1.

5. Produce and Consume Messages

Start a console producer:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events

Type a few lines of messages, then press Ctrl-D to finish. Consume them from another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.10:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

At this point, the single-node deployment, topic convergence, and message read/write are complete. For further status checks, see Day-to-Day Administration.

Part 2: Deploy a Three-Node Production Baseline

The three-node example is a brand-new kf-main cluster built from three combined nodes. It can tolerate the loss of one controller; business topics use RF=3/minISR=2, and the scram production security profile is enabled.

1. Define the Secured Cluster and Its Resources

Add the following group to your existing all.children:

all:
  children:
    # keep your existing groups

    kf-main:
      hosts:
        10.10.10.11: { kafka_seq: 1 }
        10.10.10.12: { kafka_seq: 2 }
        10.10.10.13: { kafka_seq: 3 }
      vars:
        kafka_cluster: kf-main
        kafka_data: /data/kafka
        kafka_heap_opts: '-Xms4G -Xmx4G'
        kafka_security: scram

        kafka_parameters:
          num.partitions: 12
          num.network.threads: 6
          num.io.threads: 16
          log.retention.hours: 168
          log.segment.bytes: 1073741824

        kafka_users:
          - name: quickstart-app
            password: "{{ vault_kafka_quickstart_password }}"
            acls:
              - resource: topic
                name: quickstart.
                pattern: prefixed
                operations: [Read, Write, Describe]
              - resource: group
                name: quickstart.
                pattern: prefixed
                operations: [Read]
              - resource: cluster
                name: kafka-cluster
                operations: [Describe, IdempotentWrite]
            quota:
              producer_byte_rate: 10485760
              consumer_byte_rate: 20971520

        kafka_topics:
          - name: quickstart.events
            partitions: 12
            replication_factor: 3
            config:
              min.insync.replicas: 2
              cleanup.policy: delete
              retention.ms: 604800000

vault_kafka_quickstart_password must be supplied by your existing Ansible Vault, KMS, or another secret-injection mechanism, and must be at least 12 characters. Never commit a real password directly to Git, logs, or tickets.

The key semantics of this configuration:

• all three nodes omit kafka_role, so they consistently use combined;
• the new cluster is bootstrapped directly as dynamic KRaft;
• scram enables node TLS, controller mTLS, SCRAM-SHA-512, ACLs, and deny-by-default all at once;
• with three brokers, the initial replication policy is automatically derived as RF=3 and minISR=2;
• quickstart.events is created explicitly with 12 partitions and 3 replicas;
• quickstart-app can read and write quickstart.* topics, read quickstart.* groups, and use the idempotent producer;
• at most two brokers run kafka_exporter, while all three Kafka JVMs run the JMX exporter.

If the three brokers truly reside in different failure domains, you may add kafka_rack: az-a/az-b/az-c to all nodes respectively. Do not use fictitious rack labels to manufacture a disaster-recovery guarantee that does not exist; for the detailed rules, see Cluster Configuration: Rack.

2. Bring Under Management and Deploy

If the nodes are not yet under management:

./node.yml --check -l kf-main
./node.yml -l kf-main

When deploying Kafka, you must select all three members:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

You cannot use -l 10.10.10.11 alone: every selected cluster must be complete, and a partial selection is refused. Selecting several complete clusters at once (-l kf-dev,kf-main) or running bare against all clusters is allowed.

3. Verify the Health of All Three Nodes

From the admin node, check the three Kafka services:

ansible kf-main -b -m command -a 'systemctl is-active kafka'

Run the full health check on any broker:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties

Query the quorum and the topic:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 10.10.10.11:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic quickstart.events

Before launch, you should see one active controller, three current voters, and three available brokers; every quickstart.events partition has three replicas with ISR=3, and there are no offline, under-replicated, or under-min-ISR partitions.

Part 3: Connect Application Clients

1. Distribute the CA Public Certificate

Securely copy the public CA certificate from the admin node to the application host:

files/pki/ca/ca.crt  ->  /etc/kafka-client/pigsty-ca.crt

ca.crt is a public certificate that is safe to distribute. Never copy, expose, or distribute files/pki/ca/ca.key. On the application host, the CA file should be owned by root and set read-only. An application host already managed by Pigsty needs no copy: the NODE module has installed the same CA at /etc/pki/ca.crt, which the client can reference directly.

2. Create the Client Configuration

On the application host, create /etc/kafka-client/client.properties:

bootstrap.servers=10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="quickstart-app" password="<secret-from-vault>";

ssl.truststore.type=PEM
ssl.truststore.location=/etc/kafka-client/pigsty-ca.crt
ssl.endpoint.identification.algorithm=https

The Kafka Java client supports SASL_SSL + SCRAM and a PEM truststore. A real application should inject the password from a secret manager at runtime rather than committing a file containing the password to the repository. For the complete set of fields, see the Kafka 4.3 SASL/SCRAM and producer configuration references.

3. Why Applications Should Connect Directly to Multiple Brokers

The Kafka client is itself cluster-aware. bootstrap.servers is used only to obtain the initial metadata; once connected, the client uses that metadata to connect directly to the leader broker of each partition and refreshes its routing whenever a leader changes. The standard practice in production is therefore to:

• configure at least two, and usually three, broker addresses in bootstrap.servers, located in different failure domains;
• allow the application to reach 9092 on all brokers, and ensure that each broker’s advertised inventory_hostname is resolvable and routable;
• let producers and consumers rely on the Kafka client’s own retry, metadata-refresh, idempotence, and consumer-group protocols;
• not place HAProxy, a Keepalived VIP, a layer-4 load balancer, or a layer-7 reverse proxy in front of the Kafka data plane.

A single VIP or LB can neither substitute for the broker addresses in Kafka’s metadata nor transparently forward a connection to the correct partition leader; it only adds complexity around long-lived connection state, fault localization, and capacity planning. If your platform must provide a unified discovery entry point, a DNS name or TCP load balancer can serve as a bootstrap-only entry point, but each broker’s advertised.listeners must still return an address the client can reach directly, and the application must not be permitted to reach only the LB. Scenarios that span NAT, the public internet, Kubernetes, or multiple networks require a dedicated externally reachable address and an additional listener per broker; the current module fixes the inventory address as advertised.listeners and does not support such mappings.

4. Authenticate and Read/Write with the Application Identity

On an application host with the Kafka 4.3 CLI installed, run:

kafka-console-producer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events

When consuming, use a group prefix permitted by the ACL:

kafka-console-consumer.sh \
  --bootstrap-server 10.10.10.11:9092,10.10.10.12:9092,10.10.10.13:9092 \
  --command-config /etc/kafka-client/client.properties \
  --topic quickstart.events \
  --group quickstart.demo \
  --from-beginning

A production application should also explicitly review its client semantics:

Client-side acks, retries, idempotence, batching, compression, and offset strategy are application configuration and should not be written into the broker’s kafka_parameters.

Part 4: Change Core Parameters

Always express persistent intent for Kafka by editing pigsty.yml, never by editing /etc/kafka/server.properties directly. Common intents map as follows:

Example: Adjust the Heap and Broker Default Parameters

Suppose that after load testing you decide to raise the heap to 6G, increase the thread counts, and change the default retention for new topics to 72 hours:

kf-main:
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms6G -Xmx6G'
    kafka_parameters:
      num.partitions: 12
      num.network.threads: 8
      num.io.threads: 24
      log.retention.hours: 72
      log.segment.bytes: 1073741824

Do not copy 6G/8/24 verbatim; these values must be determined by load-testing against your CPU, memory, connection count, message size, partition count, disk, and page cache.

Example: Add Partitions and Shorten Topic Retention

Increase quickstart.events from 12 partitions to 24 and change the retention to three days:

kafka_topics:
  - name: quickstart.events
    partitions: 24
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 259200000

Partitions cannot be reduced. When replication_factor differs from what is live, the role refuses ordinary convergence and requires an explicit partition reassignment; it never relocates existing replicas automatically.

Apply the Change

Whether you change static parameters or dynamic resources, run the full state machine:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

Do not run -t kafka_config alone. The role decides automatically: a static change triggers a strict node-by-node rolling restart; when only dynamic resources such as topics or users change, Kafka is not restarted.

The following keys belong to the role itself and must not be placed in kafka_parameters:

kafka_parameters:
  min.insync.replicas: 2          # wrong: owned by the role
  default.replication.factor: 3   # wrong: owned by the role
  listeners: ...                  # wrong: owned by the role

For all 15 public parameters, their defaults, and the reserved keys, see the Parameter Reference.

Part 5: Key Pre-Launch Checks

Topology and Data Safety

• Production uses at least three brokers and an odd number of controllers; for critical or large clusters, consider a separated 3-controller + N-broker topology;
• topic RF, minISR, and the producer’s acks form a consistent failure model;
• kafka_rack expresses only real failure domains, and replica placement has been verified;
• data-disk capacity, throughput, latency, retention, peak write rate, and recovery time have been load-tested;
• there is an explicit reassignment plan for after a new broker joins, since existing topics’ RF is not raised automatically;
• the procedures for Kafka data backup/rebuild and disaster recovery are defined, and failed-node replacement and member retirement have been rehearsed.

Security and Network

• a new production cluster uses kafka_security: scram from bootstrap onward;
• application passwords are injected by Vault/KMS/a secret manager and never reach Git or logs;
• only the CA public certificate is distributed to clients, never the CA private key;
• clients can resolve and reach the inventory_hostname of every broker directly;
• 9092/9093 are open only to the principals that need them, and 9308/9404 only to the monitoring network;
• a review checklist for application principals, topic/group/cluster ACLs, and quotas is in place;
• a protected rotation of internal credentials and certificates is scheduled.

Operations and Monitoring

• /usr/local/bin/pigsty-kafka-health cluster reports healthy;
• the dynamic quorum has exactly one leader, and every expected controller is in the current voters;
• there are no offline, under-replicated, or under-min-ISR partitions;
• produce/consume verification is done over the real application network with a real principal;
• the Kafka Overview, Kafka Instance, Kafka Topic, and Kafka Consumer dashboards show healthy data;
• alert routing, log search, capacity thresholds, on-call responsibilities, and rollback conditions are confirmed;
• upgrades, feature-level changes, topic deletion, user deletion, and cluster teardown each have a separate approval process.

For detailed alerts and PromQL, see Monitoring and Alerting; for metric semantics, see Metric Definitions.

Documentation Index and Next Steps

We recommend continuing along the following path:

One recommended reading path is: Quick Start → Cluster Configuration → Parameter Reference → Day-to-Day Administration → Playbook → Monitoring and Alerting → FAQ.

2 - Configuration

The KAFKA module expresses cluster intent through 15 persistent public parameters; everything else — topology, listeners, storage subdirectories, replication safety, authorization, and Exporter placement — is derived by the role in a single, consistent way. For a first deployment, start with the Quickstart; for the full field reference, see Parameters.

Pre-Deployment Checklist

Before filling in the inventory, confirm at least the following:

• The target hosts are already commissioned by NODE, the software repository is reachable, and inventory_hostname is directly routable by every Kafka member and client.
• A single operation will use -l to select precisely all members of one kafka_cluster — not a single node, a subset of members, or multiple clusters.
• kafka_seq is unique within the cluster, the controller count is odd, and the broker count matches your fault domains and capacity targets.
• Ports 9092, 9093, 9308, and 9404 do not collide, and Infra nodes can reach both metrics ports.
• kafka_data maps to a dedicated filesystem, sized for retention, write peaks, replication traffic, recovery time, and growth headroom.
• Production uses kafka_security: scram; node and admin clocks are synchronized, the Pigsty CA is available, and application passwords come from a secret source such as Vault.
• Topic partitions, replicas, min.insync.replicas, and retention policy — along with client acks, retries, and consumer recovery strategy — have been reviewed.
• Scale-out and scale-in, partition reassignment, upgrades, backup, restore, and controller membership changes each have their own runbook.

Roles and Topology

kafka_role accepts only three values:

kafka_role is all-or-nothing: cluster members either all omit it (and consistently use combined) or all declare it explicitly — a mix is refused during the identity precheck. A cluster must contain at least one controller-capable node and at least one broker-capable node; an even number of controllers produces a warning, and production typically uses 3 controllers.

Single-Node Development Cluster

A single node serves as both broker and controller. It cannot tolerate a node failure and is suitable only for development, testing, and feature validation:

kf-dev:
  hosts:
    10.10.10.10: { kafka_seq: 1 }
  vars:
    kafka_cluster: kf-dev

The role derives RF=1 and minISR=1 from the initial broker count. Do not use a single-node topology or the default plaintext security profile directly in production.

Three-Node Combined Deployment

All three nodes serve as both broker and controller — a compact production starting point. Omit every role field to use the default combined:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1 }
    10.10.10.12: { kafka_seq: 2 }
    10.10.10.13: { kafka_seq: 3 }
  vars:
    kafka_cluster: kf-main
    kafka_heap_opts: '-Xms4G -Xmx4G'
    kafka_security: scram
    kafka_parameters:
      num.partitions: 3
      num.network.threads: 6
      num.io.threads: 16
    kafka_topics:
      - name: order.events
        partitions: 12
        replication_factor: 3
        config:
          min.insync.replicas: 2
          cleanup.policy: delete

The initial three brokers automatically get the role-owned replication policy of RF=3 and minISR=2. You neither need nor are allowed to override the internal-topic RF, default.replication.factor, or min.insync.replicas in kafka_parameters. The 4G heap in the example is only illustrative; in production, balance the JVM heap, the operating-system page cache, and any other processes on the same host through load testing.

Separating Controllers and Brokers

Critical or larger clusters can separate the control plane from the data plane. Because explicit roles are present, every member must declare its role:

kf-main:
  hosts:
    10.10.10.11: { kafka_seq: 1, kafka_role: controller }
    10.10.10.12: { kafka_seq: 2, kafka_role: controller }
    10.10.10.13: { kafka_seq: 3, kafka_role: controller }
    10.10.10.21: { kafka_seq: 4, kafka_role: broker }
    10.10.10.22: { kafka_seq: 5, kafka_role: broker }
    10.10.10.23: { kafka_seq: 6, kafka_role: broker }
  vars:
    kafka_cluster: kf-main
    kafka_security: scram

A controller-only node does not listen on 9092 and does not run the protocol Exporter; it still exposes KRaft and JVM state through JMX. At most two kafka_exporter instances are placed on the broker-capable nodes with the lowest kafka_seq.

Dynamic KRaft and the Bootstrap Manifest

A new cluster uses the dynamic quorum directly: every node renders controller.quorum.bootstrap.servers, and no static controller.quorum.voters is generated. At the first format:

• The Cluster ID is generated randomly, not hashed from the cluster name;
• The Directory IDs of the initial controllers are generated randomly and frozen;
• Each node is formatted explicitly with either --initial-controllers or --no-initial-controllers mode;
• After the first bootstrap startup, the role waits for the dynamic quorum to elect a leader, and verifies that every initial controller’s Directory ID has entered the live quorum.

The bootstrap-only facts live on every cluster member:

/etc/kafka/manifest.yml

Each member of a scram cluster also holds /etc/kafka/secrets.yml. The admin node keeps no kafka state at all: the manifest and secrets are resolved from any member copy on every run, and the issued node certificates live in the shared PKI tree (files/pki/kafka/, with CSRs under files/pki/csr/) and are simply re-signed from the Pigsty CA when absent. The manifest records only the cluster identity, the initial controller identities, the security profile, and the initial RF/minISR. The live cluster is always the authority on runtime facts:

• When the manifest conflicts with the live identity or security profile, ordinary playbooks fail closed;
• When an old manifest exists but all data disks are empty, reviving the old cluster is refused;
• When no member holds a manifest copy while storage is already formatted, the role fails closed and asks you to restore the file on any member first;
• An already-formatted scram cluster likewise fails closed when no member holds the secret material.

The manifest is the cluster’s birth certificate: after the first commission, membership is authoritative in the live Raft state. Combined/controller nodes newly declared in the inventory are then joined to the dynamic quorum by the playbook (fresh format → observer catch-up → add-controller promotion), and retirement is a kafka-rm.yml strict-subset run (automatic remove-controller and broker unregistration) — see Expand Cluster and Shrink Cluster.

Identity Parameters

An already-formatted node reads cluster.id and node.id from ${kafka_data}/metadata/meta.properties and cross-checks them against the manifest and the inventory; the initial controllers’ Directory IDs are compared against the live quorum after startup. An identity mismatch is a protective failure and must not be worked around by deleting meta.properties or wiping data.

Network and Listeners

The role exposes only ports — not bind addresses, advertised addresses, or the listener map:

The fixed listener conventions are as follows:

• The broker listener binds 0.0.0.0, and the controller listener binds inventory_hostname;
• The broker’s advertised.listeners uses inventory_hostname;
• The controller bootstrap address also uses inventory_hostname;
• plaintext: both BROKER and CONTROLLER use PLAINTEXT;
• scram: BROKER uses SASL_SSL + SCRAM-SHA-512, and CONTROLLER uses mutual TLS.

Clients must therefore be able to resolve and directly reach every broker’s inventory_hostname. The current v1 does not support NAT, public-address mapping, multiple client networks for the same broker, or arbitrary raw listener overrides; these scenarios cannot be assembled around through kafka_parameters.

Kafka’s standard access model is a smart client connecting directly to brokers: bootstrap.servers is configured with several seed addresses, and once the client fetches cluster metadata it connects directly to the partition leaders. HAProxy, a Keepalived VIP, or a cloud LB should not be used as the regular Kafka data-plane entry point, because they are unaware of Kafka metadata and partition leaders and cannot exempt clients from reaching every advertised.listeners address. A DNS or TCP LB can at most serve as an optional bootstrap discovery entry point; even then, the application network must still reach all brokers directly. See Quickstart: Connecting Application Clients for details.

The minimum required network flows:

The metrics ports are HTTP, and even when Kafka uses scram they should be restricted to the monitoring network by firewall.

Storage, Heap, and Rack

You set only the root directory:

kafka_data: /data/kafka

The role derives the topic data directory ${kafka_data}/data and the KRaft metadata directory ${kafka_data}/metadata in a fixed way. kafka_data must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg.

Production planning should account for at least retention, message peaks, replication traffic, partition/segment counts, disk latency and throughput, file descriptors, recovery time, JVM heap, and page cache. The current role generates only a single log.dirs; multi-disk JBOD, disk replacement, and automatic data migration require separate runbooks.

For deployments spanning fault domains, declare kafka_rack consistently on all broker-capable nodes:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }
10.10.10.22: { kafka_seq: 5, kafka_role: broker, kafka_rack: az-b }
10.10.10.23: { kafka_seq: 6, kafka_role: broker, kafka_rack: az-c }

Broker-capable nodes must either all set the rack or all omit it. Changing the rack triggers a safe rolling restart but does not automatically migrate existing replicas.

Replication Policy

At the first bootstrap, the policy is derived from the initial broker count:

replication_factor = min(3, broker_count)
min_insync_replicas = max(1, replication_factor - 1)

The initial default RF for future topics, the internal-topic RF, and the cluster minISR are all written into the manifest and frozen. After scaling out:

• default.replication.factor keeps its initial value; Kafka 4.3 does not allow changing it online through dynamic broker configuration;
• The RF of existing internal/business topics is not raised automatically;
• The role does not report “brokers have joined” as “data is balanced”;
• An RF change must use a reviewed kafka-reassign-partitions.sh plan; raising the static default additionally requires controller high availability or an explicit maintenance window, and takes effect through a full-cluster safe rolling restart.

Producer acks, idempotence, retries, batching, and compression are client policy, not parameters of the Kafka broker role.

kafka_parameters

kafka_parameters is the only broker-parameter escape hatch. It defaults to {} and is rendered only onto broker-capable nodes. It is suited to non-role-owned keys such as num.partitions, thread counts, buffers, retention, and segments.

The following patterns are owned by the role and must not be overridden:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

If any reserved key appears, the identity preflight fails outright before any file is written.

Security and Declarative Resources

kafka_security: scram is a complete production profile, not a set of switches to be combined at will. It automatically enables:

• Per-node certificates issued by the Pigsty CA;
• Mutual TLS on the controller listener;
• SASL_SSL + SCRAM-SHA-512 for broker/client and inter-broker traffic;
• StandardAuthorizer, deny-by-default, and role-owned admin/monitoring identities;
• Convergence of the minimal monitoring ACL before the protocol Exporter starts.

Application resources are declared through two domain objects:

kafka_security: scram
kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520
kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete

Resource convergence semantics: topic creation is idempotent, partitions only increase, and only explicitly declared config is updated; an RF change is refused with a prompt to run reassignment. A declared user’s password, ACLs, and the quota fields you provide converge idempotently. Removing a topic or user entry is not an implicit deletion procedure.

The security profile cannot be switched by an ordinary playbook after bootstrap. Internal credentials and certificates can use protected rotation, but an online migration from plaintext to scram still requires an explicit state machine that is planned for the future.

Packages and File Layout

The role installs java-runtime and kafka-stack through platform mappings. The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0; the actual versions still depend on the target platform’s repository and the installed packages.

These files are managed by the role. Persistent intent belongs in pigsty.yml. Do not edit generated files directly on the nodes, and do not copy passwords, private keys, or role-owned secret contents into inventory, logs, or tickets.

3 - Parameters

The KAFKA role deliberately exposes only 15 persistent parameters. Details such as topology, listeners, security implementation, storage subdirectories, replication safety, and Exporter placement are derived by the role in a single, consistent way, and cannot be overridden as additional persistent variables.

Parameter Overview

kafka_cluster and kafka_seq must be defined; kafka_role has a real default. The cluster’s roles are either all omitted or all declared explicitly.

Identity and Topology

kafka_cluster

The required cluster identity. It must start with a letter or digit and contain only letters, digits, underscores, and hyphens:

kafka_cluster: kf-main

It is used to discover the complete cluster membership, generate instance names, and locate the bootstrap manifest. Every kafka.yml lifecycle operation must use a precise -l to select all members of this cluster.

kafka_seq

A required non-negative integer, unique within the same kafka_cluster, which becomes the KRaft node.id directly:

10.10.10.11: { kafka_seq: 1 }

The instance name is derived as ${kafka_cluster}-${kafka_seq}. Once a node has been formatted, do not change or reuse a sequence number that still has associated data.

kafka_role

Defaults to combined, and accepts only:

When all cluster members omit it, they consistently use combined; as soon as any member sets it explicitly, all members must set it explicitly. No legacy role aliases are provided.

kafka_cluster_id

Unset by default, used only to assert the identity of an existing cluster during takeover or recovery. It must be a 22-character Kafka UUID:

kafka_cluster_id: MkU3OEVBNTcwNTJENDM2Qk

Do not set it for an ordinary new cluster. The role generates the Cluster ID randomly and writes it into every member’s /etc/kafka/manifest.yml. This parameter does not relabel existing data; it fails closed when it conflicts with the manifest or meta.properties.

kafka_rack

An optional broker fault-domain label, rendered as broker.rack:

10.10.10.21: { kafka_seq: 4, kafka_role: broker, kafka_rack: az-a }

All broker-capable nodes must either all declare it or all omit it. Controller-only nodes do not use this value. Changing the rack is a static change that goes through a strict rolling restart, but does not reassign existing replicas.

Storage, JVM, and Network

kafka_data

The data root directory, defaulting to /data/kafka:

kafka_data: /data/kafka

The role derives ${kafka_data}/data and ${kafka_data}/metadata in a fixed way. This path must be a dedicated absolute path, and cannot be /, /data, /var, /etc, /opt, /usr, /home, /root, or /pg. kafka-rm.yml deletes the entire root directory by default, so do not mix other services or business files into it.

kafka_heap_opts

The Kafka JVM heap, defaulting to:

kafka_heap_opts: '-Xms1G -Xmx1G'

In production, set it according to load and memory load testing. Typically keep Xms and Xmx equal, and leave enough memory for the operating-system page cache and other processes.

kafka_port

The broker/client listener port, defaulting to 9092, and listening only on broker-capable nodes. plaintext mode uses PLAINTEXT; scram mode uses SASL_SSL + SCRAM-SHA-512.

kafka_controller_port

The KRaft controller listener port, defaulting to 9093 (the conventional Kafka KRaft port), and listening only on controller-capable nodes. When sharing a node with other services, verify yourself that the ports do not collide; the role does not automatically detect cross-service port usage.

The four public ports must all differ from one another. The broker listener binds 0.0.0.0, while the controller listener, the broker advertised address, and the controller bootstrap address all use inventory_hostname in a fixed way, with no separate address parameters.

kafka_parameters

Defaults to {}. It is the only Kafka broker-parameter escape hatch, rendered only onto broker-capable nodes:

kafka_parameters:
  num.partitions: 12
  num.network.threads: 6
  num.io.threads: 16
  log.retention.hours: 168
  log.segment.bytes: 1073741824

The following keys or patterns are owned by the role and cannot be overridden through this mapping:

process.roles
node.id
controller.quorum.*
listeners
advertised.listeners
listener.security.protocol.map
inter.broker.listener.name
controller.listener.names
log.dirs
metadata.log.dir
min.insync.replicas
default.replication.factor
offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr
share.coordinator.state.topic.replication.factor
share.coordinator.state.topic.min.isr
broker.rack
authorizer.class.name
super.users
allow.everyone.if.no.acl.found
sasl.*
ssl.*
listener.*

Identity, listeners, security, storage, and replication policy must remain single-authority; when a reserved key is included, the preflight fails outright.

Observability

kafka_jmx_exporter_port

The JMX Exporter HTTP port, defaulting to 9404. The role injects the JMX Exporter Java agent unconditionally into every Kafka JVM and registers it as job=kafka; there is no separate toggle parameter. The lifecycle health gate uses the role-owned Kafka CLI/metadata channel and does not depend on JMX. Infra monitoring nodes must be able to reach this port; the endpoint does not automatically enable HTTPS because of kafka_security: scram, so it should be protected by the monitoring network and firewall.

kafka_exporter_port

The HTTP port of the protocol-level kafka_exporter, defaulting to 9308. The role configures, starts, and registers it only on the first two broker-capable nodes after sorting by kafka_seq; a single-broker cluster runs only one. The monitoring target file is refreshed on every full run according to the current placement, but a stale Exporter service on a node that was previously selected is not stopped automatically by an ordinary playbook.

The Kafka protocol version, TLS/SCRAM parameters, and replica placement used by the Exporter are all internal role conventions, with no additional public toggles or options parameters.

Security and Resources

kafka_security

Defaults to plaintext, and accepts only:

scram simultaneously configures the Pigsty CA-issued node certificates, the role-owned admin/monitoring/internal identities, and the ordering in which TLS/SCRAM and ACLs are enabled. The security profile is written into the bootstrap manifest; once the cluster is formatted, an ordinary rerun can neither switch plaintext to scram nor switch back.

Node certificate validity follows Pigsty’s shared CA parameter cert_validity (7300d by default); the KAFKA module has no separate certificate validity parameter.

kafka_users

Defaults to [], and may only be declared in scram mode. Each object accepts only name, password, acls, and quota:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: order.
        pattern: prefixed
        operations: [Read, Write, Describe]
      - resource: group
        name: order.
        pattern: prefixed
        operations: [Read]
      - resource: transactional_id
        name: order.
        pattern: prefixed
        operations: [Write, Describe]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

Constraints:

• name is unique within the list; password is required, must be at least 12 characters, and should reference a secret management system;
• ACL resource is one of topic, group, transactional_id, cluster;
• pattern is literal (default) or prefixed;
• Operations are Read, Write, Create, Delete, Alter, Describe, ClusterAction, DescribeConfigs, AlterConfigs, IdempotentWrite;
• Quota keys are producer_byte_rate, consumer_byte_rate, request_percentage, controller_mutation_rate.

The role converges the SCRAM password, the complete ACL set, and the explicitly provided quota fields for a declared user. Removing a user entry does not implicitly delete the principal or credentials; deletion/revocation requires a separate, audited operation.

kafka_topics

Defaults to []. Each object accepts only name, partitions, replication_factor, and config:

kafka_topics:
  - name: order.events
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      cleanup.policy: delete
      retention.ms: 604800000

The identity precheck only validates that name is unique within the list. Partition and RF validity (at least 1, and RF not exceeding the current broker count) is decided by Kafka at creation time, so such errors surface during the resource convergence stage rather than under --check. The convergence semantics are:

• The topic is created idempotently when it does not exist;
• Partitions may only increase; decreasing fails;
• When RF differs from the live state, ordinary convergence is refused and an explicit reassignment is required;
• Only the keys declared in config are updated;
• Removing a topic from the list never deletes the topic.

Transient Protected Operational Variables

The following variables are used only via the command-line -e for one-off operational actions. They are not part of the 15 persistent API parameters and should not be written into pigsty.yml:

The two rotation actions are mutually exclusive, and must target a precise, complete cluster. kafka-rm.yml deletes the data directory and node-local /etc/kafka recovery state by default; kafka_rm_data=false retains both. Before running it, explicitly confirm the target cluster and your backup/rebuild intent. For the commands and full semantics, see Playbooks.

kafka_safeguard

For kafka-rm.yml only, false by default. When set to true, the removal role aborts before deregistration, retirement, service shutdown, and deletion. This is a boolean safety switch; it does not probe whether the cluster is alive.

kafka_rm_data

For kafka-rm.yml only, true by default. When enabled it deletes the entire kafka_data directory and /etc/kafka, the latter holding the manifest, credential copies, and the recovery state required to re-adopt retained storage. Setting it to false keeps both, but monitoring targets are still deregistered, services still stopped, and runtime integration config still removed.

kafka_rm_pkg

For kafka-rm.yml only, false by default. When set to true it uninstalls the kafka-stack packages from the platform mapping (the Kafka, Kafka Exporter, and JMX Exporter payload); the shared Java runtime is never uninstalled.

4 - Administration

The KAFKA module installs Kafka under /opt/kafka, manages the service with Systemd, and keeps its persistent intent in pigsty.yml. The files generated on the nodes are not meant to be edited by hand.

All of the Kafka CLI examples below use the role-generated /etc/kafka/admin.properties. Even when the current profile is plaintext, keep --command-config on every command: that way the command structure stays the same when you switch the admin channel to scram. Replace <broker>:9092 with a reachable inventory_hostname and port.

Quick Reference

Cluster definition and parameters are covered in Configuration, playbook semantics in Playbooks, and monitoring in Monitoring.

Status Check

Check the service and recent logs on any Kafka node:

systemctl status kafka
systemctl is-enabled kafka
journalctl -u kafka --since '-30 min' --no-pager

The protocol exporter runs only on at most two broker-capable nodes with the lowest kafka_seq. On the selected nodes, also check:

systemctl status kafka_exporter
journalctl -u kafka_exporter --since '-30 min' --no-pager

Check listeners and metric endpoints:

ss -lntp | grep -E ':9092|:9093|:9308|:9404'
curl -fsS http://<kafka-ip>:9404/metrics | grep -E '^(jmx_scrape_error|kafka_server_raft_state|kafka_server_broker_messages_in_total)'
curl -fsS http://<exporter-ip>:9308/metrics | grep -E '^(kafka_brokers|kafka_topic_partitions)'

kafka_up and kafka_exporter_up are recording metrics on the VictoriaMetrics side and do not necessarily appear on the raw endpoints. The JMX endpoint should contain jmx_scrape_error 0.0, JVM metrics, and kafka_ metrics matching the node’s role.

Health Check

The role’s lifecycle gates do not rely on JMX. They check the dynamic quorum, unavailable partitions, under-replication, and under-min-ISR through the same admin channel:

sudo -u kafka /usr/local/bin/pigsty-kafka-health cluster \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties

Only healthy: true in the returned JSON means the gate passes. It is suitable for read-only diagnostics but is no substitute for end-to-end business validation.

The script also ships a built-in parser regression suite (pigsty-kafka-health selftest) that every playbook run executes right after installation; if the selftest fails, the health predicate itself cannot be trusted — stop making changes and investigate.

KRaft Quorum Status

Query the dynamic quorum from any available broker:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --status

Key things to verify:

• LeaderId exists and matches an expected controller;
• CurrentVoters matches the expected membership (a joining node appears under CurrentObservers first);
• MaxFollowerLag and MaxFollowerLagTimeMs are not growing continuously;
• Exactly one active controller shows on the dashboards.

To confirm the dynamic quorum (KIP-853) feature level, inspect kraft.version with /opt/kafka/bin/kafka-features.sh ... describe.

Inspect controller replication state:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  describe --replication

If there is no leader, a member lags persistently, or the voter set differs from expectation, stop other changes first and preserve the logs, the manifest, and meta.properties as evidence before analyzing. Remove a dead voter through the shrink or replace flows; never rewrite quorum state by hand.

Manage Topics

Production topics should be declared in kafka_topics in pigsty.yml:

kafka_topics:
  - name: orders
    partitions: 12
    replication_factor: 3
    config:
      min.insync.replicas: 2
      retention.ms: 604800000

Converge after editing the declaration:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role creates topics idempotently, only grows partitions, and only modifies the declared config keys. An RF change fails and demands an explicit partition reassignment; removing an entry from the inventory does not delete the topic.

Read-only topic inspection:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --topic orders

Ad-hoc or externally managed topics can be created with the Kafka CLI, but they are not written back into pigsty.yml. Never let declarative and manual management own the same topic. Topic deletion is a business-data deletion: it requires separate approval, exact-name confirmation, and a recovery plan, so no generic delete command is given here.

Manage Users and ACLs

With kafka_security: scram, application identities should be managed through kafka_users:

kafka_users:
  - name: order-service
    password: "{{ vault_kafka_order_password }}"
    acls:
      - resource: topic
        name: orders
        operations: [Read, Write, Describe]
      - resource: group
        name: order-worker
        operations: [Read]
    quota:
      producer_byte_rate: 10485760
      consumer_byte_rate: 20971520

A full playbook run idempotently converges the password, the user’s ACL set, and the explicitly given quota fields. Never commit passwords in plaintext or print them into logs. Removing a user entry does not delete the principal/credentials automatically; deletion or full revocation needs a separately reviewed procedure.

Verify Read/Write

Use a test topic for end-to-end validation. The console producer/consumer share the same client config file:

/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke

Consume in another terminal:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --topic ops-smoke \
  --from-beginning \
  --group ops-smoke-check

Production acceptance should run from the real client network, covering DNS/advertised.listeners, certificate validation, ACLs, producer ACKs, consumer commits, and end-to-end latency — not just the broker-local path.

Manage Consumer Groups

List and inspect consumer groups:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --list

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group order-worker

Judge lag against the consumption rate and business SLO: a short backlog can be batch-processing behavior, while sustained growth with consumption slower than production means the group cannot catch up. Resetting offsets may duplicate or skip messages, so it requires separate approval, exact group/topic confirmation, and a replay plan.

Config Cluster

After editing pigsty.yml, run against complete clusters:

./kafka.yml --check -l kf-main
./kafka.yml -l kf-main

The role picks the path from live health and the static fingerprint:

• Cluster unhealthy or stopped: start only the stopped controllers, restore and catch up the quorum, then start the brokers; if static changes coexist, the still-online members proceed into the strict rolling afterwards;
• Controller-capable nodes awaiting quorum join: each catches up as an observer and is promoted with add-controller, one at a time;
• Healthy cluster with new pure brokers: format, start, and confirm registration one at a time;
• Healthy cluster with static changes: strict node-by-node rolling, with pre/post gates on controller majority and voter catch-up, offline partitions, under-min-ISR, and ISR catch-up around every restart;
• No static change: Kafka is not restarted.

Do not bypass the full state machine with -t kafka_config. Dynamic topic/user/ACL/quota convergence lives in the kafka_provision resource stage; whether a static change restarts anything is decided by the role.

Expand Cluster

A healthy cluster takes new members declared directly in the inventory: kafka_role: broker, combined, or controller all work. Give each new node a never-used kafka_seq (a host can belong to only one Kafka cluster at a time), make sure the node is managed by Pigsty, then still target complete clusters:

./node.yml  --check -l 10.10.10.14    # manage the new node
./kafka.yml --check -l kf-main        # dry run first
./kafka.yml -l kf-main                # admit / join new members one at a time

The role picks the path per member type and handles one new node at a time:

• Pure broker: format, start, and verify the broker is registered and not fenced (admit);
• Combined / controller: format fresh with --no-initial-controllers, start as an observer and catch up on metadata, promote with add-controller, then verify it entered the voter set with the cluster fully healthy (join).

The quorum-join-hosts / broker-admission-hosts summary at the end of the run lists the nodes actually processed. Two reminders:

• Adding a controller-capable node changes controller.quorum.bootstrap.servers on every member, so the existing nodes go through one gated strict rolling round afterwards — this is expected;
• Scaling out to an even controller count prints a warning: an even quorum adds no fault tolerance, keep the count odd.

Joining does not migrate existing partitions onto the new broker. Generate, review, and monitor a kafka-reassign-partitions.sh plan separately, control the disk/network load, and prepare a rollback. “The service is registered” is not “the expansion is complete.”

The replication policy does not scale up with the broker count either. In particular, Kafka 4.3’s default.replication.factor cannot be changed dynamically: after scaling from 1 broker to 3, it remains the RF=1 set at initial build, and any future topic without an explicit RF is still created with RF=1. First complete the reassignment of existing partitions, then plan for controller high availability or a maintenance window, and finally let the new static default take effect through a safe full-cluster rolling restart. Do not bypass the downtime gates just to change a default.

Shrink Cluster

Selecting a strict subset of a cluster with kafka-rm.yml retires members (selecting the whole cluster is a teardown). Retirement removes the leaving node from live metadata through a surviving member:

./kafka-rm.yml -l 10.10.10.13     # retire one member: remove voter entry, unregister broker, clean the host

The execution order is: deregister monitoring targets → stop services → remove-controller to remove the KRaft voter entry (if the member is a voter; strictly serialized when retiring several members) → kafka-cluster.sh unregister to drop the broker registration → clean local config and data (controlled by kafka_rm_data). Afterwards, delete the member’s entry from pigsty.yml.

Before retiring, confirm yourself that: the remaining controllers still form a majority, the controller count stays odd, and the remaining broker count is not below the highest topic RF. If the retiring broker still hosts partition replicas, the role prints a warning: those partitions stay under-replicated until a replacement with the same kafka_seq rejoins (it re-inherits the assignment and resyncs automatically), or until you reassign them explicitly. A planned shrink should drain with a reassignment first, then retire.

Replace Failed Node

When a node is permanently lost (disk gone, machine scrapped), keep its IP and kafka_seq and replace it in three steps:

./kafka-rm.yml -l 10.10.10.13     # 1. retire the dead member: remove voter entry & broker registration (works while unreachable)
./node.yml     -l 10.10.10.13     # 2. provision the replacement machine (repaired or new, same IP)
./kafka.yml    -l kf-main         # 3. rejoin: format, catch up, admit/promote; replica assignment is re-inherited and resynced

All metadata operations in step 1 are delegated to a surviving member, so it works even when the node itself is unreachable; it also cleans up the monitoring target, so the dead node stops firing KafkaDown. In step 3, a broker with the same kafka_seq automatically re-inherits its former partition assignment and resynchronizes from the surviving replicas — no manual reassignment needed.

If you skip step 1 and rerun kafka.yml against a re-imaged node directly, the role fails fast in the config phase, reporting the stale voter entry’s directory ID together with the exact kafka-rm.yml command — run it and retry. The join flow is safely re-entrant: if any step is interrupted, rerunning kafka.yml continues from live state.

Change Address or Port

The role always uses inventory_hostname as the broker’s advertised address and the controller’s bootstrap address. Changing an inventory address, kafka_port, or kafka_controller_port affects client metadata, broker communication, or the quorum, and counts as a high-risk static change: check DNS, certificate SANs, routing, firewalls, bootstrap addresses, monitoring targets, and all cluster members in lockstep.

Rotate Credentials and Certificates

A formatted, healthy scram cluster supports two mutually exclusive protected actions: internal credential rotation and certificate rotation. Both require exact complete clusters and a matching kafka_rotate_confirm confirmation string, and running --check first is recommended. Certificates are re-issued by the same Pigsty CA, old and new certificates trust each other, and the rotation takes effect node by node through the strict rolling restart.

For the exact commands and failure semantics, see Playbook: Protected Rotation. The security profile itself is a bootstrap-only property; these actions do not imply support for online migration from plaintext to scram.

Data Protection and Recovery

Kafka’s data protection relies on replicas across failure domains, correct min-ISR, producer ACKs, and a rehearsed recovery procedure. The current role does not provide Kafka data backup, automatic broker drain (a planned shrink needs a manual reassignment first), or cross-region disaster recovery.

When a disk or node fails:

1. First look at the Kafka Overview/Node dashboards, the quorum, ISR, offline partitions, and under-min-ISR partitions;
2. Preserve the evidence: journalctl -u kafka, node metrics, the manifest, server.properties, and meta.properties;
3. Confirm the node’s role, node.id, cluster ID, directory ID, and the availability of the remaining replicas;
4. Once the node is confirmed unrecoverable, follow Replace Failed Node: retire with kafka-rm.yml → provision with node.yml → rejoin with kafka.yml; if the disk survives and only the service misbehaves, do not rush to retire or delete meta.properties — try an ordinary converge first;
5. Data-movement operations such as reassignment or RF changes still deserve a separately reviewed runbook.

Log Diagnostics

journalctl -u kafka -f
journalctl -u kafka_exporter -f
journalctl SYSLOG_IDENTIFIER=kafka --since today
journalctl SYSLOG_IDENTIFIER=kafka_exporter --since today

VictoriaLogs/Grafana queries:

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter

The usual diagnostic order is: service logs → listening ports → admin-channel health → dynamic quorum → broker/partition/ISR → client addresses and certificates/ACLs → consumer lag. For dashboards and alert mappings, see Monitoring.

5 - Playbook

The KAFKA module ships two playbooks: kafka.yml deploys an Apache Kafka 4.1+ dynamic KRaft cluster and converges its security, resource, and monitoring state; kafka-rm.yml tears down a cluster or removes a member.

kafka.yml

./kafka.yml --check -l kf-main   # dry run first
./kafka.yml -l kf-main           # create or converge one cluster
./kafka.yml                      # bare run: create / converge ALL kafka clusters

The limit rule is: every selected cluster must be complete. You may select one cluster, several clusters, or run bare against the whole inventory (strictly serial within a cluster, concurrent across clusters); a partial selection of a cluster’s members is refused outright.

Check mode validates the public API, the full cluster, roles, racks, ports, the manifest, and any inspectable file changes, but it skips formatting, service startup, and live health acceptance. A successful --check is therefore not a guarantee of a successful runtime.

Execution Stages

kafka.yml is itself a thin wrapper: a single play runs the node_id and kafka roles in sequence, mirroring the structure of pgsql.yml. Inside the role, the lifecycle is split into six task stages; all cross-node ordering (parallel bootstrap, one-controller-at-a-time join, one-broker-at-a-time admission, strict node-by-node rolling) is handled centrally by the launch stage:

The play uses any_errors_fatal: true. When a stage fails, dangerous forward progress stops; once you fix the cause, you can re-run against the full cluster, and the role recovers from live state and the persistent fingerprint instead of blindly reformatting.

Lifecycle Paths

The config stage uses the role’s own admin channel to judge cluster health and select exactly one downstream path:

Cold Start, First Deployment, or Repair

When the cluster is stopped or the health predicate does not hold, it enters converge:

1. Start all controller-capable nodes;
2. Wait for the controller listener and a dynamic quorum leader;
3. On the first bootstrap, verify that the initial controller directory IDs have entered the live quorum;
4. Start the pure brokers;
5. Wait for the broker listener and require the full cluster to be healthy;
6. Persist the static fingerprint only after the config has been proven to run successfully.

JMX plays no part in the lifecycle gates: the decisions for startup, admission, and rolling are made entirely on the role’s own Kafka CLI/metadata admin channel.

Adding Brokers or Controllers to a Healthy Cluster

Newly formatted kafka_role: broker nodes are admitted one at a time (admit): after starting, each must be registered and not fenced before the next one proceeds.

New combined/controller nodes join the dynamic quorum one at a time instead (join): on a commissioned cluster the node is formatted fresh with --no-initial-controllers, starts as an observer and catches up on metadata, then the role promotes it with add-controller and verifies through the health post-check that it entered the voter set with the cluster fully healthy. The join flow is re-entrant: an interrupted run continues from live state on the next rerun, and if the node’s node.id still has a stale voter entry from a dead predecessor, the config phase fails fast with the exact kafka-rm.yml retirement command.

Admission/join only proves membership; existing partitions are not migrated onto the new broker automatically, so you must run an explicit reassignment separately.

Static Changes on a Healthy Cluster

When the rendered static fingerprint changes, the strict rolling restart handles one node at a time:

• Before restarting, it checks the controller majority, that all voters have zero lag and recently completed catch-up, offline partitions, under-replicated, under-min-ISR, and the effective ISR of each partition once the target is excluded;
• After restarting, it requires the target controller to be back as a voter and re-caught-up, the target broker to be registered and not fenced, and its replicas to re-enter the ISR;
• Any failed gate immediately stops the remaining nodes.

If a fault to repair and a static change coexist, converge only starts the stopped members and does not restart the still-online members in parallel; once the quorum recovers and catches up, the static changes that have not yet been loaded proceed into the strict rolling restart.

If the static fingerprint is unchanged, Kafka is not restarted. Dynamic resource changes still take effect online during the resource-convergence stage.

Task Tags

An ordinary configuration change should run the full kafka.yml and let the role choose its own lifecycle path. Stage tags are meant primarily for development, diagnostics, and controlled repair; you cannot bypass the full state machine with -t kafka_config or by limiting to a single node.

Identity, Formatting, and Manifest

Before writing any config, the role validates that:

• Every selected cluster contains all of its members;
• kafka_seq is unique, and the roles are either all omitted or all explicit;
• There is at least one controller and one broker;
• Racks are either all present or all absent across the broker-capable nodes;
• Ports are valid and non-conflicting, and the role-owned keys are not overridden by kafka_parameters;
• The manifest, security profile, meta.properties, and the live cluster identity are consistent.

A new cluster randomly generates the cluster ID and the initial controller directory IDs and formats each node in explicit dynamic-quorum mode. When ${kafka_data}/metadata/meta.properties already exists, it validates the cluster ID and node ID locally; initial controller directory IDs are compared against the live quorum only on the first bootstrap — after commissioning, membership is authoritative in the live Raft state. The role never reformats existing storage automatically.

The authoritative bootstrap manifest lives on every cluster member:

/etc/kafka/manifest.yml

Each member of a scram cluster additionally has /etc/kafka/secrets.yml; the admin node keeps no kafka state and resolves both from any member copy on every run. The live cluster is the authoritative runtime fact, but an ordinary playbook will not silently rewrite either side on conflict:

• When no member has a manifest copy but the storage is already formatted, it fails closed and asks you to restore the file on any member first;
• When a manifest exists but all data disks are empty, it fails closed;
• When the cluster ID, security profile, or controller identity conflicts, it fails closed;
• A new node whose node.id still has a stale predecessor voter entry in the quorum fails fast and asks you to retire it with kafka-rm.yml first.

Do not delete meta.properties, the manifest, or the secrets to bypass these protections.

Static Fingerprint and Recoverable Re-runs

The role computes an expected fingerprint over the static files that affect the Kafka process, and writes /etc/kafka/.pigsty-applied-static.sha256 only after one of the following holds:

• Converge has successfully started and passed the global health check;
• The strict rolling restart has restarted this node, let it catch up, and passed the post-restart gates.

If the run is interrupted, changes that have not been proven to take effect are not recorded as “applied.” The next full re-run can still recognize the pending static restart.

Resource Convergence and Monitoring Registration

Once the cluster is fully healthy, the resource-convergence and monitoring stages run in order:

1. Converge the role-owned dynamic cluster min-ISR;
2. Idempotently process the credentials, ACLs, and declared quotas of kafka_users;
3. Idempotently process the creation, partition growth, and explicit config of kafka_topics;
4. Check internal-topic RF drift, but do not reassign automatically;
5. Configure and start the protocol exporter on the first two broker-capable nodes, ordered by kafka_seq;
6. Refresh the file-discovery targets on all infra nodes.

Each instance maps to one target file, and both the JMX target and the (selected nodes’) protocol-exporter target live under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

The target files are refreshed on every full run to match the current exporter placement; target deletion is handled by the deregistration step of kafka-rm.yml.

Protected Rotation

The rotation variables are one-shot extra-vars and should not be written into pigsty.yml. The two actions are mutually exclusive and only one may run at a time; the prerequisites are that all members are formatted, the cluster is healthy, the security profile is scram, the role-owned secret material exists, and kafka_rotate_confirm matches the cluster name exactly.

Internal Credential Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_credentials=true \
  -e kafka_rotate_confirm=kf-main

The role uses active/standby internal identities: it first updates the inactive credential through the live admin channel, then atomically switches the local protected record, and enters the normal strict rolling restart. The old active identity is kept as the next round’s standby, so a re-run after an interruption is recoverable.

Certificate Rotation

./kafka.yml --check -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

./kafka.yml -l kf-main \
  -e kafka_rotate_certificates=true \
  -e kafka_rotate_confirm=kf-main

The role discards the node certificates already issued in the shared PKI tree, re-issues a private key and certificate for each node from the same Pigsty CA, updates the PEM certificate bundle on the nodes, and enters the strict rolling restart. Because the old and new certificates are issued by the same CA and trust each other, no staged trust swap is needed; if the health precheck fails, the rotation does not begin and the existing certificates on the nodes are left unchanged.

kafka-rm.yml

Removal is not in kafka.yml; it uses the separate kafka-rm.yml playbook. Selecting all members of a cluster with -l is a teardown, selecting a strict subset is member retirement; both share the same execution order:

Deregister the VictoriaMetrics targets (kafka_deregister) → stop and disable the kafka/kafka_exporter services (kafka) → remove the KRaft voter entry and broker registration through a surviving member (kafka_retire, which only has a surviving member to work through when a strict subset is selected) → delete exporter config, Systemd environment/units, and helper scripts (kafka_config) → delete the data directories and node-local /etc/kafka recovery state (kafka_data, controlled by kafka_rm_data) → optionally uninstall the packages (kafka_pkg, controlled by kafka_rm_pkg).

The safeguard switch is kafka_safeguard: when set to true (on the command line or in the inventory), the playbook aborts immediately and deletes nothing. An identity conflict, an exporter anomaly, or an ordinary startup failure is not a reason to delete data — converge with kafka.yml first and read the failure reason.

Cluster Teardown

./kafka-rm.yml -l kf-main                          # Remove the cluster: deregister monitoring and stop services; deletes data and /etc/kafka recovery state by default
./kafka-rm.yml -l kf-main -e kafka_rm_data=false   # Keep data and /etc/kafka recovery state; remove only service integration
./kafka-rm.yml -l kf-main -e kafka_rm_pkg=true     # Also uninstall the kafka-stack packages (the shared Java runtime is not removed)

Member Retirement

./kafka-rm.yml -l 10.10.10.13                      # Retire one member: drop its voter entry and broker registration, then clean the node

The playbook removes the leaving node’s KRaft voter entry through a surviving member (remove-controller, strictly serialized for several members) and drops its broker registration (unregister) before the local cleanup. Every metadata action is delegated to the survivor, so this also works for nodes that are already dead and unreachable — this is step one of Replace Failed Node.

Automated retirement does not remove the need for planning: after the shrink, the remaining controllers should stay odd-numbered and keep a live majority, and the remaining broker count must not fall below the highest topic RF; when the retiring broker still hosts partition replicas, the playbook prints a warning — a planned shrink should drain with a reassignment first.

Playbook Boundaries

Neither playbook performs partition reassignment and data balancing, topic/user deletion, online plaintext → scram migration, version upgrades and feature-level finalization, or data backup and disaster recovery, and neither deploys ecosystem components such as Connect, Schema Registry, MirrorMaker, or Cruise Control. For the full list see Module Boundaries; for day-to-day read-only checks and resource management, see Administration.

6 - Monitoring

Pigsty gives the KAFKA module a unified observability stack that combines metrics, logs, dashboards, and alerts. Monitoring covers both the Kafka JVM internals and the Kafka protocol view, so you never end up seeing only that the process is alive without visibility into partitions, ISR, and consumer lag, nor seeing only cluster metadata without visibility into the JVM, request queues, and KRaft controller health.

Scrape Architecture

The KAFKA module uses two complementary exporters:

On each Infra node, the role generates one file-discovery target per instance. The JMX target and the protocol exporter target (on the selected nodes) both live in the same file, under the same kafka scrape job:

/infra/targets/kafka/<kafka_instance>.yml

A single-broker cluster runs only one protocol exporter; a multi-broker cluster runs at most two. Controller-only nodes register only the JMX target; brokers that were not selected and controller-only nodes have no protocol exporter target, which is expected behavior. The target file is refreshed to match the current placement on every full run; deletion of an instance target is handled by the deregistration step in kafka-rm.yml.

Label Model

Both target types are registered under the same job=kafka scrape job, and are distinguished by the presence or absence of the role label.

JMX Target

Protocol Exporter Target

A protocol exporter target carries only cls, ins, ip, and instance (10.10.10.11:9308); it has no role or node_id labels. The recording rules on the vmagent side use this to distinguish the two availability types: kafka_up is up{job="kafka",role=~".+"}, and kafka_exporter_up is up{job="kafka",role=""}.

The exporter queries the entire Kafka cluster through a broker, so the two exporters of the same cluster may return an identical view of topics, partitions, and consumer groups. The cluster-level recording rules first deduplicate across exporter instances, then aggregate the logical cluster rates. In scram mode, the TLS/SCRAM parameters the exporter needs to connect to Kafka are generated automatically from the role’s own monitoring identity.

Grafana Dashboards

Pigsty ships four complementary dashboards:

Kafka Overview

Cluster and global overview. cls=All is the overview across all Kafka clusters; once you select a specific cls, the same dashboard becomes the overview for that Kafka cluster, rather than a separate set of panels.

Main content:

• Inventory of clusters, brokers, topics, partitions, and consumer groups
• Broker availability, exporter health, and cluster workload
• Leaderless, under-replicated, ISR deficit, and non-preferred replica
• Topic offset progress, consumer commit progress, and total lag
• Consumer group members, lag ranking, and topic/group drill-down
• Kafka/exporter log volume, firing alerts, and log detail

Common variables: cls, members, topic, group, topk.

Kafka Instance

Use the ins variable to select any Kafka broker/controller JVM, including controller-only nodes, correlated with host resources.

Main content:

• Instance identity, role, JMX availability, and scrape quality
• JVM heap, GC, threads, buffer pool, CPU, FD, and uptime
• Broker throughput, replication state, request errors/latency/queues, and handler/network idle
• KRaft member state, metadata log, controller health, and event latency
• Node CPU/memory, disk I/O, network, filesystem, and Kafka logs

Common variables: cls, ins, ip.

Kafka Topic

Use cls and topic to select a logical topic and inspect topic/partition state from the protocol view.

Main content:

• Topic and partition inventory, leader, replicas, ISR, and preferred leader
• Current offset, retention span, and message append rate
• Leaderless, ISR deficit, and non-preferred replica
• Associated consumer groups, commit progress, and lag

Common variables: cls, topic, topk.

Kafka Consumer

Use cls and group to select a consumer group and inspect members, committed offsets, consumption progress, and backlog.

Main content:

• Consumer group inventory and member count
• Committed offsets by group/topic/partition
• Commit rate, total lag, maximum partition lag, and backlog trend
• Drill-down from group to topic/partition

Common variables: cls, group, topic, topk.

Choosing a Dashboard

Recording Rules

The Kafka rule file lives at /infra/rules/kafka.yml. The main recorded metrics are:

Rates derived from offset changes represent progress, not client request counts. Log truncation, offset rollback, or an exporter restart can produce a transient negative change; the rules use clamp_min(..., 0) to keep only forward progress.

Alert Rules

An unclean leader election can mean data loss. Immediately preserve the controller/broker logs, confirm the affected topics and replicas, and only then decide on a recovery action.

Common PromQL

Check scrape targets:

kafka_up
kafka_exporter_up
up{job="kafka"}

Check the replication health of a cluster:

sum by (cls) (kafka_server_replica_manager_under_replicated_partitions{job="kafka"})
sum by (cls) (kafka_server_replica_manager_under_min_isr_partitions{job="kafka"})
max by (cls) (kafka_controller_offline_partition_count{job="kafka"})

Check consumer lag:

topk(20, kafka_consumergroup_lag_sum{cls="kf-main"})

Check request saturation and latency:

kafka_server_request_handler_idle_ratio{job="kafka",cls="kf-main"}
max by (ins,request,quantile) (
  kafka_network_request_total_time_seconds{job="kafka",cls="kf-main",quantile=~"0.95|0.99"}
)

Log Queries

Kafka services write stdout and stderr to Journald; the node’s Vector Journald source forwards them to VictoriaLogs, all under job:syslog.

job:syslog unit:kafka
job:syslog app:kafka
job:syslog unit:kafka_exporter
ip:10.10.10.11 job:syslog (unit:kafka OR app:kafka)

The log panel on the Kafka Instance dashboard uses similar queries and shows time, level, systemd unit, and message. When diagnosing, align the logs with the KRaft, ISR, request queue, GC, disk I/O, and network metrics from the same time window.

Verifying the Monitoring Chain

Verify the raw endpoints on a Kafka node:

curl -fsS http://<kafka-ip>:9404/metrics | grep '^jmx_scrape_error'
curl -fsS http://127.0.0.1:9308/metrics | grep '^kafka_brokers'

Check file discovery on an Infra node (one file per instance; the file for a selected node contains both the JMX and the protocol exporter targets):

ls -l /infra/targets/kafka/
cat /infra/targets/kafka/kf-main-1.yml

Then query up{job="kafka"} in VictoriaMetrics (or the recorded metrics kafka_up and kafka_exporter_up). After a failed scrape, custom exporter metrics may briefly retain stale samples, so endpoint liveness should be judged by Prometheus’s native up. If the raw endpoints are fine but the recorded metrics are missing, check file discovery, the VictoriaMetrics target, network reachability, rule loading, and labels, in that order. If the JMX HTTP endpoint is fine but jmx_scrape_error is 1, check the Kafka logs and the MBean matching in /etc/kafka/jmx_exporter.yml.

For complete metric semantics, see Metric Definitions.

7 - Metrics

The KAFKA module uses two kinds of metric sources, both registered under the same job=kafka scrape job: the JMX target (with a role label) collects the internal state of each JVM, while the protocol exporter target (without a role label) collects the state of the logical cluster, topics, partitions, and consumer groups over the Kafka protocol. The protocol exporter is placed only on the one or two broker-capable nodes with the smallest kafka_seq, and a single-broker cluster runs just one.

The JMX configuration is an allow-list: it exports only the JVM baseline and a bounded set of broker, replication, request-path, and KRaft metrics. High-cardinality per-client and per-partition JMX MBeans are deliberately excluded; partition detail is supplied by the protocol exporter.

Common Labels

The job for both target types is kafka; whether a series carries the role label is what distinguishes the two.

Some metrics also carry dimensions such as topic, partition, broker, consumergroup, request, version, error, quantile, state, or operation.

Availability and Scrape Metrics

Protocol Exporter Metrics

The following metrics come from the protocol exporter target. Multiple exporters of the same cluster see the same logical cluster state, so any direct cluster aggregation must deduplicate by semantics rather than simply summing across all ins.

Broker and Topic

current_offset - oldest_offset estimates the currently retained offset span, but an offset count is not a byte count, and for a compacted topic it is not an exact message count either.

Consumer Group

Ephemeral consumers that never commit an offset, clients that use external offset storage, and groups that have not yet consumed a topic will not necessarily produce these time series.

Exporter Itself

JMX: JVM Baseline

excludeJvmMetrics: false makes the JMX Exporter expose the standard JVM/process metrics. The Kafka Instance dashboard mainly uses:

JMX: Broker Traffic

These are broker-wide totals with no topic dimension, which keeps the JMX series count from growing with the number of topics. Topic-level offsets and progress come from the protocol exporter.

JMX: Replication and Storage

Under Replicated means the replicas are not all in sync. Under Min ISR is more serious: the write-availability or durability condition has fallen below the configured minimum ISR. At Min ISR has not crossed the line yet, but there is no remaining replica headroom.

JMX: Request Path

When investigating high latency, look at request volume, error codes, P95/P99, both queues, handler/processor idle, GC, CPU, disk I/O, and network together. A low idle ratio alone is not enough to pinpoint where the bottleneck is.

JMX: KRaft and Broker Metadata

log_end_offset - high_watermark helps gauge metadata commit lag; also factor in the member role, current leader, epoch, and controller event latency.

JMX: Controller

These MBeans exist only in Kafka processes that carry the controller role:

A healthy cluster should have exactly one active controller. Any increase in offline_partition_count, metadata_errors_total, or unclean_leader_elections_total should be treated as a priority.

Recording Rule Metrics

Offset Progress

JVM and Broker

Cardinality and Interpretation Notes

• Do not directly sum multiple kafka_exporter results for the same cls; they may be duplicate views of the same cluster.
• kafka_topic_partition_current_offset is an offset, not an exact count of bytes, requests, or business events.
• Consumer lag covers only groups that are visible in Kafka and have committed an offset.
• A controller-only node lacking broker metrics and protocol exporter metrics is a normal role difference; a broker that was not selected lacking protocol exporter metrics is a normal result of placement.
• When a given MBean does not exist in a specific Kafka version/role, the corresponding JMX series will not appear either; interpret this together with role.
• Per-client/per-partition JMX metrics are excluded by the allow-list to avoid unpredictable time-series cardinality.

For how to use the dashboards and alerts, see Monitoring.

8 - FAQ

How mature is the current KAFKA module?

The current role implements a production-grade v1 baseline: dynamic KRaft, full cluster guardrails, cold-start/repair, serial broker admission and dynamic controller join, member retirement (including dead nodes), three-command failed-node replacement, strict rolling restart, TLS/SCRAM/ACL, declarative convergence of topics/users, internal credential and certificate rotation, and the full monitoring pipeline.

It is not a managed Kafka product. Production still requires kafka_security: scram, an odd number of controllers, sufficient brokers/RF/minISR, plus your own capacity planning, reassignment/data balancing, upgrade, backup, restore, and failure drills. The default plaintext is only suitable for development or a trusted, isolated network.

Why is there no ZooKeeper and no controller.quorum.voters?

This module targets Kafka 4.1+ and uses native dynamic KRaft, with no ZooKeeper installed and no static quorum created. All members render controller.quorum.bootstrap.servers; new clusters are formatted explicitly with --initial-controllers/--no-initial-controllers, and after startup the role verifies that the directory IDs of the initial controllers have joined the live quorum.

The initial controller identity is written into the bootstrap manifest, but it is only a birth certificate: after the cluster’s first commission, live quorum membership is authoritative in Raft itself. Later controller additions and removals are orchestrated by the playbooks — additions go through kafka.yml’s observer catch-up + add-controller join flow, removals through a kafka-rm.yml strict-subset retirement (automatic remove-controller) — you only edit the inventory and run the matching playbook.

What is the difference between combined, broker, and controller?

• combined: acts as both broker and controller, listening on 9092 and 9093; this is the default;
• broker: pure data plane, listening only on 9092;
• controller: pure control plane, listening only on 9093.

The cluster roles must either be omitted entirely and consistently use combined, or be declared explicitly for every member. The old role aliases are no longer provided.

Will controller port 9093 collide with Alertmanager?

No. Pigsty’s Alertmanager listens on alertmanager_port 9059 with cluster port 9094, clear of the KRaft controller’s conventional port 9093. If you changed those ports and created a clash, adjust kafka_controller_port for that cluster — the role only enforces that the four Kafka ports 9092, 9093, 9308, and 9404 differ from one another, and does not detect port conflicts with other services.

The service is up, but a remote client cannot connect?

A broker’s advertised.listeners always uses inventory_hostname. After connecting to the bootstrap server, a client must also resolve and reach every broker address returned in the metadata.

Check in order:

grep '^advertised.listeners' /etc/kafka/server.properties
ss -lntp | grep ':9092'
getent hosts <inventory-hostname>

A scram client must additionally check the CA, SASL mechanism, username/password, and ACLs. The current v1 does not offer custom advertised addresses, multiple listeners, or NAT/public mapping; if a client cannot route directly to inventory_hostname, that network model is outside the current core contract and cannot be worked around by overriding the raw listener via kafka_parameters.

Why does it report a Cluster ID, Node ID, or Directory ID mismatch?

The role cross-checks the bootstrap manifest, ${kafka_data}/metadata/meta.properties, the inventory, and the live dynamic quorum. Common causes include:

• kafka_cluster or kafka_seq was changed;
• another cluster’s data disk was mounted on the current node;
• an incorrect kafka_cluster_id was given during restore/takeover;
• the controller data directory or directory ID does not match the live voter records;
• the wrong target cluster was selected, or a stale manifest was used.

This is a protective failure. Do not delete meta.properties, the manifest, or run kafka-rm.yml. First confirm data ownership, the remaining replicas, the true Cluster/Node/Directory identity, and the recovery target.

What happens if the manifest is lost or only an old one remains?

Every cluster member keeps an authoritative copy of the manifest at /etc/kafka/manifest.yml (a scram cluster also has /etc/kafka/secrets.yml). The admin node keeps no kafka state and resolves both from any member copy on every run, so replacing the admin node or losing the local checkout does not affect cluster management. Only when all member copies are lost while the storage has already been formatted does the role fail closed and prompt you to restore the file on any member first; a formatted scram cluster likewise fails closed when no member holds the secret material. Issued node certificates are cached under files/pki/kafka/ and are simply re-signed from the Pigsty CA when absent.

Conversely, if the manifest exists but all Kafka data disks are empty, the role fails closed to avoid accidentally reviving a vanished cluster under an old identity. If you genuinely intend to rebuild, you must first run kafka-rm.yml and follow an explicit rebuild procedure.

Why are some keys in kafka_parameters rejected?

Identity, the dynamic quorum, listeners, storage, replication, rack, and security must have a single source of authority, so those keys are owned by the role: if any one of them appears, the identity precheck fails before anything is written. For the complete reserved list, see kafka_parameters.

Use the corresponding public parameters instead. The role provides no variables for advertised addresses, path subdirectories, the listener map, or exporter options.

How do I enable TLS, SCRAM, and ACLs?

Set on a new cluster:

kafka_security: scram

This enables the Pigsty CA node certificates, controller mTLS, broker/client SASL_SSL + SCRAM-SHA-512, StandardAuthorizer, and default-deny all at once. Application users declare their passwords, ACLs, and optional quotas via kafka_users.

The security mode is a bootstrap-only property. A formatted cluster cannot switch online from plaintext to scram via ordinary playbooks; that requires a separate migration state machine. A healthy scram cluster can rotate internal credentials or certificates through protected actions.

Do kafka_topics and kafka_users delete resources?

No. Removing an entry from the inventory never implicitly deletes a topic or a user.

Topics are created idempotently, partitions only increase, and only the declared configs are updated; an RF change requires an explicit reassignment. A declared user has its password, complete ACL set, and the given quota fields converged. Topic deletion, user deletion, and full privilege revocation are all separate, audited operations.

What is the difference between the JMX Exporter and kafka_exporter?

The JMX Exporter is injected into every Kafka JVM and collects JVM, broker, replication, request-path, and KRaft internal metrics, registered as a job=kafka target with a role label.

kafka_exporter queries the logical cluster, topics, partitions, offsets, consumer groups, and lag over the Kafka protocol, registered as a target under the same job=kafka but without a role label. The role runs it only on the first two broker-capable nodes ordered by kafka_seq; a single-broker cluster runs one, and pure controllers run none.

The two are complementary. The lifecycle health gate uses the role’s own Kafka CLI/metadata channel and does not depend on either exporter.

Why does a particular broker or pure controller have no kafka_exporter?

This is the expected derived placement. The protocol exporter returns a view of the entire logical cluster, not node metrics; capping it at two replicas avoids a monitoring single point of failure while keeping the cost of duplicate scraping in check.

Check the current targets (one file per instance; the files of selected nodes contain the :9308 protocol exporter target):

ls -l /infra/targets/kafka/
grep 9308 /infra/targets/kafka/*.yml

A full run refreshes each instance’s target file according to the current placement, so you should not run against a single node just to register labels. Note: if the exporter placement moves due to a topology change, the old kafka_exporter service on a formerly selected node is not stopped automatically by ordinary playbooks and must be cleaned up manually or via kafka-rm.yml.

Why is the JMX endpoint reachable but jmx_scrape_error=1?

HTTP reachability only means the Java agent is loaded; jmx_scrape_error=1 means the MBean scrape failed this round:

journalctl -u kafka --since '-30 min' --no-pager
curl -fsS http://<kafka-ip>:9404/metrics | head -n 40

Check whether /etc/kafka/jmx_exporter.yml matches the currently installed Kafka/JMX Exporter packages, and whether the JVM has passed startDelaySeconds. Real startup acceptance requires jmx_scrape_error 0.0, JVM metrics, and at least one kafka_ metric matching the role.

Why is there no Consumer Lag data?

Common causes: the consumer does not use a group, does not commit offsets to Kafka, stores offsets in an external system, the group has not yet consumed the target topic, or the protocol exporter has a TLS/SCRAM/ACL/network problem.

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server <broker>:9092 \
  --command-config /etc/kafka/admin.properties \
  --describe --group <group>

Then check kafka_exporter_up, the exporter logs, the dashboard variables, and the raw kafka_consumergroup_* metrics. Endpoint liveness is judged by Prometheus’s native up; do not substitute a custom metric that may briefly linger after a scrape failure.

Why can’t the cluster metrics from two kafka_exporters be added together?

Both exporters query the same logical cluster and may return the same topic/partition/consumer-group state; summing them directly double-counts. Pigsty’s kafka:cls:* recording rules first deduplicate across the exporter replicas, then aggregate to the cluster.

Should applications go through HAProxy, a Keepalived VIP, or an LB?

No. Kafka producers and consumers are cluster-aware clients: once they reach any seed in bootstrap.servers and fetch metadata, they connect directly to each partition leader. A VIP or generic TCP LB neither understands partition leaders nor rewrites the broker addresses in metadata; putting one in the data plane only adds long-connection state, an extra point of failure, and troubleshooting complexity.

If a platform mandates a single discovery entry point, DNS or a TCP LB may serve bootstrap only, but advertised.listeners still returns a client-reachable address for each broker, and the application network must reach every broker. Exposure across NAT, the public internet, multiple networks, or Kubernetes requires a dedicated external address and an additional listener per broker; the current module always advertises the inventory address and does not support such mapping.

See Quickstart: why applications should connect directly to multiple brokers and Cluster Config: network and listeners.

Can I just add or remove a broker or controller?

Yes. Edit the inventory and let the playbooks orchestrate every step of the KRaft membership change:

• Add: declare the new member in the inventory (broker, combined, and controller all work) and run ./kafka.yml -l <cls> against the complete cluster (you cannot limit the run to the new node only). Pure brokers are formatted, started, and verified as registered one at a time; combined/controller nodes are formatted with --no-initial-controllers, catch up as observers, then get promoted with add-controller. One node at a time, with health gates throughout.
• Remove: ./kafka-rm.yml -l <ip> (a strict subset of the cluster) performs remove-controller and the broker unregistration through a surviving member — it works even when the node is unreachable — then delete the member from the inventory.

You still own the planning: keep the controller count odd with a live majority after the change, make one membership change at a time, and drain the partitions off a removed broker first (or let a same-kafka_seq replacement take them over). After a node joins, existing partitions are not migrated automatically; you must run and monitor reassignment separately — “the broker is registered” does not mean “capacity is already balanced.”

Which parameter controls the package version?

The role uses package_map['java-runtime'] and package_map['kafka-stack']; there is no kafka_version, scala_version, or exporter version parameter. The actual versions are determined by the Pigsty repository for the target platform and the installed packages.

The payload verified on 2026-07-16 is Kafka 4.3.1, kafka_exporter 1.9.0, and JMX Exporter 1.6.0. An upgrade still requires a separate review of compatibility, backup/rollback, rolling order, and feature level; you cannot simply swap the packages.

How do I safely wipe Kafka data?

kafka.yml never performs cleanup; deletion lives only in the separate kafka-rm.yml playbook. Selecting a whole cluster with -l (or running bare for all clusters) is a teardown; selecting a strict subset is member retirement. By default kafka_rm_data=true permanently deletes the data/KRaft metadata, node-local /etc/kafka recovery state, and monitoring targets; kafka_rm_data=false keeps the data and recovery state, and kafka_safeguard=true aborts any deletion.

The playbook has no extra gate such as a confirmation string, so before running it you must manually confirm the exact -l target, a recoverable backup or a clear intent to rebuild, and the business-decommissioned status. For the full semantics, see Playbook: kafka-rm.yml.