Path 10 · lesson 4 of 6
Kafka security: TLS, SASL and ACLs
How a connection is encrypted, how a client proves who it is, and how ACLs decide what that principal may do, with the exceptions each failure produces.
What you will learn
- Read a broker's listener configuration and say which traffic is encrypted and which is authenticated
- Choose between mutual TLS, SCRAM, OAUTHBEARER, PLAIN and Kerberos for a given environment
- Write least-privilege ACLs for a producer service and a consumer service, including the group and transactional id
- Configure a Spring Boot client for SASL_SSL with SCRAM and keep the secrets out of the repository
- Map
SslAuthenticationException,SaslAuthenticationException,TopicAuthorizationExceptionandGroupAuthorizationExceptionto their causes - Plan certificate and credential rotation so that expiry is never the way you find out
Before you start
The concept
A Kafka cluster with default settings accepts any connection, in clear text, and lets it do anything: read every topic, delete topics, change configuration. The Kafka documentation says so directly: security is optional. Securing a cluster means answering three separate questions for every connection, and Kafka has a separate mechanism for each.
| Question | Mechanism | Where it is configured |
|---|---|---|
| Can anyone on the network read or alter the bytes? | Encryption with TLS (Kafka's configuration still calls it SSL) | The security protocol of the listener |
| Who is connecting? | Authentication: a TLS client certificate (mutual TLS) or a SASL mechanism | The listener, plus credentials on the client |
| Is this identity allowed to do this? | Authorisation: ACLs evaluated by an authorizer | authorizer.class.name on every broker and controller, and the ACLs themselves |
The first two are properties of a listener. A broker can listen on several ports, each with a name and one of four security protocols. The protocol name tells you two things at once: whether the bytes are encrypted, and whether SASL is used to authenticate.
| Security protocol | Encrypted | How the client is identified |
|---|---|---|
PLAINTEXT | No | Not at all. Every client is the anonymous principal |
SSL | Yes | By a client certificate if the broker sets ssl.client.auth=required (mutual TLS). Otherwise anonymous |
SASL_PLAINTEXT | No | By SASL, over an unencrypted connection. Not for production: with PLAIN the password crosses the network readable |
SASL_SSL | Yes | By SASL, inside a TLS connection. The usual choice for application clients |
Authentication produces a principal, written User:name. With SASL it is the user name. With mutual TLS it is, by default, the full distinguished name of the certificate, such as User:CN=payment-service,OU=Payments,O=Example, which ssl.principal.mapping.rules can shorten. Everything after that point, ACLs and quotas alike, refers to the principal and never to a password or a certificate.
An ACL is one rule of the form: principal P is allowed (or denied) operation O from host H on resources matching pattern R. The resource has a type (Topic, Group, Cluster, TransactionalId, DelegationToken, User), a name, and a pattern type: literal for an exact name (or * for all), prefixed for every name that starts with the given text.
Why it exists
Kafka started as an internal pipeline on a trusted network, and for years the network boundary was the only protection. That stops working as soon as a cluster is shared. A topic of payment events and a topic of page views sit on the same brokers. Any service that can reach port 9092 can subscribe to either, and a test tool pointed at the wrong bootstrap address can write garbage into a production topic or delete it.
- Confidentiality: without TLS, anyone who can capture traffic between a client and a broker, or between brokers, reads every record.
- Accountability: without authentication, the broker cannot tell the Payment Service from a laptop. There is nothing to attach a permission or a quota to.
- Containment: without ACLs, one compromised or misconfigured service has the whole cluster. With least-privilege ACLs it has its own topics and its own group.
- Protection from accidents: most ACL denials in practice stop mistakes, not attackers. The wrong environment, a consumer joining another team's group and stealing its partitions, a script that deletes topics by pattern.
The cost is operational. Certificates expire, passwords must be rotated, and every new topic, group id or transactional id needs an ACL before the code that uses it is deployed. Most security incidents on a Kafka cluster are not break-ins. They are outages caused by the security configuration itself: an expired certificate, a renamed group without an ACL, a rotated password that one service did not pick up. That is why this lesson spends as much time on failure symptoms as on setup.
How it works inside
Listeners in a KRaft cluster
listeners gives each listener a name, host and port. listener.security.protocol.map assigns a security protocol to each name. inter.broker.listener.name chooses the listener brokers use for replication, and controller.listener.names the listener of the KRaft controllers, which may not be the same as the inter-broker one. A broker-only node must still define the controller listener's security settings, because it connects to the controllers as a client.
Give clients their own listener, so that replication and controller traffic can be isolated at the network level. Application clients never talk to the controller listener: admin requests such as creating a topic go to a broker, which forwards them to the active controller together with the client's principal, and the controller authorises them. This is why the authorizer must be configured on controllers as well as brokers, and why the controller listener needs the same care as the others. Whoever can authenticate there as a broker can change cluster metadata.
What TLS checks, and when
During the TLS handshake the broker presents its certificate. The client checks that it chains to a certificate authority in its truststore, that it is within its validity dates, and, because ssl.endpoint.identification.algorithm defaults to https, that the host name it connected to appears in the certificate, preferably as a Subject Alternative Name. With ssl.client.auth=required the client presents a certificate from its keystore and the broker does the same checks against its own truststore. The value requested is discouraged by the documentation: clients without a certificate still connect, which looks like security and is not.
These checks happen at the handshake only. An established connection keeps working after the certificate behind it expires. This shapes how a certificate expiry looks in production: nothing breaks at the expiry instant, and failures appear one connection at a time as clients reconnect, restart or open a connection to a broker they had not used yet.
SASL mechanisms
| Mechanism | How it proves identity | Fits when |
|---|---|---|
PLAIN | User name and password sent to the broker. Must only be used over TLS. The built-in implementation keeps passwords in the broker's JAAS configuration; production use needs a callback handler that checks an external store | A managed service issues API keys, or passwords live in an external directory |
SCRAM-SHA-256 / SCRAM-SHA-512 | Challenge and response: the password itself is not sent. Kafka stores salted credentials in the KRaft metadata log, created with kafka-configs.sh or kafka-storage.sh. Use with TLS | A self-managed cluster that wants user names and passwords without extra infrastructure |
OAUTHBEARER | A token from an OAuth 2 identity provider, validated by the broker against the provider's keys. The client refreshes it before expiry. The default unsecured-token implementation is for development only | The organisation already issues short-lived tokens to services |
GSSAPI | Kerberos tickets | An existing Kerberos or Active Directory environment |
Mutual TLS is the alternative to SASL: the certificate is the identity. It needs no password store, at the price of issuing and rotating a certificate per service. By default a SASL session is authenticated once, when the connection opens. Setting connections.max.reauth.ms on the broker makes clients re-authenticate periodically, so that a deleted credential or an expired token stops working on existing connections too.
How the authorizer decides
In a KRaft cluster the built-in authorizer is org.apache.kafka.metadata.authorizer.StandardAuthorizer, set through authorizer.class.name on every node. It stores ACLs in the metadata log, so they replicate like any other metadata. For each request it applies these rules in order.
- If the principal is listed in
super.users, allow. Super users bypass all ACLs, so keep the list to the brokers' own principal and a small set of administrators. - If any matching ACL denies the operation, deny, however many ACLs allow it.
- If a matching ACL allows the operation, allow. An allow for Read, Write, Delete or Alter also implies Describe on that resource.
- If no ACL matches the resource at all, deny, unless
allow.everyone.if.no.acl.found=true, in which case allow.
What each client needs
| Client | Resource | Operations | Why |
|---|---|---|---|
| Producer | Topic | Write, Describe | Produce requests need Write. Metadata lookups need Describe, which Write implies |
| Idempotent producer | Topic | Write | Since Kafka 2.8 Write on the topic is enough. Older brokers demanded IdempotentWrite on the Cluster resource |
| Transactional producer | TransactionalId | Write, Describe | Finding the transaction coordinator needs Describe, starting and ending transactions need Write. Add Read on the Group if it commits consumer offsets in the transaction |
| Consumer | Topic | Read, Describe | Fetching needs Read on the topic, and so does committing an offset for it |
| Consumer | Group | Read | Joining the group, heartbeating and committing offsets all check Read on the group. Group access is checked before topic access |
| Consumer with a dead letter topic | Dead letter topic | Write | The consumer is a producer for that topic |
The table explains the most common surprise. A service that can read a topic perfectly well with a console tool still cannot consume in production if its group has no ACL, because the group is a resource of its own. kafka-acls.sh has --producer and --consumer shortcuts that generate these sets; note that --producer also grants Create on the topic, which you may not want.
Secrets, rotation and quotas
- Secrets: passwords and private keys belong in a secret manager and reach the application as environment variables or mounted files. Since Kafka 2.7, keys and certificates can also be given in PEM form directly in the configuration, which suits secrets injected at runtime.
- Broker certificates: a broker's keystore can be replaced without a restart, per listener, with
kafka-configs.sh. The new certificate must be signed by the same certificate authority as the old one, or clients will reject it. - Changing the certificate authority: add the new authority to every truststore first, then issue certificates from it, then remove the old authority. Truststores can also be updated dynamically on brokers.
- SCRAM passwords: changing a password with
kafka-configs.shtakes effect for new connections. To rotate without an outage, create a second user with the same ACLs, move the service to it, then delete the first. - Quotas: once every client has a principal,
kafka-configs.shcan capproducer_byte_rate,consumer_byte_rateandrequest_percentageper user. By default clients have no limit. A quota throttles a client that misbehaves, by accident or on purpose, before it starves the others.
Architecture
Drawing diagram…
A flowchart in which a client connects to the CLIENT listener, passes or fails the TLS handshake, passes or fails SCRAM authentication, becomes a principal, and is then checked against super users, deny ACLs and allow ACLs. Separate BROKER and CONTROLLER listeners carry replication and metadata traffic.
flowchart TD
APP["Payment Service, user payment-service"] --> CL["Listener CLIENT, SASL_SSL, port 9093"]
CL --> TLS{"TLS handshake passes?"}
TLS -- "No" --> E1["SslAuthenticationException"]
TLS -- "Yes" --> SASL{"SCRAM credentials valid?"}
SASL -- "No" --> E2["SaslAuthenticationException"]
SASL -- "Yes" --> PR["Principal User payment-service"]
PR --> SU{"In super.users?"}
SU -- "Yes" --> OK["Request handled"]
SU -- "No" --> DENY{"Matching deny ACL?"}
DENY -- "Yes" --> E3["Topic or Group AuthorizationException"]
DENY -- "No" --> ALLOW{"Matching allow ACL?"}
ALLOW -- "Yes" --> OK
ALLOW -- "No" --> E3
BR["Other brokers"] --> BL["Listener BROKER, SASL_SSL, port 9094"]
BL --> REP["Replication"]
BR --> CTL["Listener CONTROLLER, SASL_SSL, port 9095"]
CTL --> META["KRaft metadata log, holds ACLs and SCRAM credentials"]- The three gates are in a fixed order and each has its own exception. That makes the exception name the fastest diagnostic: an
SslAuthenticationExceptionmeans the request never reached SASL, and an authorisation exception means both earlier gates passed. - The last decision has no allow by default. With
allow.everyone.if.no.acl.foundat its default offalse, a resource nobody wrote an ACL for is closed to everyone except super users. - ACLs and SCRAM credentials live in the KRaft metadata log, which is why the controller listener is drawn with the same protection as the others. Leaving it on
PLAINTEXTwould expose the store that the other two listeners rely on.
Message flow
Drawing diagram…
A sequence diagram in which a consumer opens a TLS connection, authenticates with SCRAM, and then sends JoinGroup, Fetch and OffsetCommit requests. For each request the broker asks the authorizer whether the principal may read the group or the topic.
sequenceDiagram participant C as Payment Service consumer participant B as Broker, CLIENT listener participant A as StandardAuthorizer C->>B: TCP connect to kafka-1 port 9093 B-->>C: TLS certificate for kafka-1 Note over C: Checks CA, validity dates and host name C->>B: SASL handshake, mechanism SCRAM-SHA-512 C->>B: SCRAM client proof for user payment-service B-->>C: Authenticated Note over B: Connection principal is User payment-service C->>B: JoinGroup for group payment-service B->>A: Read on Group payment-service? A-->>B: Allowed by a prefixed ACL B-->>C: Member of the group C->>B: Fetch orders.placed partition 2 B->>A: Read on Topic orders.placed? A-->>B: Allowed by a literal ACL B-->>C: Records C->>B: OffsetCommit for group payment-service B->>A: Read on Group, then Read on Topic? A-->>B: Allowed B-->>C: Committed
- Authentication happens once, when the connection opens. Authorisation happens on every request, against the current ACLs, so adding or removing an ACL takes effect without the client reconnecting.
- The group is checked before the topic, and joining a group is the first thing a consumer does. A missing group ACL therefore stops a consumer before it has fetched a single record, with
GroupAuthorizationException. - An offset commit checks both the group and the topic. A principal that can read a topic but not commit for the group cannot make progress as a group member.
In a real system
Incident: the consumer that was renamed
A typical order platform. The Order Service publishes to orders.placed (12 partitions, keyed by order id). The Payment Service consumes it in the group payment-service as the SCRAM user payment-service. The cluster runs StandardAuthorizer with ACLs managed from a repository: Read on topic orders.placed and Read on the literal group payment-service.
A developer wants to replay the last day of orders into a new settlement table, and takes the usual shortcut: change the group id to payment-service-v2 so the consumer starts from the earliest offset. The change passes review. Nobody connects a group id in application.yml with an ACL in another repository.
| Time | What happened | What people saw |
|---|---|---|
| 14:05 | The new version rolls out. Each instance authenticates, asks to join payment-service-v2 and is refused | One ERROR per instance with GroupAuthorizationException: Not authorized to access group: payment-service-v2, then silence |
| 14:05 | Spring's listener container treats authorisation errors as fatal by default and stops. The application itself keeps running | Pods are Running and Ready. The health endpoint is green. The deployment is marked successful |
| 14:06 to 14:55 | No instance consumes orders.placed. Orders are placed and never charged | The lag dashboard is filtered on the new group, which never formed, so it shows no data instead of a climbing line |
| 14:55 | Support reports orders stuck in 'awaiting payment' | An engineer finds the single ERROR line in the logs from 50 minutes earlier |
| 15:02 | A prefixed ACL for groups starting with payment-service is applied and the service is restarted | The group forms and works through 50 minutes of backlog |
The broker did exactly what it was configured to do. What turned a correct denial into a 50-minute outage was everything around it: an ACL that matched one exact name, a container that stops quietly, a health check that did not include the listener, and a lag alert that could not tell 'no lag' from 'no group'.
- Grant group access with a prefixed pattern owned by the service, such as
payment-service, so that-v2and-replayare covered. Do the same for dead letter and retry topics. - Keep ACLs in the same change as the configuration that needs them, or check in the pipeline that every topic and group id in the service's configuration has a matching ACL.
- Make a stopped listener container fail the health check, or set
authExceptionRetryIntervalso the container retries and recovers once the ACL exists. - Alert on a group having no active members and on the consumed-records rate falling to zero, not only on lag.
The other classic of this kind is expiry. The Break card Expired for some, fine for others is a real case of a certificate in the chain expiring while the server's own certificate was fine. The exercise below is the Kafka version.
Java and Spring Boot
The broker side of a SASL_SSL listener, the commands that create a SCRAM user and its least-privilege ACLs, and a Spring Boot client configured from environment variables. spring.kafka.security.protocol, spring.kafka.ssl.bundle and spring.kafka.properties.* are Spring Boot properties; everything under properties is passed to the Kafka client unchanged.
process.roles=brokerlisteners=CLIENT://kafka-1.internal.example:9093,BROKER://kafka-1.internal.example:9094inter.broker.listener.name=BROKERcontroller.listener.names=CONTROLLERlistener.security.protocol.map=CLIENT:SASL_SSL,BROKER:SASL_SSL,CONTROLLER:SASL_SSLsasl.enabled.mechanisms=SCRAM-SHA-512sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512config.providers=fileconfig.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProviderssl.keystore.type=PKCS12ssl.keystore.location=/etc/kafka/secrets/kafka-1.keystore.p12ssl.keystore.password=${file:/etc/kafka/secrets/store.properties:keystore.password}ssl.truststore.type=PKCS12ssl.truststore.location=/etc/kafka/secrets/truststore.p12ssl.truststore.password=${file:/etc/kafka/secrets/store.properties:truststore.password}authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizersuper.users=User:kafka-broker;User:kafka-adminallow.everyone.if.no.acl.found=falseWhy it is written this way
- Three named listeners, all
SASL_SSL. The controller listener is in the protocol map but not inlisteners, because this node is a broker only: it connects to the controllers and needs to know how. super.usersis separated by semicolons, because certificate names can contain commas. It holds the principal the brokers use among themselves and one administrator, nothing else.allow.everyone.if.no.acl.found=falseis the default, written out so that nobody has to wonder. A topic without ACLs is closed.- The store passwords are read through a configuration provider from a file outside the main configuration. The JAAS configuration with the brokers' own credentials is omitted here.
- The same
authorizer.class.nameandsuper.usersmust be set on the controller nodes, because forwarded admin requests are authorised there.
#!/usr/bin/env bashset -euo pipefailBOOTSTRAP=kafka-1.internal.example:9093ADMIN="--command-config /etc/kafka/admin.properties" # SASL_SSL settings of User:kafka-admin# 1. SCRAM credentials, stored in the KRaft metadata logbin/kafka-configs.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --alter \ --add-config "SCRAM-SHA-512=[password=$ORDER_SERVICE_PASSWORD]" \ --entity-type users --entity-name order-servicebin/kafka-configs.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --alter \ --add-config "SCRAM-SHA-512=[password=$PAYMENT_SERVICE_PASSWORD]" \ --entity-type users --entity-name payment-service# 2. Producer: may write to orders.placed and nothing elsebin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --add \ --allow-principal User:order-service \ --operation Write --operation Describe --topic orders.placed# 3. Consumer: read the topic, use its own groups, write its dead letter topicbin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --add \ --allow-principal User:payment-service \ --operation Read --operation Describe --topic orders.placedbin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --add \ --allow-principal User:payment-service \ --operation Read --group payment-service --resource-pattern-type prefixedbin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --add \ --allow-principal User:payment-service \ --operation Write --operation Describe --topic orders.placed-dlt# 4. Abuse protection: cap what one principal can push or pull (bytes per second)bin/kafka-configs.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --alter \ --add-config 'producer_byte_rate=10485760,consumer_byte_rate=20971520' \ --entity-type users --entity-name order-service# 5. Review what a principal has, and everything that affects one topicbin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --list --principal User:payment-servicebin/kafka-acls.sh --bootstrap-server "$BOOTSTRAP" $ADMIN --list --topic orders.placed --resource-pattern-type matchWhy it is written this way
- The group ACL is
prefixed: it coverspayment-service,payment-service-v2andpayment-service-replay. A literal ACL is what caused the incident above. - The producer gets Write and Describe only. The
--producershortcut would also grant Create, which lets a typo in a topic name create a topic on clusters with auto-creation enabled. - No
IdempotentWriteACL is needed: on Kafka 2.8 and later an idempotent producer is authorised by Write on the topic. A transactional producer would additionally need Write and Describe on its--transactional-id. --resource-pattern-type matchon--listshows literal, wildcard and prefixed ACLs that affect the topic. A plain--list --topicshows only ACLs on that exact name and hides the prefixed ones.- Passwords come from the environment of the person or pipeline running the script, never from the script itself. The byte rates are example values.
spring: ssl: bundle: pem: kafka: truststore: certificate: ${KAFKA_CA_CERT_PATH:file:/etc/kafka/ca/ca.crt} kafka: bootstrap-servers: ${KAFKA_BOOTSTRAP_SERVERS} security: protocol: SASL_SSL ssl: bundle: kafka properties: sasl.mechanism: SCRAM-SHA-512 sasl.jaas.config: >- org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_USERNAME}" password="${KAFKA_PASSWORD}"; consumer: group-id: payment-service auto-offset-reset: earliest enable-auto-commit: false listener: auth-exception-retry-interval: 30sWhy it is written this way
spring.kafka.security.protocolsets the client'ssecurity.protocol. Without it the client speaksPLAINTEXTto a TLS port and the connection fails without a clear message.- The truststore is a Spring Boot SSL bundle built from the certificate authority's PEM file. The classic alternative is
spring.kafka.ssl.trust-store-locationwithtrust-store-passwordandtrust-store-type. Only the authority's public certificate is needed, so this file is not a secret. sasl.jaas.configmust end with a semicolon and names one login module. The user name and password are placeholders resolved from environment variables that the platform fills from its secret store. Nothing secret is committed.- Host name verification is on by default (
ssl.endpoint.identification.algorithm=https). Leave it. If it fails, fix the certificate's Subject Alternative Names or the bootstrap address, not the check. auth-exception-retry-intervalsets the container propertyauthExceptionRetryInterval. Without it, Spring for Apache Kafka treats authentication and authorisation errors as fatal and stops the listener container while the application keeps running: the silent state in the incident. With it, the container logs the error and tries again, so it recovers once the missing ACL or the corrected credential is in place. It does not replace alerting: the service is still not consuming while it retries.- For mutual TLS in place of SASL, the protocol would be
SSLand the bundle would also carry akeystorewith the service's certificate and private key.
Configuration
The settings that decide whether a connection is encrypted, authenticated and authorised. Broker defaults are from the Kafka 4.4 broker configuration reference; the authorizer settings are from the security chapter.
listener.security.protocol.mapDefault SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
- What it does
- Assigns one of PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL to each named listener.
- When to change it
- Name listeners by purpose (CLIENT, BROKER, CONTROLLER) and map every one explicitly. No production listener should be PLAINTEXT.
security.protocol (client)Default PLAINTEXT
- What it does
- The protocol the client uses. It must match the listener it connects to. In Spring Boot:
spring.kafka.security.protocol. - When to change it
- Set to
SASL_SSL, orSSLfor mutual TLS. A mismatch shows as timeouts and disconnects, not as a clear error.
sasl.mechanism (client)Default GSSAPI
- What it does
- The SASL mechanism the client offers. The broker must have it in
sasl.enabled.mechanisms, whose default is also GSSAPI. - When to change it
- Set on both sides whenever SASL is used without Kerberos, for example
SCRAM-SHA-512.
ssl.client.auth (broker)Default none
- What it does
- Whether the broker asks clients for a certificate:
required,requestedornone. - When to change it
- Use
requiredfor mutual TLS. Avoidrequested: clients without a certificate still get in, as the anonymous principal.
ssl.endpoint.identification.algorithmDefault https
- What it does
- Makes the connecting side check that the host name matches the certificate. An empty value disables the check.
- When to change it
- Keep the default. Disabling it removes the protection against a server presenting someone else's valid certificate.
ssl.enabled.protocolsDefault TLSv1.2,TLSv1.3
- What it does
- The TLS versions accepted. Both sides must share at least one.
- When to change it
- Leave alone unless policy requires TLS 1.3 only, and then check that every client's Java version supports it.
authorizer.class.nameDefault "" (empty: no authorisation)
- What it does
- The class that authorises requests. For KRaft:
org.apache.kafka.metadata.authorizer.StandardAuthorizer. - When to change it
- Set on every broker and controller. Enable it only after the ACLs for existing clients are in place, or use the phased approach below.
allow.everyone.if.no.acl.foundDefault false
- What it does
- When true, a resource that no ACL matches is open to all principals. Resources with any ACL are enforced normally.
- When to change it
- Use
trueonly while migrating a running cluster to ACLs, and switch it back once every resource has rules.
super.usersDefault none
- What it does
- Principals that skip ACL checks, separated by semicolons, for example
User:kafka-broker;User:kafka-admin. - When to change it
- The brokers' own principal must be able to replicate, either as a super user or through ClusterAction on the Cluster resource. Keep human administrators off the list where you can.
connections.max.reauth.msDefault 0 (no re-authentication)
- What it does
- When positive, the broker tells SASL clients a session lifetime and disconnects connections that do not re-authenticate in time.
- When to change it
- Set it when credentials can be revoked or tokens expire, so that revocation reaches connections that are already open.
spring.kafka.listener.auth-exception-retry-intervalDefault none (auth errors stop the container)
- What it does
- How long the listener container waits before retrying after an authentication or authorisation exception.
- When to change it
- Set it, for example to 30 seconds, so a consumer recovers by itself once a missing ACL is added.
Hands-on exercise
Troubleshooting: failures that started on their own at the weekend
Work from symptoms and tool output to the cause of an authentication failure, explain its odd timing, and decide the fix. All output below is illustrative: written for this exercise in the format the tools produce.
Steps · 0 of 6 done
What you should see
Cause: the TLS certificate of kafka-2 expired at 02:00; the other two brokers were renewed in October and kafka-2 was missed. It is the first gate, so SASL credentials and ACLs are not involved. About a third fails because only connections to kafka-2 are affected, which means the partitions it leads. Timing: certificates are checked only during the handshake, so connections opened before 02:00 kept working; failures appeared as clients opened new connections to kafka-2 after idle disconnects or restarts, and an instance started at 03:30 had no old connection to it at all. Fix: issue a new certificate for kafka-2 from the same certificate authority and load it by updating that listener's keystore dynamically with kafka-configs.sh for that broker (or restart only kafka-2). Do not disable certificate or host name validation on clients. Prevention: monitor the notAfter date of every listener on every broker and alert weeks ahead, and renew through automation that covers every node.
What can go wrong
A literal group ACL and a changed group id
- Symptom
- After a deployment a consumer processes nothing. One
GroupAuthorizationExceptionper instance in the log, then no further errors. The application reports healthy. - Cause
- The ACL names the old group exactly. Group access is checked when the consumer joins, and Spring's container stops on authorisation errors by default.
- Fix
- Grant Read on the group with a prefixed pattern owned by the service, deploy ACL changes with or before the code, and set
authExceptionRetryIntervalor fail the health check when a container stops.
Certificate expiry discovered by the clients
- Symptom
- With no change deployed,
SslAuthenticationException: SSL handshake failedappears on some connections and spreads over hours. Restarting a client makes it worse. - Cause
- A broker, client or authority certificate passed its validity date. Existing connections survive until they are reopened, so the failure is gradual.
- Fix
- Renew and reload the keystore (dynamically on brokers). Track expiry dates as a monitored metric with alerts well ahead, and automate renewal for every node.
allow.everyone.if.no.acl.found left on
- Symptom
- A security review finds that any authenticated service can read newly created topics, although ACLs exist for the older ones.
- Cause
- The flag was enabled for the migration to ACLs and never turned off. Every resource without an ACL is public, including each new topic.
- Fix
- Add ACLs for all existing resources, set the flag back to
false, and create ACLs in the same step as topics.
Host name verification switched off to get it working
- Symptom
- Everything connects. Clients carry
ssl.endpoint.identification.algorithmset to an empty string, added during setup and never removed. - Cause
- The broker certificates lacked the right Subject Alternative Names, often because the authority did not copy them from the signing request, and disabling the check was quicker.
- Fix
- Reissue certificates with the brokers' advertised host names as SANs, verify with
openssl x509 -text -noout, and remove the override.
Wildcard ACLs and oversized super user lists
- Symptom
- ACLs exist, but a mistake in one service still reads or deletes another team's topics.
- Cause
- Services were granted
--topic '*'or--operation All, or application principals were put insuper.usersto make errors go away. - Fix
- Grant specific operations on literal or prefixed names per service. Reserve
super.usersfor the brokers and a small administrative principal.
Password rotated in place
- Symptom
- After a SCRAM password change, instances fail with
SaslAuthenticationExceptionas they restart, while long-running ones keep working for a while. - Cause
- The credential was altered before every instance had the new secret. Open connections stay authenticated; new ones use whatever password the instance was started with.
- Fix
- Rotate by adding a second user with identical ACLs, moving the service to it, and deleting the old user afterwards.
How to troubleshoot
SslAuthenticationException: SSL handshake failed
Check
openssl s_client -connect kafka-1.internal.example:9093 </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
- How to read it
- The nested cause tells which check failed. A
CertificateExpiredExceptionor anotAfterin the past means expiry.unable to find valid certification pathmeans the client's truststore lacks the issuing authority.No subject alternative names matchingmeans the host name the client used is not in the certificate. Run the check against every broker: one stale node is enough.
SaslAuthenticationException: Authentication failed ... invalid credentials
Check
kafka-configs.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --describe --entity-type users --entity-name payment-service
- How to read it
- If no SCRAM credential is listed for the mechanism the client uses, the user does not exist for that mechanism (SCRAM-SHA-256 and SCRAM-SHA-512 are separate credentials). If one is listed, the password in the client's secret is wrong or stale. Also compare the client's
sasl.mechanismwith the broker'ssasl.enabled.mechanisms.
TopicAuthorizationException: Not authorized to access topics: [orders.placed]
Check
kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --topic orders.placed --resource-pattern-type match
- How to read it
- Lists every literal, wildcard and prefixed ACL that affects the topic. A producer needs Write and a consumer needs Read for its exact principal string. Check for a typo in the principal, for a deny rule, and that the topic name in the exception is the one you expected: retry and dead letter topics need their own ACLs.
GroupAuthorizationException: Not authorized to access group: payment-service-v2
Check
kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --principal User:payment-service
- How to read it
- Look for a Group resource with Read that matches the group id in the exception. A literal ACL for another name, or none, is the cause. The topic ACL being correct does not help, because the group is authorised first.
A Spring consumer is running but consumes nothing, with no repeating error
Check
kafka-consumer-groups.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --describe --group payment-service-v2 --members
- How to read it
- No members while the pods are up means the listener containers have stopped. Search the application log from start-up for a single
AuthorizationExceptionorAuthenticationException: by default the container stops after the first one.
A transactional producer fails with TransactionalIdAuthorizationException, or an old cluster rejects an idempotent producer with ClusterAuthorizationException
Check
kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --transactional-id order-service-tx --resource-pattern-type match
- How to read it
- A transactional producer needs Write and Describe on its transactional id. If the id carries a per-instance suffix, the ACL must be prefixed.
ClusterAuthorizationExceptionon an idempotent producer points to a broker older than 2.8, which requiredIdempotentWriteon the Cluster resource.
You need to know what the broker denied and for whom
Check
The broker's `kafka-authorizer.log` (logger `kafka.authorizer.logger`) and the client metric `failed-authentication-total`
- How to read it
- Denied operations are logged at INFO with the principal, operation, resource and the reason. Allowed operations are only logged at DEBUG. A rising
failed-authentication-totalon clients, or the matching rate on brokers, separates an authentication problem from an authorisation one.
Key takeaways
- Encryption, authentication and authorisation are three separate switches. The listener's security protocol covers the first two, the authorizer and its ACLs the third.
- Everything hangs on the principal. ACLs and quotas name
User:..., however that identity was proved. - Without a matching ACL the answer is deny, a deny rule beats any allow, and super users skip the check altogether.
- A consumer needs Read on its group as well as on its topic. Grant groups, retry topics and transactional ids by prefix per service.
- The exception names the gate:
SslAuthenticationExceptionis TLS,SaslAuthenticationExceptionis credentials,TopicAuthorizationExceptionandGroupAuthorizationExceptionare ACLs. - Certificates are checked at the handshake, so an expiry spreads gradually as connections reopen. Monitor expiry dates and rotate before them.
- Spring's listener container stops on an auth error by default. Make that state visible or let it retry.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A new Inventory Service starts in production as SCRAM user inventory-service, group id inventory-service. An engineer has confirmed with kafka-acls.sh --list --topic orders.placed that the user has Read and Describe on the topic. The service logs GroupAuthorizationException: Not authorized to access group: inventory-service once and then consumes nothing.
What is missing?
Scenario 2 of 3
A cluster is being moved to ACLs. To avoid breaking 40 existing services, the team sets allow.everyone.if.no.acl.found=true and enables StandardAuthorizer. Next they add one ACL: User:payment-service may Read topic payments.requested. Within a minute the Order Service, which produces to payments.requested, starts failing with TopicAuthorizationException.
Why did adding a read permission break the producer?
Scenario 3 of 3
At 09:00 a platform team replaces the SCRAM password of user order-service with kafka-configs.sh, then updates the secret. The 6 running instances keep producing without errors. At 11:20 two instances are rescheduled onto new nodes; due to a caching delay they start with the old secret and fail with SaslAuthenticationException. The other 4 are still fine.
What explains the 4 healthy instances, and what should the rotation procedure be?
References
References: Apache Kafka 4.4 documentation: Security overview ↗ · Apache Kafka 4.4 documentation: Listener configuration ↗ · Apache Kafka 4.4 documentation: Encryption and authentication using SSL ↗ · Apache Kafka 4.4 documentation: Authentication using SASL ↗ · Apache Kafka 4.4 documentation: Authorization and ACLs ↗ · Apache Kafka 4.4 documentation: Broker configs (dynamic keystore and truststore updates) ↗ · Apache Kafka 4.4 documentation: Multi-tenancy (quotas) ↗ · KIP-679: Producer will enable the strongest delivery guarantee by default (idempotent producer ACL change) ↗ · Spring Boot reference: Apache Kafka support ↗ · Spring Boot reference: SSL bundles ↗ · Spring for Apache Kafka reference: Message listener containers (authExceptionRetryInterval) ↗ · Report a problem