Skip to content

Path 10 · lesson 4 of 6

Advanced32 minApache Kafka 4.4 documentation (KRaft only, `StandardAuthorizer`) · Java 21 · Spring Boot 3.5 with Spring for Apache Kafka 3.3 (kafka-clients 3.9). Property names checked against the Kafka 4.4 security chapter and the Spring Boot application properties reference

Kafka security: TLS, SASL and ACLs

How a connection is encrypted, how a client proves who it is, and how ACLs decide what that principal may do, with the exceptions each failure produces.

What you will learn

  • Read a broker's listener configuration and say which traffic is encrypted and which is authenticated
  • Choose between mutual TLS, SCRAM, OAUTHBEARER, PLAIN and Kerberos for a given environment
  • Write least-privilege ACLs for a producer service and a consumer service, including the group and transactional id
  • Configure a Spring Boot client for SASL_SSL with SCRAM and keep the secrets out of the repository
  • Map SslAuthenticationException, SaslAuthenticationException, TopicAuthorizationException and GroupAuthorizationException to their causes
  • Plan certificate and credential rotation so that expiry is never the way you find out

Before you start

The concept

A Kafka cluster with default settings accepts any connection, in clear text, and lets it do anything: read every topic, delete topics, change configuration. The Kafka documentation says so directly: security is optional. Securing a cluster means answering three separate questions for every connection, and Kafka has a separate mechanism for each.

QuestionMechanismWhere it is configured
Can anyone on the network read or alter the bytes?Encryption with TLS (Kafka's configuration still calls it SSL)The security protocol of the listener
Who is connecting?Authentication: a TLS client certificate (mutual TLS) or a SASL mechanismThe listener, plus credentials on the client
Is this identity allowed to do this?Authorisation: ACLs evaluated by an authorizerauthorizer.class.name on every broker and controller, and the ACLs themselves

The first two are properties of a listener. A broker can listen on several ports, each with a name and one of four security protocols. The protocol name tells you two things at once: whether the bytes are encrypted, and whether SASL is used to authenticate.

Security protocolEncryptedHow the client is identified
PLAINTEXTNoNot at all. Every client is the anonymous principal
SSLYesBy a client certificate if the broker sets ssl.client.auth=required (mutual TLS). Otherwise anonymous
SASL_PLAINTEXTNoBy SASL, over an unencrypted connection. Not for production: with PLAIN the password crosses the network readable
SASL_SSLYesBy SASL, inside a TLS connection. The usual choice for application clients

Authentication produces a principal, written User:name. With SASL it is the user name. With mutual TLS it is, by default, the full distinguished name of the certificate, such as User:CN=payment-service,OU=Payments,O=Example, which ssl.principal.mapping.rules can shorten. Everything after that point, ACLs and quotas alike, refers to the principal and never to a password or a certificate.

An ACL is one rule of the form: principal P is allowed (or denied) operation O from host H on resources matching pattern R. The resource has a type (Topic, Group, Cluster, TransactionalId, DelegationToken, User), a name, and a pattern type: literal for an exact name (or * for all), prefixed for every name that starts with the given text.

Why it exists

Kafka started as an internal pipeline on a trusted network, and for years the network boundary was the only protection. That stops working as soon as a cluster is shared. A topic of payment events and a topic of page views sit on the same brokers. Any service that can reach port 9092 can subscribe to either, and a test tool pointed at the wrong bootstrap address can write garbage into a production topic or delete it.

  • Confidentiality: without TLS, anyone who can capture traffic between a client and a broker, or between brokers, reads every record.
  • Accountability: without authentication, the broker cannot tell the Payment Service from a laptop. There is nothing to attach a permission or a quota to.
  • Containment: without ACLs, one compromised or misconfigured service has the whole cluster. With least-privilege ACLs it has its own topics and its own group.
  • Protection from accidents: most ACL denials in practice stop mistakes, not attackers. The wrong environment, a consumer joining another team's group and stealing its partitions, a script that deletes topics by pattern.

The cost is operational. Certificates expire, passwords must be rotated, and every new topic, group id or transactional id needs an ACL before the code that uses it is deployed. Most security incidents on a Kafka cluster are not break-ins. They are outages caused by the security configuration itself: an expired certificate, a renamed group without an ACL, a rotated password that one service did not pick up. That is why this lesson spends as much time on failure symptoms as on setup.

How it works inside

Listeners in a KRaft cluster

listeners gives each listener a name, host and port. listener.security.protocol.map assigns a security protocol to each name. inter.broker.listener.name chooses the listener brokers use for replication, and controller.listener.names the listener of the KRaft controllers, which may not be the same as the inter-broker one. A broker-only node must still define the controller listener's security settings, because it connects to the controllers as a client.

Give clients their own listener, so that replication and controller traffic can be isolated at the network level. Application clients never talk to the controller listener: admin requests such as creating a topic go to a broker, which forwards them to the active controller together with the client's principal, and the controller authorises them. This is why the authorizer must be configured on controllers as well as brokers, and why the controller listener needs the same care as the others. Whoever can authenticate there as a broker can change cluster metadata.

What TLS checks, and when

During the TLS handshake the broker presents its certificate. The client checks that it chains to a certificate authority in its truststore, that it is within its validity dates, and, because ssl.endpoint.identification.algorithm defaults to https, that the host name it connected to appears in the certificate, preferably as a Subject Alternative Name. With ssl.client.auth=required the client presents a certificate from its keystore and the broker does the same checks against its own truststore. The value requested is discouraged by the documentation: clients without a certificate still connect, which looks like security and is not.

These checks happen at the handshake only. An established connection keeps working after the certificate behind it expires. This shapes how a certificate expiry looks in production: nothing breaks at the expiry instant, and failures appear one connection at a time as clients reconnect, restart or open a connection to a broker they had not used yet.

SASL mechanisms

MechanismHow it proves identityFits when
PLAINUser name and password sent to the broker. Must only be used over TLS. The built-in implementation keeps passwords in the broker's JAAS configuration; production use needs a callback handler that checks an external storeA managed service issues API keys, or passwords live in an external directory
SCRAM-SHA-256 / SCRAM-SHA-512Challenge and response: the password itself is not sent. Kafka stores salted credentials in the KRaft metadata log, created with kafka-configs.sh or kafka-storage.sh. Use with TLSA self-managed cluster that wants user names and passwords without extra infrastructure
OAUTHBEARERA token from an OAuth 2 identity provider, validated by the broker against the provider's keys. The client refreshes it before expiry. The default unsecured-token implementation is for development onlyThe organisation already issues short-lived tokens to services
GSSAPIKerberos ticketsAn existing Kerberos or Active Directory environment

Mutual TLS is the alternative to SASL: the certificate is the identity. It needs no password store, at the price of issuing and rotating a certificate per service. By default a SASL session is authenticated once, when the connection opens. Setting connections.max.reauth.ms on the broker makes clients re-authenticate periodically, so that a deleted credential or an expired token stops working on existing connections too.

How the authorizer decides

In a KRaft cluster the built-in authorizer is org.apache.kafka.metadata.authorizer.StandardAuthorizer, set through authorizer.class.name on every node. It stores ACLs in the metadata log, so they replicate like any other metadata. For each request it applies these rules in order.

  1. If the principal is listed in super.users, allow. Super users bypass all ACLs, so keep the list to the brokers' own principal and a small set of administrators.
  2. If any matching ACL denies the operation, deny, however many ACLs allow it.
  3. If a matching ACL allows the operation, allow. An allow for Read, Write, Delete or Alter also implies Describe on that resource.
  4. If no ACL matches the resource at all, deny, unless allow.everyone.if.no.acl.found=true, in which case allow.

What each client needs

ClientResourceOperationsWhy
ProducerTopicWrite, DescribeProduce requests need Write. Metadata lookups need Describe, which Write implies
Idempotent producerTopicWriteSince Kafka 2.8 Write on the topic is enough. Older brokers demanded IdempotentWrite on the Cluster resource
Transactional producerTransactionalIdWrite, DescribeFinding the transaction coordinator needs Describe, starting and ending transactions need Write. Add Read on the Group if it commits consumer offsets in the transaction
ConsumerTopicRead, DescribeFetching needs Read on the topic, and so does committing an offset for it
ConsumerGroupReadJoining the group, heartbeating and committing offsets all check Read on the group. Group access is checked before topic access
Consumer with a dead letter topicDead letter topicWriteThe consumer is a producer for that topic

The table explains the most common surprise. A service that can read a topic perfectly well with a console tool still cannot consume in production if its group has no ACL, because the group is a resource of its own. kafka-acls.sh has --producer and --consumer shortcuts that generate these sets; note that --producer also grants Create on the topic, which you may not want.

Secrets, rotation and quotas

  • Secrets: passwords and private keys belong in a secret manager and reach the application as environment variables or mounted files. Since Kafka 2.7, keys and certificates can also be given in PEM form directly in the configuration, which suits secrets injected at runtime.
  • Broker certificates: a broker's keystore can be replaced without a restart, per listener, with kafka-configs.sh. The new certificate must be signed by the same certificate authority as the old one, or clients will reject it.
  • Changing the certificate authority: add the new authority to every truststore first, then issue certificates from it, then remove the old authority. Truststores can also be updated dynamically on brokers.
  • SCRAM passwords: changing a password with kafka-configs.sh takes effect for new connections. To rotate without an outage, create a second user with the same ACLs, move the service to it, then delete the first.
  • Quotas: once every client has a principal, kafka-configs.sh can cap producer_byte_rate, consumer_byte_rate and request_percentage per user. By default clients have no limit. A quota throttles a client that misbehaves, by accident or on purpose, before it starves the others.

Architecture

Drawing diagram…

A flowchart in which a client connects to the CLIENT listener, passes or fails the TLS handshake, passes or fails SCRAM authentication, becomes a principal, and is then checked against super users, deny ACLs and allow ACLs. Separate BROKER and CONTROLLER listeners carry replication and metadata traffic.

flowchart TD
  APP["Payment Service, user payment-service"] --> CL["Listener CLIENT, SASL_SSL, port 9093"]
  CL --> TLS{"TLS handshake passes?"}
  TLS -- "No" --> E1["SslAuthenticationException"]
  TLS -- "Yes" --> SASL{"SCRAM credentials valid?"}
  SASL -- "No" --> E2["SaslAuthenticationException"]
  SASL -- "Yes" --> PR["Principal User payment-service"]
  PR --> SU{"In super.users?"}
  SU -- "Yes" --> OK["Request handled"]
  SU -- "No" --> DENY{"Matching deny ACL?"}
  DENY -- "Yes" --> E3["Topic or Group AuthorizationException"]
  DENY -- "No" --> ALLOW{"Matching allow ACL?"}
  ALLOW -- "Yes" --> OK
  ALLOW -- "No" --> E3
  BR["Other brokers"] --> BL["Listener BROKER, SASL_SSL, port 9094"]
  BL --> REP["Replication"]
  BR --> CTL["Listener CONTROLLER, SASL_SSL, port 9095"]
  CTL --> META["KRaft metadata log, holds ACLs and SCRAM credentials"]
One request from the Payment Service passes three gates on the client listener; brokers and controllers use their own listeners.
  • The three gates are in a fixed order and each has its own exception. That makes the exception name the fastest diagnostic: an SslAuthenticationException means the request never reached SASL, and an authorisation exception means both earlier gates passed.
  • The last decision has no allow by default. With allow.everyone.if.no.acl.found at its default of false, a resource nobody wrote an ACL for is closed to everyone except super users.
  • ACLs and SCRAM credentials live in the KRaft metadata log, which is why the controller listener is drawn with the same protection as the others. Leaving it on PLAINTEXT would expose the store that the other two listeners rely on.

Message flow

Drawing diagram…

A sequence diagram in which a consumer opens a TLS connection, authenticates with SCRAM, and then sends JoinGroup, Fetch and OffsetCommit requests. For each request the broker asks the authorizer whether the principal may read the group or the topic.

sequenceDiagram
  participant C as Payment Service consumer
  participant B as Broker, CLIENT listener
  participant A as StandardAuthorizer
  C->>B: TCP connect to kafka-1 port 9093
  B-->>C: TLS certificate for kafka-1
  Note over C: Checks CA, validity dates and host name
  C->>B: SASL handshake, mechanism SCRAM-SHA-512
  C->>B: SCRAM client proof for user payment-service
  B-->>C: Authenticated
  Note over B: Connection principal is User payment-service
  C->>B: JoinGroup for group payment-service
  B->>A: Read on Group payment-service?
  A-->>B: Allowed by a prefixed ACL
  B-->>C: Member of the group
  C->>B: Fetch orders.placed partition 2
  B->>A: Read on Topic orders.placed?
  A-->>B: Allowed by a literal ACL
  B-->>C: Records
  C->>B: OffsetCommit for group payment-service
  B->>A: Read on Group, then Read on Topic?
  A-->>B: Allowed
  B-->>C: Committed
A consumer authenticates once per connection and is authorised on every request, for the group and for the topic.
  1. Authentication happens once, when the connection opens. Authorisation happens on every request, against the current ACLs, so adding or removing an ACL takes effect without the client reconnecting.
  2. The group is checked before the topic, and joining a group is the first thing a consumer does. A missing group ACL therefore stops a consumer before it has fetched a single record, with GroupAuthorizationException.
  3. An offset commit checks both the group and the topic. A principal that can read a topic but not commit for the group cannot make progress as a group member.

In a real system

Incident: the consumer that was renamed

A typical order platform. The Order Service publishes to orders.placed (12 partitions, keyed by order id). The Payment Service consumes it in the group payment-service as the SCRAM user payment-service. The cluster runs StandardAuthorizer with ACLs managed from a repository: Read on topic orders.placed and Read on the literal group payment-service.

A developer wants to replay the last day of orders into a new settlement table, and takes the usual shortcut: change the group id to payment-service-v2 so the consumer starts from the earliest offset. The change passes review. Nobody connects a group id in application.yml with an ACL in another repository.

TimeWhat happenedWhat people saw
14:05The new version rolls out. Each instance authenticates, asks to join payment-service-v2 and is refusedOne ERROR per instance with GroupAuthorizationException: Not authorized to access group: payment-service-v2, then silence
14:05Spring's listener container treats authorisation errors as fatal by default and stops. The application itself keeps runningPods are Running and Ready. The health endpoint is green. The deployment is marked successful
14:06 to 14:55No instance consumes orders.placed. Orders are placed and never chargedThe lag dashboard is filtered on the new group, which never formed, so it shows no data instead of a climbing line
14:55Support reports orders stuck in 'awaiting payment'An engineer finds the single ERROR line in the logs from 50 minutes earlier
15:02A prefixed ACL for groups starting with payment-service is applied and the service is restartedThe group forms and works through 50 minutes of backlog

The broker did exactly what it was configured to do. What turned a correct denial into a 50-minute outage was everything around it: an ACL that matched one exact name, a container that stops quietly, a health check that did not include the listener, and a lag alert that could not tell 'no lag' from 'no group'.

  • Grant group access with a prefixed pattern owned by the service, such as payment-service, so that -v2 and -replay are covered. Do the same for dead letter and retry topics.
  • Keep ACLs in the same change as the configuration that needs them, or check in the pipeline that every topic and group id in the service's configuration has a matching ACL.
  • Make a stopped listener container fail the health check, or set authExceptionRetryInterval so the container retries and recovers once the ACL exists.
  • Alert on a group having no active members and on the consumed-records rate falling to zero, not only on lag.

The other classic of this kind is expiry. The Break card Expired for some, fine for others is a real case of a certificate in the chain expiring while the server's own certificate was fine. The exercise below is the Kafka version.

Java and Spring Boot

The broker side of a SASL_SSL listener, the commands that create a SCRAM user and its least-privilege ACLs, and a Spring Boot client configured from environment variables. spring.kafka.security.protocol, spring.kafka.ssl.bundle and spring.kafka.properties.* are Spring Boot properties; everything under properties is passed to the Kafka client unchanged.

broker/server.properties (security part)Properties
process.roles=brokerlisteners=CLIENT://kafka-1.internal.example:9093,BROKER://kafka-1.internal.example:9094inter.broker.listener.name=BROKERcontroller.listener.names=CONTROLLERlistener.security.protocol.map=CLIENT:SASL_SSL,BROKER:SASL_SSL,CONTROLLER:SASL_SSLsasl.enabled.mechanisms=SCRAM-SHA-512sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512config.providers=fileconfig.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProviderssl.keystore.type=PKCS12ssl.keystore.location=/etc/kafka/secrets/kafka-1.keystore.p12ssl.keystore.password=${file:/etc/kafka/secrets/store.properties:keystore.password}ssl.truststore.type=PKCS12ssl.truststore.location=/etc/kafka/secrets/truststore.p12ssl.truststore.password=${file:/etc/kafka/secrets/store.properties:truststore.password}authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizersuper.users=User:kafka-broker;User:kafka-adminallow.everyone.if.no.acl.found=false

Why it is written this way

  • Three named listeners, all SASL_SSL. The controller listener is in the protocol map but not in listeners, because this node is a broker only: it connects to the controllers and needs to know how.
  • super.users is separated by semicolons, because certificate names can contain commas. It holds the principal the brokers use among themselves and one administrator, nothing else.
  • allow.everyone.if.no.acl.found=false is the default, written out so that nobody has to wonder. A topic without ACLs is closed.
  • The store passwords are read through a configuration provider from a file outside the main configuration. The JAAS configuration with the brokers' own credentials is omitted here.
  • The same authorizer.class.name and super.users must be set on the controller nodes, because forwarded admin requests are authorised there.

Configuration

The settings that decide whether a connection is encrypted, authenticated and authorised. Broker defaults are from the Kafka 4.4 broker configuration reference; the authorizer settings are from the security chapter.

listener.security.protocol.mapDefault SASL_SSL:SASL_SSL,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT
What it does
Assigns one of PLAINTEXT, SSL, SASL_PLAINTEXT or SASL_SSL to each named listener.
When to change it
Name listeners by purpose (CLIENT, BROKER, CONTROLLER) and map every one explicitly. No production listener should be PLAINTEXT.
security.protocol (client)Default PLAINTEXT
What it does
The protocol the client uses. It must match the listener it connects to. In Spring Boot: spring.kafka.security.protocol.
When to change it
Set to SASL_SSL, or SSL for mutual TLS. A mismatch shows as timeouts and disconnects, not as a clear error.
sasl.mechanism (client)Default GSSAPI
What it does
The SASL mechanism the client offers. The broker must have it in sasl.enabled.mechanisms, whose default is also GSSAPI.
When to change it
Set on both sides whenever SASL is used without Kerberos, for example SCRAM-SHA-512.
ssl.client.auth (broker)Default none
What it does
Whether the broker asks clients for a certificate: required, requested or none.
When to change it
Use required for mutual TLS. Avoid requested: clients without a certificate still get in, as the anonymous principal.
ssl.endpoint.identification.algorithmDefault https
What it does
Makes the connecting side check that the host name matches the certificate. An empty value disables the check.
When to change it
Keep the default. Disabling it removes the protection against a server presenting someone else's valid certificate.
ssl.enabled.protocolsDefault TLSv1.2,TLSv1.3
What it does
The TLS versions accepted. Both sides must share at least one.
When to change it
Leave alone unless policy requires TLS 1.3 only, and then check that every client's Java version supports it.
authorizer.class.nameDefault "" (empty: no authorisation)
What it does
The class that authorises requests. For KRaft: org.apache.kafka.metadata.authorizer.StandardAuthorizer.
When to change it
Set on every broker and controller. Enable it only after the ACLs for existing clients are in place, or use the phased approach below.
allow.everyone.if.no.acl.foundDefault false
What it does
When true, a resource that no ACL matches is open to all principals. Resources with any ACL are enforced normally.
When to change it
Use true only while migrating a running cluster to ACLs, and switch it back once every resource has rules.
super.usersDefault none
What it does
Principals that skip ACL checks, separated by semicolons, for example User:kafka-broker;User:kafka-admin.
When to change it
The brokers' own principal must be able to replicate, either as a super user or through ClusterAction on the Cluster resource. Keep human administrators off the list where you can.
connections.max.reauth.msDefault 0 (no re-authentication)
What it does
When positive, the broker tells SASL clients a session lifetime and disconnects connections that do not re-authenticate in time.
When to change it
Set it when credentials can be revoked or tokens expire, so that revocation reaches connections that are already open.
spring.kafka.listener.auth-exception-retry-intervalDefault none (auth errors stop the container)
What it does
How long the listener container waits before retrying after an authentication or authorisation exception.
When to change it
Set it, for example to 30 seconds, so a consumer recovers by itself once a missing ACL is added.

Hands-on exercise

Troubleshooting: failures that started on their own at the weekend

Work from symptoms and tool output to the cause of an authentication failure, explain its odd timing, and decide the fix. All output below is illustrative: written for this exercise in the format the tools produce.

Steps · 0 of 6 done

What can go wrong

A literal group ACL and a changed group id
Symptom
After a deployment a consumer processes nothing. One GroupAuthorizationException per instance in the log, then no further errors. The application reports healthy.
Cause
The ACL names the old group exactly. Group access is checked when the consumer joins, and Spring's container stops on authorisation errors by default.
Fix
Grant Read on the group with a prefixed pattern owned by the service, deploy ACL changes with or before the code, and set authExceptionRetryInterval or fail the health check when a container stops.
Certificate expiry discovered by the clients
Symptom
With no change deployed, SslAuthenticationException: SSL handshake failed appears on some connections and spreads over hours. Restarting a client makes it worse.
Cause
A broker, client or authority certificate passed its validity date. Existing connections survive until they are reopened, so the failure is gradual.
Fix
Renew and reload the keystore (dynamically on brokers). Track expiry dates as a monitored metric with alerts well ahead, and automate renewal for every node.
allow.everyone.if.no.acl.found left on
Symptom
A security review finds that any authenticated service can read newly created topics, although ACLs exist for the older ones.
Cause
The flag was enabled for the migration to ACLs and never turned off. Every resource without an ACL is public, including each new topic.
Fix
Add ACLs for all existing resources, set the flag back to false, and create ACLs in the same step as topics.
Host name verification switched off to get it working
Symptom
Everything connects. Clients carry ssl.endpoint.identification.algorithm set to an empty string, added during setup and never removed.
Cause
The broker certificates lacked the right Subject Alternative Names, often because the authority did not copy them from the signing request, and disabling the check was quicker.
Fix
Reissue certificates with the brokers' advertised host names as SANs, verify with openssl x509 -text -noout, and remove the override.
Wildcard ACLs and oversized super user lists
Symptom
ACLs exist, but a mistake in one service still reads or deletes another team's topics.
Cause
Services were granted --topic '*' or --operation All, or application principals were put in super.users to make errors go away.
Fix
Grant specific operations on literal or prefixed names per service. Reserve super.users for the brokers and a small administrative principal.
Password rotated in place
Symptom
After a SCRAM password change, instances fail with SaslAuthenticationException as they restart, while long-running ones keep working for a while.
Cause
The credential was altered before every instance had the new secret. Open connections stay authenticated; new ones use whatever password the instance was started with.
Fix
Rotate by adding a second user with identical ACLs, moving the service to it, and deleting the old user afterwards.

How to troubleshoot

SslAuthenticationException: SSL handshake failed

Check

openssl s_client -connect kafka-1.internal.example:9093 </dev/null 2>/dev/null | openssl x509 -noout -subject -issuer -enddate
How to read it
The nested cause tells which check failed. A CertificateExpiredException or a notAfter in the past means expiry. unable to find valid certification path means the client's truststore lacks the issuing authority. No subject alternative names matching means the host name the client used is not in the certificate. Run the check against every broker: one stale node is enough.
SaslAuthenticationException: Authentication failed ... invalid credentials

Check

kafka-configs.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --describe --entity-type users --entity-name payment-service
How to read it
If no SCRAM credential is listed for the mechanism the client uses, the user does not exist for that mechanism (SCRAM-SHA-256 and SCRAM-SHA-512 are separate credentials). If one is listed, the password in the client's secret is wrong or stale. Also compare the client's sasl.mechanism with the broker's sasl.enabled.mechanisms.
TopicAuthorizationException: Not authorized to access topics: [orders.placed]

Check

kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --topic orders.placed --resource-pattern-type match
How to read it
Lists every literal, wildcard and prefixed ACL that affects the topic. A producer needs Write and a consumer needs Read for its exact principal string. Check for a typo in the principal, for a deny rule, and that the topic name in the exception is the one you expected: retry and dead letter topics need their own ACLs.
GroupAuthorizationException: Not authorized to access group: payment-service-v2

Check

kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --principal User:payment-service
How to read it
Look for a Group resource with Read that matches the group id in the exception. A literal ACL for another name, or none, is the cause. The topic ACL being correct does not help, because the group is authorised first.
A Spring consumer is running but consumes nothing, with no repeating error

Check

kafka-consumer-groups.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --describe --group payment-service-v2 --members
How to read it
No members while the pods are up means the listener containers have stopped. Search the application log from start-up for a single AuthorizationException or AuthenticationException: by default the container stops after the first one.
A transactional producer fails with TransactionalIdAuthorizationException, or an old cluster rejects an idempotent producer with ClusterAuthorizationException

Check

kafka-acls.sh --bootstrap-server kafka-1.internal.example:9093 --command-config admin.properties --list --transactional-id order-service-tx --resource-pattern-type match
How to read it
A transactional producer needs Write and Describe on its transactional id. If the id carries a per-instance suffix, the ACL must be prefixed. ClusterAuthorizationException on an idempotent producer points to a broker older than 2.8, which required IdempotentWrite on the Cluster resource.
You need to know what the broker denied and for whom

Check

The broker's `kafka-authorizer.log` (logger `kafka.authorizer.logger`) and the client metric `failed-authentication-total`
How to read it
Denied operations are logged at INFO with the principal, operation, resource and the reason. Allowed operations are only logged at DEBUG. A rising failed-authentication-total on clients, or the matching rate on brokers, separates an authentication problem from an authorisation one.

Key takeaways

  • Encryption, authentication and authorisation are three separate switches. The listener's security protocol covers the first two, the authorizer and its ACLs the third.
  • Everything hangs on the principal. ACLs and quotas name User:..., however that identity was proved.
  • Without a matching ACL the answer is deny, a deny rule beats any allow, and super users skip the check altogether.
  • A consumer needs Read on its group as well as on its topic. Grant groups, retry topics and transactional ids by prefix per service.
  • The exception names the gate: SslAuthenticationException is TLS, SaslAuthenticationException is credentials, TopicAuthorizationException and GroupAuthorizationException are ACLs.
  • Certificates are checked at the handshake, so an expiry spreads gradually as connections reopen. Monitor expiry dates and rotate before them.
  • Spring's listener container stops on an auth error by default. Make that state visible or let it retry.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A new Inventory Service starts in production as SCRAM user inventory-service, group id inventory-service. An engineer has confirmed with kafka-acls.sh --list --topic orders.placed that the user has Read and Describe on the topic. The service logs GroupAuthorizationException: Not authorized to access group: inventory-service once and then consumes nothing.

What is missing?

Scenario 2 of 3

A cluster is being moved to ACLs. To avoid breaking 40 existing services, the team sets allow.everyone.if.no.acl.found=true and enables StandardAuthorizer. Next they add one ACL: User:payment-service may Read topic payments.requested. Within a minute the Order Service, which produces to payments.requested, starts failing with TopicAuthorizationException.

Why did adding a read permission break the producer?

Scenario 3 of 3

At 09:00 a platform team replaces the SCRAM password of user order-service with kafka-configs.sh, then updates the secret. The 6 running instances keep producing without errors. At 11:20 two instances are rescheduled onto new nodes; due to a caching delay they start with the old secret and fail with SaslAuthenticationException. The other 4 are still fine.

What explains the 4 healthy instances, and what should the rotation procedure be?

References

References: Apache Kafka 4.4 documentation: Security overview ↗ · Apache Kafka 4.4 documentation: Listener configuration ↗ · Apache Kafka 4.4 documentation: Encryption and authentication using SSL ↗ · Apache Kafka 4.4 documentation: Authentication using SASL ↗ · Apache Kafka 4.4 documentation: Authorization and ACLs ↗ · Apache Kafka 4.4 documentation: Broker configs (dynamic keystore and truststore updates) ↗ · Apache Kafka 4.4 documentation: Multi-tenancy (quotas) ↗ · KIP-679: Producer will enable the strongest delivery guarantee by default (idempotent producer ACL change) ↗ · Spring Boot reference: Apache Kafka support ↗ · Spring Boot reference: SSL bundles ↗ · Spring for Apache Kafka reference: Message listener containers (authExceptionRetryInterval) ↗ · Report a problem