Skip to content

Path 4 · lesson 3 of 5

Intermediate26 minApache Kafka 4.x (defaults checked against the 4.0 and 4.4 docs) · Java 21 · Spring Boot 3.4 · Spring for Apache Kafka 3.3 (kafka-clients 3.8 to 3.9)

Offset management and commits

What a committed offset is, where it is stored, when automatic and manual commits happen, what Spring's ack modes do, and how to reset a group's position.

What you will learn

  • State exactly what an offset commit stores and what it does not acknowledge
  • Say when auto-commit fires and identify code where it can skip unprocessed records
  • Choose between commitSync and commitAsync and combine them safely
  • Pick a Spring AckMode for a listener from its documented behaviour
  • Predict where a consumer starts for a new group, an expired group and a restarted one
  • Rewind or skip a consumer group with kafka-consumer-groups.sh --reset-offsets

Before you start

The concept

A consumer has two offsets for every partition it reads. The position is the offset of the next record poll will return. It lives in the consumer's memory and advances every time poll hands out records. The committed offset is the position that has been stored durably for the group. It is the only one that survives a crash, a restart or a rebalance: whoever reads the partition next for this group starts from the committed offset.

A commit is a small write that says: for group G, topic T, partition P, the next record to read is offset N. It is one number per partition. It is not a list of handled records and it is not a per-record acknowledgement. Committing offset 47 claims that everything below 47 is done, whether or not that is true.

When you commit relative to processing decides the delivery guarantee. Commit before processing and a crash in between loses the records: at-most-once. Commit after processing and a crash in between repeats them: at-least-once. There is no ordering of the two steps that gives exactly-once on its own, because processing and committing are two separate writes to two systems.

Crash between processing and commit

Simulation · not a real cluster

Run the payment consumer with the crash placed after processing and before the commit. Then move the commit before processing and crash again.

Why it exists

A traditional queue tracks each message on the broker: delivered, acknowledged, redeliver after a timeout. That is flexible, but the broker carries state for every in-flight message of every consumer. Kafka's log makes a much cheaper scheme possible. Records in a partition are read in order, so the progress of a group through a partition can be summarised by a single number. The broker stores one small value per group and partition, however many records have gone by.

Keeping that number in Kafka, rather than in each consumer's local disk, is what makes groups work. When a partition moves to another member during a rebalance, or a pod is replaced by a new one, the new reader asks the coordinator for the committed offset and continues from there. And because progress is only a number, it can be moved on purpose: rewind it to replay a day of events after fixing a bug, or advance it to skip a backlog nobody needs.

The cost of the single-number design is the callout above. A commit cannot express that one record in the middle failed. Handling individual failures needs something else: retries, a dead letter topic, or an idempotent consumer. Those are the subject of the reliability path.

How it works inside

Where commits are stored

Commits go to the group's coordinator, which appends them to the internal topic __consumer_offsets. That topic is compacted: the record key identifies the group, topic and partition, the value holds the offset with some metadata, and compaction keeps the latest value for each key. The coordinator also keeps the offsets in memory so that it can answer offset lookups quickly, and it acknowledges a commit only after the replicas of the offsets topic have it. By default the topic has 50 partitions (offsets.topic.num.partitions) and replication factor 3 (offsets.topic.replication.factor).

Committed offsets do not live for ever. Once a group has no members left, its offsets are removed after offsets.retention.minutes, 7 days by default. A consumer that comes back after that finds no committed offset and behaves like a brand-new group.

Automatic commits

With enable.auto.commit=true, which is the Kafka client default, the consumer commits its positions about every auto.commit.interval.ms (5 seconds). The commit is not made by an independent timer that understands your processing. In the classic consumer the check runs inside poll: if the interval has elapsed, the consumer commits the positions it has reached, meaning everything earlier polls returned. It also commits when the consumer is closed and before partitions are given up in a rebalance.

This gives at-least-once delivery on one condition, which the KafkaConsumer Javadoc spells out: you must finish processing everything a poll returned before you call poll again or close the consumer. If the loop processes each batch completely on the polling thread, a commit made by the next poll only ever covers finished work. After a crash, the records processed since the last auto-commit are delivered again: at-least-once with a window of up to the commit interval plus one batch.

Manual commits: sync and async

commitSynccommitAsync
Blocks the poll threadYes, until success, an unrecoverable error, or default.api.timeout.msNo
On failureThrows to the callerPasses the error to the callback if one was given, otherwise discards it
Typical costOne round trip to the coordinator per callNegligible for the caller
Use it forThe last commit before giving up a partition or shutting downRoutine commits after each batch
Both commit only to Kafka. Called without arguments they commit the offsets returned by the last poll.

A common combination is commitAsync after each batch, because a single failed commit is harmless when the next one supersedes it, and commitSync in onPartitionsRevoked and on shutdown, where there is no next one. Both also accept an explicit map of partition to OffsetAndMetadata, which lets you commit less than the whole batch, for example after each partition's records.

Only members of the current group generation may commit. A consumer that exceeded max.poll.interval.ms and was removed from the group gets CommitFailedException from commitSync. By then its partitions belong to someone else, and the work since the previous commit will be done again by the new owner.

Spring ack modes

Spring's listener container does its own committing. Since version 2.3 it sets enable.auto.commit to false unless you set it explicitly, and commits according to ContainerProperties.AckMode. The definitions below are those of the Spring for Apache Kafka reference, for a container not using transactions.

AckModeWhen the container commits
BATCH (default)When all the records returned by the poll have been processed
RECORDWhen the listener returns after processing each record
TIMEWhen all records from the poll have been processed, provided ackTime has passed since the last commit
COUNTWhen all records from the poll have been processed, provided ackCount records have been received since the last commit
COUNT_TIMELike TIME and COUNT, when either condition is true
MANUALThe listener calls Acknowledgment.acknowledge(). After that the same semantics as BATCH apply: the commit is made once the records from the poll have been processed
MANUAL_IMMEDIATEImmediately when the listener calls Acknowledgment.acknowledge()
The container uses commitSync by default (syncCommits is true) and commitAsync when it is set to false.

BATCH suits most listeners: one commit per poll, and a crash repeats at most one batch. RECORD narrows the repeat to one record at the price of a commit round trip per record. The manual modes are for when the listener itself must decide, for instance when it acknowledges only after an asynchronous step has completed. In every mode a commit still means everything below this offset is done.

When there is no committed offset

auto.offset.reset is consulted in exactly two situations: the group has no committed offset for the partition, or the committed offset no longer exists on the broker because retention deleted that part of the log. In every other start, restart or rebalance it is ignored and the committed offset is used. Changing it on a group that already has commits has no effect.

ValueStart position when no valid committed offset exists
latest (default)The end of the log. Records written before the consumer first got the partition are skipped
earliestThe oldest record still retained. The whole retained history is processed
noneNo automatic choice: the consumer throws an exception and you decide
by_duration:<duration>The offset for the current time minus an ISO-8601 duration, for example by_duration:PT1H. Listed in the 4.x consumer configuration; older clients do not accept it

Moving the position on purpose

Inside a consumer, seek(partition, offset), seekToBeginning and seekToEnd change the in-memory position, and the next poll reads from there. In Spring, a listener that implements ConsumerSeekAware receives a callback for the same purpose. From outside, kafka-consumer-groups.sh --reset-offsets rewrites the committed offsets of a group. The documentation tells you to make sure the consumer instances are inactive first. The tool only displays the result unless you add --execute, since --dry-run is its default.

Offsets & at-least-once

Simulation, not a real cluster

The animated Build lesson walks through commit timing and a crash, record by record.

Step 1 · The pattern

Read, process, remember your place

The consumer reads a message, does the work (here, sending an email) and periodically commits its offset: a bookmark saying 'I have handled everything up to here'.

  • message
  • processed
Open the full animated lesson

Architecture

Drawing diagram…

Flowchart of a partition log divided into four ranges: offsets 0 to 41 processed and committed, 42 to 46 processed but not committed, 47 to 49 returned by poll and in progress, and 50 to 57 not yet returned. The committed offset points at 42 and the position at 50. The consumer sends a commit to the group coordinator, which appends it to the compacted __consumer_offsets topic, from which the next owner of the partition looks up where to resume.

flowchart TB
  subgraph LOG["orders.events partition 2"]
    R1["Offsets 0 to 41: processed and committed"]
    R2["Offsets 42 to 46: processed, not yet committed"]
    R3["Offsets 47 to 49: returned by poll, in progress"]
    R4["Offsets 50 to 57: not yet returned"]
    R1 --> R2 --> R3 --> R4
  end
  CO["Committed offset 42"] -.-> R2
  POS["Position 50"] -.-> R4
  C["Consumer in group payment-service"]
  GC["Group coordinator"]
  OT["Internal topic __consumer_offsets, compacted"]
  NEXT["Next owner after a crash or rebalance"]
  C -->|"commit group, topic, partition 2, offset 47"| GC
  GC -->|"append latest offset per key"| OT
  OT -->|"committed offset lookup"| NEXT
  NEXT -->|"resumes at the committed offset"| LOG
One partition seen by one group: the committed offset trails the in-memory position, and the gap between them is what a crash repeats.
  • The committed offset (42) and the position (50) are different numbers at almost every moment. Lag reported by the tools is measured from the committed offset to the log end offset.
  • If the consumer dies now, the next owner starts at 42. Offsets 42 to 46 are processed a second time and 47 to 49 possibly a second time. Nothing is lost.
  • The pending commit carries 47, the offset after the last finished record, not 46.
  • __consumer_offsets is an ordinary replicated topic with compaction. Losing a broker does not lose the group's progress.

Message flow

Drawing diagram…

Sequence diagram. Consumer A resumes partition 2 at committed offset 42, polls offsets 42 to 46 and stores payments for 42, 43 and 44, then crashes before committing. After the session timeout the coordinator assigns the partition to consumer B, which reads the committed offset 42, polls the same records, stores 42 to 44 a second time and 45 and 46 for the first time, and commits offset 47.

sequenceDiagram
  participant A as Consumer A
  participant D as Payments database
  participant G as Group coordinator
  participant B as Consumer B
  A->>G: fetch committed offset for partition 2
  G-->>A: 42
  A->>A: poll returns offsets 42 to 46
  A->>D: store payments for 42, 43, 44
  Note over A: process crashes before the commit
  Note over G: no heartbeat for session.timeout.ms
  G-->>B: rebalance, partition 2 assigned to B
  B->>G: fetch committed offset for partition 2
  G-->>B: 42
  B->>B: poll returns offsets 42 to 46
  B->>D: store payments for 42, 43, 44 again
  B->>D: store payments for 45, 46
  B->>G: commit offset 47
  G-->>B: commit acknowledged
Commit after processing with a crash in between: records 42 to 44 reach the database twice.
  1. Consumer B cannot know that A already handled 42 to 44. The only shared knowledge is the committed offset, and it still says 42.
  2. This is at-least-once working as designed. The duplicates must be made harmless by the processing itself, for example a unique constraint on the order id.
  3. Had A committed 47 before writing to the database and then crashed, B would have started at 47 and the five payments would never have been stored.
  4. The commit of 47 covers all five records with a single number.

In a real system

A typical payment service consumes orders.events (12 partitions, key orderId) in the group payment-service. For every OrderPlaced event it charges the customer through a provider and stores a payment row. Two things must hold: no order is skipped, and no customer is charged twice.

  • No skipped orders comes from the commit policy: commit only after the payment row is stored. The service uses Spring's MANUAL_IMMEDIATE mode and acknowledges at the end of the listener, or simply the default BATCH.
  • No double charges cannot come from the commit policy, since at-least-once will redeliver after a crash. The service stores payments under a unique key on orderId and sends that same id to the provider as an idempotency key, so a redelivered event is detected and skipped.
  • First deploy uses auto-offset-reset: earliest, so orders placed before the service first started are not skipped.
  • Replay after a bug: a release mis-rounded amounts for two hours. After the fix, the team stops the service, resets the group to the time of the bad deploy with --to-datetime, and starts it again. The idempotent write turns the replay into corrections rather than second charges.

What can go wrong with this bookkeeping at scale is shown in the Break card The offset that went backwards. Making the processing safe to repeat is the topic of Duplicates and idempotency.

Java and Spring Boot

A plain consumer that commits explicit offsets, asynchronously per batch and synchronously when it loses partitions or shuts down. Then the Spring equivalent with manual acknowledgement, and the CLI commands for a controlled replay.

CommittingOrderConsumer.javaJava
package com.example.payments.consumer;import java.time.Duration;import java.util.Collection;import java.util.HashMap;import java.util.List;import java.util.Map;import java.util.Properties;import java.util.concurrent.atomic.AtomicBoolean;import org.apache.kafka.clients.consumer.ConsumerConfig;import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;import org.apache.kafka.clients.consumer.ConsumerRecord;import org.apache.kafka.clients.consumer.ConsumerRecords;import org.apache.kafka.clients.consumer.KafkaConsumer;import org.apache.kafka.clients.consumer.OffsetAndMetadata;import org.apache.kafka.common.TopicPartition;import org.apache.kafka.common.errors.WakeupException;import org.apache.kafka.common.serialization.StringDeserializer;import org.slf4j.Logger;import org.slf4j.LoggerFactory;public class CommittingOrderConsumer implements Runnable, ConsumerRebalanceListener {    private static final Logger log = LoggerFactory.getLogger(CommittingOrderConsumer.class);    private final AtomicBoolean closed = new AtomicBoolean(false);    private final KafkaConsumer<String, String> consumer;    /** Next offset to read for each partition this consumer has finished work on. */    private final Map<TopicPartition, OffsetAndMetadata> processed = new HashMap<>();    public CommittingOrderConsumer(String bootstrapServers) {        Properties props = new Properties();        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);        props.put(ConsumerConfig.GROUP_ID_CONFIG, "payment-service");        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");        this.consumer = new KafkaConsumer<>(props);    }    @Override    public void run() {        try {            consumer.subscribe(List.of("orders.events"), this);            while (!closed.get()) {                ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));                for (ConsumerRecord<String, String> record : records) {                    handle(record);                    processed.put(new TopicPartition(record.topic(), record.partition()),                            new OffsetAndMetadata(record.offset() + 1));                }                if (!records.isEmpty()) {                    consumer.commitAsync(Map.copyOf(processed), (offsets, exception) -> {                        if (exception != null) {                            log.warn("async commit failed for {}", offsets, exception);                        }                    });                }            }        } catch (WakeupException e) {            if (!closed.get()) {                throw e;            }        } finally {            try {                consumer.commitSync(processed);            } catch (RuntimeException e) {                log.warn("final commit failed, the last batch will be read again", e);            } finally {                consumer.close();            }        }    }    @Override    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {        Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();        for (TopicPartition partition : partitions) {            OffsetAndMetadata offset = processed.remove(partition);            if (offset != null) {                toCommit.put(partition, offset);            }        }        consumer.commitSync(toCommit);    }    @Override    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {        // positions are initialised from the committed offsets by the consumer itself    }    @Override    public void onPartitionsLost(Collection<TopicPartition> partitions) {        // already owned by someone else: a commit would be rejected, so only forget them        partitions.forEach(processed::remove);    }    public void shutdown() {        closed.set(true);        consumer.wakeup();    }    private void handle(ConsumerRecord<String, String> record) {        // store the payment idempotently, keyed by the order id in record.key()    }}

Why it is written this way

  • record.offset() + 1 is the value committed: the next record to read, never the offset of the record just handled.
  • The offset is recorded only after handle returns, so a commit can never run ahead of the work. If handle throws, the loop ends and the record is read again by the next owner.
  • commitAsync keeps the loop fast. A failed async commit is only logged, because the next batch's commit carries a higher offset anyway.
  • onPartitionsRevoked runs inside poll on the same thread, before the partitions go to another member. A synchronous commit there keeps the new owner from repeating finished work.
  • onPartitionsLost is overridden because its default implementation calls onPartitionsRevoked, and committing for partitions that have already been reassigned fails.
  • The final commitSync in finally covers a clean shutdown. Map.copyOf gives the async commit its own snapshot, since the map keeps changing.

Configuration

Offset-related settings with their documented defaults for Kafka 4.x, and the Spring settings that replace some of them. The client defaults are the same in the 3.8 and 3.9 clients used by Spring for Apache Kafka 3.3.

enable.auto.commitDefault true (Spring for Apache Kafka sets false unless you set it)
What it does
Whether the consumer commits its positions periodically by itself.
When to change it
Leave it false in Spring and let the container commit. In a plain consumer, turn it off whenever records are handed to other threads or a commit must follow a specific write.
auto.commit.interval.msDefault 5000 (5 seconds)
What it does
How often offsets are auto-committed when enable.auto.commit is true.
When to change it
A lower value shrinks the window of records repeated after a crash and adds commit traffic. It has no effect once auto-commit is off.
auto.offset.resetDefault latest
What it does
Where to start when the group has no committed offset for a partition, or the committed offset is no longer on the broker.
When to change it
Use earliest for services that must not miss records produced before their first start. Use none when a silent jump in either direction would be dangerous and you prefer a failure.
spring.kafka.listener.ack-modeDefault BATCH (container default)
What it does
The container's AckMode: when it commits offsets on behalf of the listener.
When to change it
Keep BATCH for most listeners. Use RECORD when repeating a whole batch after a crash is too costly, and a manual mode when the listener must decide the moment.
ContainerProperties.syncCommitsDefault true
What it does
Whether the Spring container commits with commitSync (true) or commitAsync (false).
When to change it
Leave it true unless commit latency is measurably limiting throughput. Async commits report failures only through the commit callback.
offsets.retention.minutes (broker)Default 10080 (7 days)
What it does
How long committed offsets are kept after a group has lost all its members.
When to change it
Raise it if consumers are routinely stopped for longer, such as weekly batch jobs or environments that are switched off. Otherwise they restart from auto.offset.reset.
offsets.topic.replication.factor (broker)Default 3
What it does
Replication factor of __consumer_offsets. The topic cannot be created until the cluster has that many brokers.
When to change it
Keep 3 in production. A single-broker development cluster needs it set to 1 for consumer groups to work at all.

Hands-on exercise

Move the commit and crash the consumer

Use the delivery semantics simulation to see how the position of the commit relative to processing decides between lost and repeated payments, then confirm it in the crash lab.

Steps · 0 of 5 done

Where the commit sits

Simulation · not a real cluster

Change the commit order and the crash point, and watch the committed offset against the stored payments.

Take it further in the lab: Consumer crash before offset commit

What can go wrong

Auto-commit with processing on other threads
Symptom
After a crash or a deploy, a handful of records were never processed, and there is no error and no lag.
Cause
The poll thread handed records to a pool and polled again. The auto-commit inside that poll covered records the workers had not finished.
Fix
Set enable.auto.commit=false and commit only offsets whose records are complete, from the polling thread. Pause the partitions while workers are busy, as the Javadoc suggests.
A new group that starts at the end
Symptom
A newly deployed service ignores everything already in the topic and only reacts to records produced after it started.
Cause
The group had no committed offsets and auto.offset.reset was left at its default, latest.
Fix
Set auto.offset.reset=earliest before the first start of a group that must see existing data. For a group that already started, stop it and reset its offsets with the CLI.
Offsets expired while the consumer was off
Symptom
A job that was stopped for two weeks either skips to the newest records or reprocesses the entire topic when it comes back.
Cause
The group was empty for longer than offsets.retention.minutes (7 days), so its committed offsets were deleted and auto.offset.reset applied.
Fix
Raise the broker's offset retention to cover planned downtime, or record the offsets before a long stop and restore them with --reset-offsets --from-file.
Treating the commit as a per-record acknowledgement
Symptom
One failed record disappears: later records were processed and committed, and the failed one is never seen again.
Cause
The code caught the exception, moved on, and a later commit carried an offset beyond the failed record.
Fix
Decide explicitly what happens to a failed record: stop and retry it in place, or publish it to a retry or dead letter topic before moving on. See the reliability path.
Resetting offsets while the group is running
Symptom
The reset command refuses to run and reports that offsets can only be reset when the group is inactive.
Cause
The group still has active members. They would keep committing their own positions over whatever the tool wrote, so the tool checks the group state first.
Fix
Stop all instances, confirm the group has no members with --describe --state, run the reset with --execute, then start the instances.

How to troubleshoot

Need to know where a group is in each partition

Check

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group payment-service
How to read it
CURRENT-OFFSET is the committed offset, LOG-END-OFFSET the end of the partition, LAG the difference. A CURRENT-OFFSET that does not move while the service logs processed records means commits are not happening.
A consumer started somewhere unexpected

Check

Consumer log at INFO: 'Found no committed offset for partition' followed by 'Resetting offset for partition ... to position'
How to read it
The group had no committed offset for that partition, so auto.offset.reset chose the start. Check the group id for a typo or a change, and whether the group was empty past the offset retention.
CommitFailedException in the logs

Check

The lines before it: a poll timeout or a rebalance, and the consumer metric `time-between-poll-max`
How to read it
The consumer was no longer a member when it tried to commit, usually because the batch took longer than max.poll.interval.ms. The records since the last successful commit are processed again by the new owner.
Suspect commits are slow or failing

Check

Consumer metrics `commit-latency-avg` and `commit-rate` (group consumer-coordinator-metrics)
How to read it
A commit rate near zero with steady consumption means nothing is being committed. High commit latency with RECORD or per-record manual acks means the commit round trip is limiting throughput.
The same records are processed after every restart

Check

Compare CURRENT-OFFSET from --describe with the last offset the service logged as processed
How to read it
If the committed offset equals the last processed offset instead of that offset plus one, the code commits the record's own offset. If it is far behind, commits are too rare or are failing.

Key takeaways

  • A commit stores one number per group and partition: the next offset to read. It does not acknowledge individual records.
  • Commit before processing gives at-most-once. Commit after processing gives at-least-once. Duplicates are handled in the processing, not in the commit.
  • Auto-commit runs from poll and is safe only when each batch is fully processed before the next poll or close.
  • Use commitAsync for routine commits and commitSync when partitions are revoked and at shutdown.
  • Spring turns auto-commit off and commits by AckMode. BATCH is the default, and the manual modes still commit a position, not a record.
  • auto.offset.reset applies only when there is no valid committed offset. Its default, latest, skips existing records for a new group.
  • Stop the group before resetting offsets, and read the dry run first.

Practical check

Small scenarios, one decision each. Pick what you would do, then read why.

Scenario 1 of 3

A plain-Java consumer uses default settings. To go faster, its loop submits every record from poll to a 16-thread executor and immediately polls again. After a node was killed, 37 orders were found that no instance ever processed. The consumer group shows zero lag.

What should change first?

Scenario 2 of 3

A reporting service ran for a year, then was switched off for 16 days during a migration. The broker uses default settings and the consumer has auto.offset.reset=latest. The topic retains 30 days of data. The service is started again with the same group id.

Where does it start reading?

Scenario 3 of 3

A Spring listener uses the default ack mode. max.poll.records is 500 and each record triggers an email. A pod is killed after it has sent emails for 320 records of a batch. Customers complain about receiving the same email twice.

Which change reduces the repeats to at most one record per crash without adding new components?

References

References: KafkaConsumer Javadoc (Kafka 4.4): offsets and consumer position, automatic and manual offset control ↗ · Apache Kafka 4.4 documentation: Consumer Configs ↗ · Apache Kafka 4.4 documentation: Distribution (consumer offset tracking, __consumer_offsets) ↗ · Apache Kafka 4.4 documentation: Broker Configs (offsets.retention.minutes, offsets.topic.*) ↗ · Apache Kafka 4.4 documentation: Basic Kafka Operations (managing consumer groups, --reset-offsets) ↗ · Apache Kafka 4.4 documentation: Monitoring (consumer metrics) ↗ · Spring for Apache Kafka 3.3 reference: Message Listener Containers (Committing Offsets, AckMode) ↗ · Spring for Apache Kafka 3.3 reference: Seeking to a Specific Offset ↗ · Spring Boot 3.4 reference: common application properties (spring.kafka.*) ↗ · Report a problem