Kafka

Dead Letter Queue in Kafka Streams (KIP-1034)

Sébastien Viale, Loïc Greffier

24 Jul 2026 • 5 min read

Introduction

Kafka Streams 4.2.0 was released on February 16th, and it brings more than just improvements to exception handling and the dead letter queue (DLQ) mechanism.

With KIP-1034 included in this release, Kafka Streams now provides native DLQ routing support that directly integrates with:

DeserializationExceptionHandler
ProcessingExceptionHandler
ProductionExceptionHandler

Principle

The dead letter queue pattern is about routing Kafka events that failed to be processed to a dedicated topic.

Kafka Streams dead letter queue overview

In Kafka Streams, dead letter queues have historically relied on common practices:

Manual try-catch blocks in topology processors to make them exception-proof.
Use of split and branch operators to route failed records.
Enrich failed records with header metadata: source topic, partition, offset, exception... Enough to identify the problematic record and understand what happened.

KIP-1033 changed that a little by introducing a single ProcessingExceptionHandler that unifies processing error handling in one place. However, just like the DeserializationExceptionHandler and ProductionExceptionHandler handlers, it still requires a dedicated producer instance to route records to a dead letter queue. A built-in mechanism for that is still missing.

Origins

The idea for the Kafka Streams native DLQ contribution originally came from our Kstreamplify library.

In Kstreamplify, the design of the DLQ feature was based on the ProcessingResult and TopologyErrorHandler APIs. Each one respectively:

Marks failed and successful records in the output of stateless operations.
Splits the stream into two branches: one branch is forwarded to a given dead letter queue topic, while the other is forwarded downstream for further processing.

But this shows some limitations:

It relies on a manual try-catch mechanism.
It is limited to stateless operations (for processing exceptions).
It is limited to a single dead letter queue topic.
It is Avro-friendly by design only.

Building a dead letter queue mechanism on top of the Kafka Streams API is a good starting point. We have used it for multiple years, but it is limited and not flawless.

Interface Changes

KIP-1034 introduces the following changes.

For all exception handler interfaces, the previous handle method has been deprecated. They now provide a new handleError method:

Response handleError(final ErrorHandlerContext context, 
                     final Record<?, ?> record, 
                     final Exception exception);

With a new return type, Response:

class Response {
    private final Result result;
    private final List<ProducerRecord<byte[], byte[]>> deadLetterQueueRecords;
}

Through result and deadLetterQueueRecords attributes, it allows the stream to either stop or continue after an error, and to pass a list of records to be forwarded to any dead letter topic-partitions.

Dead Letter Queue Topic Name Property

The errors.dead.letter.queue.topic.name Kafka Streams property is a brand-new property introduced by KIP-1034. It acts as a dead letter queue enabler for all built-in exception handlers provided by the Kafka Streams API:

LogAndContinueExceptionHandler
LogAndFailExceptionHandler
LogAndContinueProcessingExceptionHandler
LogAndFailProcessingExceptionHandler
DefaultProductionExceptionHandler

It just needs to be set with a dead letter queue topic name:

properties.put(StreamsConfig.ERRORS_DEAD_LETTER_QUEUE_TOPIC_NAME_CONFIG, "dlq_topic");

When defined, built-in exception handlers will publish the original source record that triggered the exception, as it is at the input of the sub-topology, to the corresponding dead letter queue topic.

Example of a dead letter queue record with a missing `numberOfTires` property

Additionally, the dead letter record headers are enriched with metadata:

__streams.errors.exception: The exception name
__streams.errors.stacktrace: The exception stack trace
__streams.errors.message: The exception message
__streams.errors.topic: The source topic
__streams.errors.partition: The source partition
__streams.errors.offset: The source offset

Example of enriched dead letter queue record headers

The errors.dead.letter.queue.topic.name property is the most straightforward way to enable dead letter queue routing to a single topic, with no additional code.

Custom Exception Handler Implementation

By overriding the new handleError method in a custom exception handler implementation, it is possible to fully control what should be routed to the dead letter queue and what should not.

Let's take the following processing error handling logic:

Diagram for CustomProcessingExceptionHandler

A custom implementation for that would give:

public class CustomProcessingExceptionHandler implements ProcessingExceptionHandler {
    @Override
    public Response handleError(ErrorHandlerContext context, Record<?, ?> message, Exception exception) {
        if (message.value() instanceof DeliveryBooked deliveryBooked && deliveryBooked.numberOfTires() == null) {
            return Response.resume(List.of(new ProducerRecord<>(
                    "null-number-of-tires-dlq-topic", context.sourceRawKey(), context.sourceRawValue())));
        }

        if (exception instanceof InvalidDeliveryException) {
            return Response.resume(List.of(new ProducerRecord<>(
                    "invalid-delivery-dlq-topic",
                    ((String) message.key()).getBytes(),
                    ("Invalid deliveryBooked " + ((DeliveryBooked) message.value()).deliveryId()).getBytes())));
        }

        if (context.processorNodeId().equals("select-key-processor")) {
            return Response.resume(List.of(new ProducerRecord<>(
                    "select-key-processor-dlq-topic", context.sourceRawKey(), context.sourceRawValue())));
        }

        return Response.resume(List.of(new ProducerRecord<>(
                "default-dlq-topic",
                ((String) message.key()).getBytes(),
                ("An exception occurred " + exception.getMessage()).getBytes())));
    }
}

In custom implementations, the errors.dead.letter.queue.topic.name configuration is no longer used automatically. It is up to the implementation to explicitly reuse it through an override of the configure method.

Compared to the built-in exception handlers and the errors.dead.letter.queue.topic.name property, custom implementations make it possible to:

Route records to different dead letter queue topics.
Implement conditional routing based on the error handler context, the failed record, and the exception.
Choose the content of the dead letter record. It is possible to reuse the original source key and value from the error handler input context, or to use a different key and value.

Code Sample

A small code sample is available on GitHub:

It demonstrates:

The use of the errors.dead.letter.queue.topic.name property with the built-in LogAndContinueProcessingExceptionHandler.
A custom dead letter queue routing logic for processing exception handler.