Kafka - Michelin IT Engineering Blog

Sign in Subscribe

Kafka

A collection of 12 posts

Kafka Consumption, Made Simple and Safe

Kafka Consumption, Made Simple and Safe

Consuming Kafka events to feed an external system is one of the most common integration patterns. At Michelin, many applications use it daily. Making it reliable in production is harder than it looks. Let's walk you through the solutions we explored, and the one we built: Kafka Retryable Consumer.

Kafka Streams Dead Letter Queue in Spring Kafka 4.1

Kafka Streams Dead Letter Queue in Spring Kafka 4.1

Introduction Spring Kafka 4.1 raises the Apache Kafka baseline to 4.2.0. With this release, it introduces Spring-friendly integration for Kafka Streams KIP-1033 (processing exception handling) and KIP-1034 (dead letter queue in Kafka Streams), both originally proposed by Michelin and available respectively since Apache Kafka 3.9 and

Kafkagen: testing your Kafka clients made simple

Kafkagen: testing your Kafka clients made simple

If you ever: * used kcat, the kafka-console-producer or any other third-party CLI to produce Kafka messages, * developed a custom Kafka producer for you or your team to ease testing your Kafka client, * used any Kafka GUI like AKHQ, Kafbat and already produced messages with it, * struggled as a software engineer,

Kafka connectors migration feat. Kestra

Kafka connectors migration feat. Kestra

As described in our previous articles, we are leveraging Kafka widely at Michelin. Each factory in the group hosts its own Kafka cluster, representing 60+ clusters, to be able to work in complete autonomy. Another cluster, a Confluent Cloud cluster, serves the central needs. When topics need to be replicated

Processing Error Handling in Kafka Streams

Processing Error Handling in Kafka Streams

Introduction A major concern when developing Kafka Streams applications is handling processing errors. Processing errors occur when the implemented logic in a Kafka Streams application fails to process a record correctly. Let’s illustrate what processing errors are with a concrete example from our Supply Chain domain: a DeliveryBooked event

Data in motion, myth or reality

Data in motion, myth or reality

When I started to work in IT a couple decades ago, the center of the universe for a software engineer was databases, mostly relational ones. We were relying heavily on their transactional capabilities to ensure data consistency. Integrating an app with the rest of the world was quite a challenge.

Designing Kafka Streams Applications

Designing Kafka Streams Applications

For the past 4 years, our journey into the heart of Kafka's capabilities has been shaped by two pivotal concepts: Master Topologies and Micro Topologies. These conceptual frameworks have become the backbone of our Kafka Streams application design, offering a comprehensive and granular understanding of our end-to-end communication.

The Michelin Guide: an unexpected event driven use case

The Michelin Guide: an unexpected event driven use case

tl;dr content publication is a challenge for an institution like the Michelin guide. Event streaming between SaaS solutions using Kafka hosted in Confluent cloud was a solution for us to move from a 2 days delay publication to near real time. Although it was not easy especially from an

Moving from orchestration to choreography - Part 2

Moving from orchestration to choreography - Part 2

In the first article [https://blogit.michelin.io/choregraphy-or-orchestration-thats-the-question/] of this serie, we've been through the preliminary work to convince our organization to move from a central orchestrator (implemented with a Business Process Management tool) to a choreography of micro-services. This second article will focus on a critical

“The metamorphose” of our Information System by Implementing a distributed event streaming platform

“The metamorphose” of our Information System by Implementing a distributed event streaming platform

In this article, I want to share Michelin's experience on its first implementations of Kafka and distributed architectures, which is part of the early stages of implementing our EDA architectures. Our firsts experiences we had on a distributed platform.