Kafka - Michelin IT Engineering Blog

Sign in Subscribe

Kafka

A collection of 10 posts

Kafkagen: testing your Kafka clients made simple

Kafkagen: testing your Kafka clients made simple

If you ever: * used kcat, the kafka-console-producer or any other third-party CLI to produce Kafka messages, * developed a custom Kafka producer for you or your team to ease testing your Kafka client, * used any Kafka GUI like AKHQ, Kafbat and already produced messages with it, * struggled as a software engineer,

Kafka connectors migration feat. Kestra

Kafka connectors migration feat. Kestra

As described in our previous articles, we are leveraging Kafka widely at Michelin. Each factory in the group hosts its own Kafka cluster, representing 60+ clusters, to be able to work in complete autonomy. Another cluster, a Confluent Cloud cluster, serves the central needs. When topics need to be replicated

Processing Error Handling in Kafka Streams

Processing Error Handling in Kafka Streams

Introduction A major concern when developing Kafka Streams applications is handling processing errors. Processing errors occur when the implemented logic in a Kafka Streams application fails to process a record correctly. Let’s illustrate what processing errors are with a concrete example from our Supply Chain domain: a DeliveryBooked event

Data in motion, myth or reality

Data in motion, myth or reality

When I started to work in IT a couple decades ago, the center of the universe for a software engineer was databases, mostly relational ones. We were relying heavily on their transactional capabilities to ensure data consistency. Integrating an app with the rest of the world was quite a challenge.

Designing Kafka Streams Applications

Designing Kafka Streams Applications

For the past 4 years, our journey into the heart of Kafka's capabilities has been shaped by two pivotal concepts: Master Topologies and Micro Topologies. These conceptual frameworks have become the backbone of our Kafka Streams application design, offering a comprehensive and granular understanding of our end-to-end communication.

The Michelin Guide: an unexpected event driven use case

The Michelin Guide: an unexpected event driven use case

tl;dr content publication is a challenge for an institution like the Michelin guide. Event streaming between SaaS solutions using Kafka hosted in Confluent cloud was a solution for us to move from a 2 days delay publication to near real time. Although it was not easy especially from an

Moving from orchestration to choreography - Part 2

Moving from orchestration to choreography - Part 2

In the first article [https://blogit.michelin.io/choregraphy-or-orchestration-thats-the-question/] of this serie, we've been through the preliminary work to convince our organization to move from a central orchestrator (implemented with a Business Process Management tool) to a choreography of micro-services. This second article will focus on a critical

“The metamorphose” of our Information System by Implementing a distributed event streaming platform

“The metamorphose” of our Information System by Implementing a distributed event streaming platform

In this article, I want to share Michelin's experience on its first implementations of Kafka and distributed architectures, which is part of the early stages of implementing our EDA architectures. Our firsts experiences we had on a distributed platform.