Apache Kafka’s move from ZooKeeper to KRaft simplifies metadata management, boosts scalability, and accelerates failover. Learn how Kafka 3.9 on Amazon MSK leverages Raft consensus to streamline operations and support large-scale event streaming.
This series re-implements a simple fraud detection application that is discussed in an AWS workshop titled AWS Kafka and DynamoDB for real time fraud detection. In part 1, I demonstrated how to develop the application locally, and the app will be deployed via Amazon Managed Service for Apache Flink in this post.
Apache Flink is widely used for building real-time stream processing applications. On AWS, Kinesis Data Analytics (KDA) is the easiest option to develop a Flink app as it provides the underlying infrastructure. Re-implementing a solution from an AWS workshop, this series of posts discuss how to develop and deploy a fraud detection app using Kafka, Flink and DynamoDB. Part 1 covers local development using Docker while deployment via KDA will be discussed in part 2.
Glue Schema Registry provides a centralized repository for managing and validating schemas for topic message data and it can be utilized by many AWS services when building streaming apps. In this series, we discuss how to integrate Python Kafka producer and consumer apps In AWS Lambda with the Glue Schema Registry. In part 2, Kafka apps are deployed and their behaviour is discussed.
Glue Schema Registry provides a centralized repository for managing and validating schemas for topic message data and it can be utilized by many AWS services when building streaming apps. In this series, we discuss how to integrate Python Kafka producer and consumer apps In AWS Lambda with the Glue Schema Registry. In part 1, I illustrate the infrastructure and Kafka apps.
Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. In this post, we’ll build a Hudi DeltaStramer app on Amazon EMR and use the resulting Hudi table with Athena and Quicksight to build a dashboard.
Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. In this post, we’ll build CDC with Amazon MSK and MSK Connect.
Change data capture (CDC) on Amazon MSK and ingesting data using Apache Hudi on Amazon EMR can be used to build an efficient data lake solution. As a starting point, we’ll discuss the source database and CDC streaming infrastructure in the local environment.
