Apache Kafka’s move from ZooKeeper to KRaft simplifies metadata management, boosts scalability, and accelerates failover. Learn how Kafka 3.9 on Amazon MSK leverages Raft consensus to streamline operations and support large-scale event streaming.
In this lab, we will create a Kafka producer application using AWS Lambda, which sends fake taxi ride data into a Kafka topic on Amazon MSK. A configurable number of the producer Lambda function will be invoked by an Amazon EventBridge schedule rule. Therefore, we are able to generate test data concurrently based on the desired volume of messages.
This series updates a real time analytics app based on Amazon Kinesis from an AWS workshop. Data is ingested from multiple sources into a Kafka cluster instead and Flink (Pyflink) apps are used extensively for data ingesting and processing. As an introduction, this post compares the original architecture with the new architecture, and the app will be implemented in subsequent posts.
This series re-implements a simple fraud detection application that is discussed in an AWS workshop titled AWS Kafka and DynamoDB for real time fraud detection. In part 1, I demonstrated how to develop the application locally, and the app will be deployed via Amazon Managed Service for Apache Flink in this post.
In this post, I will illustrate how to deploy data ingestion applications using Amazon MSK and MSK Connect. Fake order data will be generated using the MSK Data Generator source connector, and the Camel DynamoDB sink connector will be configured to consume the topic messages to ingest them into a DynamoDB table.
Kafka Connect is a tool for scalably and reliably streaming data between Apache Kafka and other systems. It can be used to build real-time data pipeline on AWS effectively. In this post, I will introduce available Kafka connectors mainly for AWS services integration. Also, developing and deploying some of them will be covered in later posts.
Glue Schema Registry provides a centralized repository for managing and validating schemas for topic message data and it can be utilized by many AWS services when building streaming apps. In this series, we discuss how to integrate Python Kafka producer and consumer apps In AWS Lambda with the Glue Schema Registry. In part 2, Kafka apps are deployed and their behaviour is discussed.
Glue Schema Registry provides a centralized repository for managing and validating schemas for topic message data and it can be utilized by many AWS services when building streaming apps. In this series, we discuss how to integrate Python Kafka producer and consumer apps In AWS Lambda with the Glue Schema Registry. In part 1, I illustrate the infrastructure and Kafka apps.
Streaming ingestion from Kafka (MSK) into Redshift and Athena can be much simpler as they now support direct integration. In part 2, we discuss an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Athena. We also use AWS SAM integrated with Terraform for developing a Lambda function locally.
Streaming ingestion from Kafka (MSK) into Redshift and Athena can be much simpler as they now support direct integration. In part 1 of the simplify streaming ingestion on AWS series, we discuss an end-to-end streaming ingestion solution using EventBridge, Lambda, MSK and Redshift. We also use AWS SAM integrated with Terraform for developing a Lambda function locally.
