Amazon EventBridge enables businesses to create powerful and adaptable event-driven architectures that can help them be more responsive, agile, and competitive in today’s fast-paced environment. Some of the key benefits of EventBridge include:
- Reduction in tight coupling where event producers and consumers don’t need to know each other, and consumers can act accordingly when they’re interested in the event.
- Simplified event routing where you can create a centralised event bus that can receive events from wide variety of sources, including AWS Services, SaaS applications and custom applications
- With the right architecture, EventBridge is an asynchronous infrastructure based around events that allows more resilience to failure. In a synchronous API, an entire system can be affected by a single failed event.
AWS S3 Event Notifications are a way for Amazon S3 to automatically notify other AWS services when certain events occur within an S3 bucket. These notifications can be configured to send events to SNS Topics, SQS Queues and Lambda functions. This approach is an ideal solution for small to medium scale applications, however its use case may be limited when you’re working with enterprise-scale applications.
The launch of S3 Event Notifications with EventBridge is a long awaited match in the cloud, which makes enterprise-scaling easier with key benefits including:
- Advanced filtering
- Multiple sources and destinations
- Access to fast and reliable invocations
In this blog post, we will explore how AWS S3 Event Notifications work and how you can use them with AWS EventBridge to build automated workflows that respond to changes in your S3 buckets.
S3 Event Notification with AWS EventBridge
Advanced Filtering
Using event patterns, Amazon EventBridge enables declarative content filtering. You can use content filtering to create complex event patterns that only match events that meet very precise criteria. You can, for example, create an event pattern that matches an event if a field of the event is within a specific numeric range, if the event originates from a specific IP address, or if a specific field does not appear in the event JSON.
You can filter metadata fields like object size using numeric matching. The following filter policy only delivers events to the target if an object has a size of less than 52,428,800 bytes (50mb).
When you’re required to filter events with file extensions like .json, you can do so by using suffix matching.
Or, you may want to filter by date using prefix matching. In the example below, any filename starting with "2022-12" will pass the filter test.
There are more event filtering patterns that you can use like IP address matching and exists matching. Find out more in the official documentation here.
Solution Design
You can trigger an event not just from the S3 source, but from other AWS services too. For instance, if you have your own application, you can have custom events, or trigger events, from SaaS apps.
These events will pass through EventBridge buses. Buses can fall into any of the following categories:
- Default event bus
- Custom event bus
- SaaS event bus
Event targets can be routed to 18 AWS services including Step Functions, Kinesis Data Firehose, Kinesis Data Streams, and HTTP targets via API Destinations.
This flexibility allows your application to easily create a fan-out mechanism which can be used in an enterprise-scale scenario where independent teams can work on processing their own event.
Fast, Reliable Invocation
S3 provides at-least-once delivery of events to EventBridge, which makes your application more reliable. Pattern matching is instant and direct, which makes the whole process fast.
In case of an error or if there’s a new target on your event bus, there are two EventBridge features for archiving and replaying events that you could use.
Business-related use cases
While the demonstration in the next section is a simple example of how you can use S3 Event Notifications with EventBridge, businesses can use EventBridge for accomplishing real world problems, such as:
- Real-time data processing: EventBridge enables real-time automated data processing by triggering events when new objects are added to a bucket. This may come in handy for processing log data or studying consumer behaviour, for example.
- Having loosely coupled applications: Applications can be separated from one another by using EventBridge as a communication bus. This enables them to scale independently and lessens the effect of changes to one application on the others.
- Backup and Disaster Recovery: EventBridge may automate the replication of data to a secondary location by triggering events when objects are created, changed, or deleted from a bucket, offering a backup and disaster recovery option.
- Compliance and Governance: By combining AWS S3 Notification and AWS EventBridge, compliance and governance issues can be resolved. EventBridge can automate the auditing and tracking of changes by triggering events when objects are added, updated, or deleted from a bucket, offering an immutable record of all data-related operations.
Demonstration
Objective
The goal of this demonstration is to set up an event pipeline in which uploading a new file with the filename extension .json to an S3 bucket triggers an event that is picked up by EventBridge and sent to the CloudWatch log group as a target.
S3 EventBridge configuration
You start by creating an S3 bucket named cevo-s3-eventbridge. To enable the EventBridge notifications, open the S3 console, select the bucket, open the Properties tab and scroll down to Event Notifications, click the Edit button under Amazon EventBridge.
Select the On radio button and save the changes.
AWS EventBridge configuration
In the AWS console, open the Amazon EventBridge page, click the Rule link on the left hand menu to open up the Rules page, in the Event Bus dropdown make sure you’re on the default bus and click the Create Rule button.
Step 1: Define rule detail
Have the rule name as FilterJsonFiles and make sure that the Event bus is default and hit the Next button.
Step 2: Build event pattern
Scroll down and select the Custom pattern (JSON editor) under the Creation method. Copy the JSON pattern below into the text area. Hit the Next button to go to the next step.
The rule says that it’s capturing events from the AWS service aws.s3, the meta details of any new Object Created in the S3 bucket cevo-s3-eventbridge with an extension of .json will be sent to the selected targets.
Step 3: Select target(s)
This step is where you can add a target. To set up CloudWatch as your target, select the AWS Services radio button as the Target type, select the CloudWatch log group in the Select a target dropdown then type in cevo-s3-eventbridge in the Log Group.
Steps 4 (Configure tags) and 5 (Review and create)
Hit the Next button and complete the tags in Step 4 then go to Step 5. Once reviewed, you can now create the rule.
Testing
This test ensures that S3 event Notification calls EventBridge, that the filtering works, and that events are sent to the selected target.
Step 1: Upload files to S3 bucket
Open the cevo-s3-eventbridge S3 bucket from the console and upload files with an extension of .json and .txt. In the example, cevo.json and hello.txt files were uploaded.
Step 2: Checking Amazon EventBridge Rules Monitoring
Open the Amazon EventBridge page and click the Rules from the left hand menu.
Click the FilterJsonFiles from the list of rules to view the Rule details. In the next page, click the Monitoring tab to see the Invocation and TriggeredRules metrics. You can see that there’s only one Invocation and one count of TriggeredRules because you are only expecting cevo.json to pass the rule filter test.
Step 3: Checking the logs in CloudWatch
Open the CloudWatch log group /aws/events/cevo-s3-eventbridge
Click the latest item from the log stream list
In the log, you can only see the details of cevo.json and not the hello.txt. Further proving that the event filtering works and our test is passing.
Check the official documentation to learn more about the EventBridge event S3 message structure.
Summary
In this article, we explored valuable insights into the real world applications, advanced features, and benefits of using S3 Event Notification with EventBridge.
Through a detailed demonstration, you can successfully create an S3 bucket that sends events to EventBridge, configure an EventBridge rule with CloudWatch as a target, and effectively capture the S3 event message in CloudWatch logs.
With a solid understanding of how this powerful cloud integration works, it’s time to take your architecture to the next level by embracing an event-driven approach. By leveraging the knowledge you’ve acquired, you can shift and uplift your architecture to fully realise the potential of event-driven systems.
