Leveraging event-driven architecture to digitalise fuel dockets at Flybuys

Cevo was engaged to work in collaboration with the team at Flybuys and help bring the initiative to market with our extensive AWS experience in cloud native engineering and DevOps best practices.

The solution, termed as the “Event Hub” was built on AWS utilising serverless technologies namely, EventBridge, Lambda and SQS.

Implementation

The entry point to the system is via an API Gateway which performs functions like validation of request body and headers, rate limiting, OAuth token and scope validation. Flybuys’ partners push events via the API Gateway and these get routed to a Lambda function in a private VPC. The Lambda function puts these events onto a common event bus. Events are inspected against a set of rules which are then forwarded to a secondary event bus referred to as the ‘partner bus’. The partner bus contains granular, domain-specific rules which route to SQS queues configured as targets. Downstream services poll the queues for events and process them in a timely manner.

Prevention of data loss

If the matching rule target, such as an SQS queue, is unavailable due to an outage, EventBridge will retry sending the event for up to 24 hours using exponential backoff. If the event fails to be re-sent to the target even after 24 hours, it would be delivered to dead-letter queue associated with the rule target. This is a relatively simple safeguard that has been implemented to prevent data loss due to service level failures.

In addition to using dead-letter-queues, the ‘event archive’ feature of AWS EventBridge has been utilised to aid in the replaying of events. The archive is attached to the common event bus and matches on all rules and stores them indefinitely. This safeguards against code-level bugs introduced into the system where events are incorrectly marked as being processed successfully. By archiving all messages, EventBridge can be instructed to replay the archived events by sending them back to the event bus once the bug has been fixed.

Continuous Integration and Deployment

The infrastructure components were created and deployed using AWS Cloud Development Kit (CDK). The CDK source code would build the CDK-based stacks and deploy them into the relevant AWS accounts. This combination accelerated the deployment process and helped in achieving a high level of repeatability, reliability and automation.

A goal of the project was to have zero downtime during the deployment of the Event Hub. This was achieved using AWS CodeDeploy to deploy the Lambda function using a canary deployment strategy. On deploying an update to the Lambda function, CodeDeploy transitions traffic between the old and new versions of the function in 10% increments over a 10 minute period. AWS CloudWatch Alarm monitors the error rates of the function during deployment and if triggered, will result in the deployment being automatically rolled back to the previous version of the function.

Observability, monitoring and tracing

Due to the distributed nature of the system, we endeavoured to make sure we had good visibility across events as they traversed the various systems. To aid this, multiple dashboards and monitors were built using a popular cloud-scale observability service.

The dashboards provide a near-real-time look at the health of the system by showing things such as the number of events being sent to and received from queues as well as whether the events have been delivered to any dead-letter-queues due to outages.

A number of monitors were also created that can trigger alerts when predefined thresholds are exceeded. This enables the IT operations team to triage and act upon any infrastructure related issues in a timely manner.