So, a bunch of us Cevon’s were having a conversation on Slack the other week about a series of tasks we needed to do on a weekly basis as a part of our internal training program. In this, the first in a new series of articles, I’ll walk you through how we automated an annoying repetitive task using AWS serverless tools/services. In this article, I’ll outline the issue we are trying to address and a possible solution to the problem. We’ll finish off by getting a very rough version 0.1 of our core functionality up and running. In future articles, we’ll refactor our solution a little more and add additional functionality to streamline the solution even more.
The Problem Statement
Each Monday, the Cevo team gets together and conducts a BrownBag on Zoom, where we get together and teach each other about a particular topic. Because this provides an opportunity to enhance our skills, we like to make sure we record these sessions, so they are available to other Cevon’s who might not have been able to attend for various reasons (such as helping our customers). These recordings then need to be downloaded and stored on a Google Drive share, so they are available to the wider team. Once the recording has been uploaded to Google Drive, we then need to post a link to the recording to our brown bag Slack channel to let the team know the recording is available.
Breaking down the Problem
The first step in trying to address a problem such as this is to see if you can break it down into manageable tasks or steps. If we think about our problem statement, we can see that there are really four (4) major tasks that need to be undertaken to complete the series of tasks. These tasks would be:
1. Get notified when Zoom has finished transcoding the recording of the meeting and has the video ready for download.
2. Download the recording of the Zoom Meeting.
3. Upload the recording to the Cevo Google Drive environment
4. Post a message in a Slack channel with the URL to the recording once it’s finished copying
In this article, we’ll focus on solving the first task and building a system that can get notified when there is a new recording for us to download. In future articles, we’ll step through the remaining tasks until we have a complete solution.
Getting notified when a video is ready for download
The first task would be getting notified that there is a new Brown Bag video ready for download. There are a couple of options for how we could achieve this. We could:
– Wait for an email that Zoom will send us. But this would be difficult as we’d need to read and process the incoming emails to derive it’s the email from Zoom regarding a video ready for download.
– Poll the Zoom website or API to see if it’s ready and available for downloading yet. This isn’t an ideal solution and would be costly and inefficient to run.
– Just try to download the file 6 or 8 hours after the meeting. Zoom typically takes no longer than an hour to process video recordings, so we could simply delay the download for a set period. This isn’t ideal as it doesn’t allow for things to take longer than estimated
– See if we can set up some kind of automated notification for Zoom to let us know it’s ready (beyond just emailing us).
The last option is the most ideal as it should require less effort than grepping emails, be more reliable and faster turn around than simply delaying the download and is more efficient and easier on resources than constantly polling Zoom’s servers. So, how do we go about getting notified by Zoom?
Well, the first place to look is the Zoom Developer site and after a bit of digging, we can in fact see that Zoom offers us event notifications via the use of Webhooks (see https://marketplace.zoom.us/docs/api-reference/webhook-reference/ for more details). This means that we can simply set up a webhook endpoint and have it listen for the calls coming from Zoom.
So, What is a webhook?
At its simplest, a webhook is simply an endpoint (most commonly a REST endpoint… but not always) that a source application (in this case Zoom) can send event notifications to for the purpose of communication between applications. In our example, we will insert a hook into the Zoom application and have it send a notification to our AWS environment every time a recording is ready for download. We can then process that notification and use it to trigger a download job. Adding this “Hook” functionality is a common way for SaaS companies to enable integrations into their software without requiring large amounts of customisation on their end.
How do we build a webhook in AWS?
The easiest way to build and publish a webhook that applications can call is through the use of a lambda function linked up to an API gateway. We will set up an API gateway that will “Host” our endpoint and whenever it receives a request, it will trigger a lambda function. For the purposes of this walkthrough we’ll simply log the payload of the call to the console and return a 200 status code back to the source, but in the next article will implement the code to actually handle the call… For now, we just want to make sure we can receive the event coming from Zoom.
Step 1: Creating our Lambda Function
Firstly, we need to create the lambda function that will be responsible for handling the payload coming from Zoom.
– Open the Lambda Management Console which can be found in the Services menu under Compute
- From our Dashboard view, click the “Create Function” button located on the top right of the page. You can also find it via the “Functions” section listed on the left-hand side of the page.
- Next, we want to give our function a name and define the runtime. For the purposes of this example, we’ve used the name “PrintSamplePayload” and we’ll write it in “Node.js 14.x”. Once all of this is defined we can click “Create Function” (again, for the purposes of the example we are leaving all other settings at their default).
- For our code, all we’ll do is replace the “// TODO implement” line with “console.log(event);” as shown in the below screenshot. Go ahead and click “Deploy” once you’ve made the change.
And that’s it for the lambda function. When the lambda function gets called it will print the event payload to the console and return a 200 status code to the caller.
Step 2: Create the API Gateway that will host our Lambda function
Now that we have our Lambda function we need to create the API Gateway that will call it.
- Start by opening the API Gateway Management Console, which can be found in the Services Menu under “Networking & Content Delivery”.
- Seeing we want to create a REST API, we can simply click the “Build” button under REST API making sure NOT to select the private one (as we need our API Gateway to be publicly available so Zoom and communicate with it).
- On the Next Screen, we can select “New API” from the “Create new API” section, give our API a name (for this example we’ve gone with “ExampleZoomEndpoint”) and click “Create API”.
Our API is now created, but we require it to be able to accept some traffic… so for that, we need to add a method.
- From the Action’s menu… select “Create Method”
- Select “Any” from the dropdown list under our root (represented by the / symbol). This will simply ensure that regardless of the type of call to our endpoint, it will be forwarded to our Lambda function. Click the “tick” to confirm the selection.
- On the right-hand side of the page, we want to make sure that our “integration type” is set to “Lambda Function” and then we can start typing our Lambda function name in the type box provided. Autocomplete should make the task of getting the function name right a little easier. Click “Save” once you’re done
- At this point, you should be presented with a warning about the fact that you are giving API Gateway permission to invoke your lambda function. This is our expected behaviour, so we can simply click “OK” to this message. It’s important to note that this step must be taken separately when coding this type of solution with CloudFormation or SAM (Serverless Application Model).
Now that we have our API Gateway configured it’s time to deploy our configuration to an API Gateway Stage. This is what separates our development tasks from what’s actually running and accepting traffic. We can define as many stages as we need but for the purpose of validating we can receive event calls from Zoom, we’ll only need one stage. For more advanced applications you might need more than one stage (Dev, Test, QA, Pre-Prod, Prod) etc. To set up a Stage we can take a shortcut by simply triggering a deployment from the “Actions” menu.
- From the “Actions” menu, select “Deploy API”
- Here we can select “[New Stage]” as our Deployment Stage and give it the stage name of “dev”. We’ll skip the Stage description and Deployment description for the time being. Click “Deploy” to proceed.
After a moment we should be presented with our new “Invoke URL” which is the URL that the “dev” stage of our API Gateway is available on. This is the URL that we can pass to Zoom to receive events on. For the time being, we can simply open the link in a new window, and we should see the status code and body payload being returned by the lambda function we deployed in the previous step.
Now that we know that we can trigger our Lambda function by calling our API Gateway, the next step is to configure up our Zoom tenancy to call our endpoint when certain events occur… such as our recordings becoming available.
Step 3: Configuring Zoom to call our webhook
This is the part where we need to take a step away from AWS for a moment and focus our attention on the Zoom platform instead. We need to tell Zoom that we want them to call our webhook every time they finish processing a video recording.
- To start this we need to head over to the zoom developer site at https://developers.zoom.us
- Go ahead and “Sign In” with the account you use to log into zoom. For me, that’s with my Suite Login
- Once you’re logged in, click “Build App” from the top of the page
- For the moment, we are only after the webhook functionality so for now, we can simply click “Create” for a “Webhook Only” app type.
- For the purposes of this demo, we can call our App “RecordingDownloader” and click “Create” to continue.
- Next, we need to give Zoom some basic details about our application. Go ahead and populate the Basic and Developer Contact Information and click “Continue” (you can skip the Links fields for the moment).
- On the next screen, we are provided with our verification token which Zoom will attach to each notification request. As our API Gateway endpoint is public, anybody could send traffic to it… but we only want to be responding to messages originating from Zoom. This verification token gives us something we can use to verify it’s an authenticated notification. However, for the moment we’ll ignore the token as we simply want to verify we can receive notifications… We’ll add verification in the next article when we start processing the requests. For the moment, just take note that it exists.
Right now, all we want to do is to add an “Event Subscription” that will subscribe us to notifications when particular events within our Zoom account occur.
- To do this, simply tick the toggle next to “Event Subscriptions” and click “Add Event Subscription”
- Next, Give the Subscription a name and enter to invoke the URL we were presented when we published our API endpoint. Once that’s done we can go ahead and click on “Add Events”.
- On the “Event Types” page we can see all the different events that can occur within our Zoom Tenant. Things from meetings starting, billing and user account activity through to participants accepting/joining/leaving. You can start to see some possibilities around integration you could create, but for now, all we’re interested in is getting a notification when our recordings have been completed and are ready for download. For that, we need to tick the “All Recordings have completed” checkbox under the “Recording” section. Go ahead and click “Done”.
- Next, confirm that we have “1 event added” to our “Events” section, and we can go ahead and click “Continue” to save our changes.
At this point, we should now have an activated application within our Zoom account that will send our API Gateway endpoint a notification every time a recording is finished.
Step 4: Testing our Configuration
Now, all we need to do is go ahead and test our configuration and make sure that we get an output in our CloudWatch logs when we receive a notification from Zoom.
To do this, you’re going to want to start and finish a meeting that’s recorded. You can simply open a “New Meeting” and click “record”. When you end the meeting, the recording should automatically be generated. If all you’re doing is logging in, clicking record and waiting a few seconds before ending the meeting… you should get a notification within a couple of minutes.
After those few minutes have passed, it’s time to check out CloudWatch logs to see if the Lambda function has written anything to our logs.
- start by opening up “CloudWatch” which can be found under “Management & Governance” in the services’ menu of the AWS Management Console.
- Browse to “Log Groups” under “Logs” from the menu on the left-hand side
- Under “Log Groups” you should see a Log Group for the Lambda Function you created back in “Step 1” (in my case “PrintSamplePayload”) in the form of “/aws/lambda/%LambdaFunctionName%”. Click on the Log Group Name.
- There will be multiple log streams listed. A log stream is created for each set of executions of a lambda function (for example, each time a new version of the lambda function is deployed). You should see the most recent stream at the top of the list with the “Last Event Time” being similar to when Zoom finished processing your meeting recording. Click on the Log Stream to display the log output from the lambda function.
- And here we can see the console.log output of our Lambda function returning the payload that was sent to me from Zoom when they finished processing my meeting recording. This payload contains the URLs of the files I can download as well as the download token I’ll need to authenticate my request to download the file (certain fields in this screenshot have been blurred for security reasons).
Conclusion
And there we go, we have successfully managed to configure a webhook in our Zoom Tenant and have it send an event notification to our API Gateway… solving the first problem we outlined in our quest to automatically download our Brown bag sessions.
In the next article, I’ll walk through the process of how I use the payload coming from Zoom to actually trigger another function in AWS to download the recordings and save them in an S3 bucket for later transfer. We’ll also add some security to our solution in the form of payload verification using the tokens we were provided earlier in this piece. In future articles, I’ll outline the other actions and integrations that I’ve used to automate the entire end-to-end process.
