Searchable content is a common component of many modern applications. But for applications that rely on a streams of new user-generated content how can we create a system that reliably and rapidly indexes searchable content and also provides ways to update, and act on changes to that data later on? This post will show you an architecture that can accomplish this using AWS services like Lambda, API Gateway, S3, Systems Manager Parameter Store, and DynamoDB. We’ll also use the third party provider Algolia for search.
Recently, I needed to put together system that would take new DynamoDB table items, queue up JSON messages in first-in-first-out order, and trickle them out to a 3rd party API over time.
I love not having to maintain things later so I decided to throw together a solution on top of managed AWS services. Here’s a high-level diagram of what I came up with:
Let’s take a look at how it all works.