Request a Demo
Blog

Do it later, do it better: Lessons learned from moving to a serverless infrastructure

The advertising industry has seen a significant shift in recent years. In a time when adblocking remains a prominent challenge for the industry and the need to get the right message across to the right consumer becomes even more pressing, the industry has continued to place an increasing empathise on data.

As we have seen, audiences have become more complex and distracted and it has become more difficult for our clients to break through the noise and engage with their consumers. However, as the challenge of reaching increasingly complex audiences intensifies, advertisers have a hidden weapon they’re able to utilise in order to deliver highly-targeted campaigns that provide genuine value and return.

With the sheer amount of data being processed daily and game-changing technologies being utilised to process, analyse and evaluate it, advertisers are able to leverage this insight to understand what matters to consumers and use it as a tool to identify and reach audiences. Advertisers are able to use the data to analyse consumer behaviour; whether this is related to their shopping habits or not, to understand their lives and influences.

Never has there been more information about what people want and why, nor the channels through which to reach them. Therefore, being able to process more data at a faster rate, the more opportunities we have to take action, generate strategic analytics and make real-time decisions for more than 1000 clients worldwide.

Lesson 1 - Building a Scalable Data Analytics Tool/A semi-synchronous infrastructure not scalable

Whilst the ability to track and process vast quantities of data is essential for our clients looking to deliver effective advertising campaigns, it is no easy task. Using a semi-synchronous infrastructure, Headway processed every individual data point but this system was impossible to scale due to the amount of data processed every day.

In order to make the process more scalable, we were able to build an asynchronous and cloud based infrastructure using Amazon Simple Service Queue (SQS) that would split each procedure of tracking and analysing the data into two separate tasks.

This asynchronous approach meant that we were able to split each procedure to happen at a different time to operate independently of each other; meaning the quantity of data being tracked and analysed can be scaled at a greater rate.

Lesson 2 - Analysing the Data in Real-Time/Moving to a serverless infrastructure

The asynchronous system we put in place carried out each process at a different time and allowed us to analyse large quantities of data in almost real-time. However, with our clients so reliant on effective data analytics, almost real-time wasn’t efficient enough as we worked towards improving latency by shortening the gap between the moment we receive the data to the moment this data is available on analytics.

As a result of wanting to improve latency and the speed at which data can be analysed, we knew that we needed to switch from an SQS based architecture to a serverless infrastructure using AWS Lambda functions connected to AWS Kinesis that can carry out processes as necessary. This new system is able to prioritise tasks and focus on what is most critical, leaving non-critical jobs for later in an asynchronous and event-driven way; meaning that data can be analysed in real-time, improving latency to less than 100 milliseconds and, due to a serverless infrastructure, can be scaled infinitely.

Furthermore, using the asynchronistic infrastructure implemented within the technology, we are now able to postpone less important tasks and prioritise more critical jobs; meaning that the less vital actions can be done later and the crucial tasks can be completed faster.

In an age where ad blocking remains a prominent challenge for the industry, delivering the right message to your audiences in the right way has never been more important. Data remains a valuable asset for reaching audiences, optimise their campaigns and maximise their revenue. However, without the ability to track and analyse data in real-time, advertisers could be missing out on valuable opportunities to take action and make strategic decisions instantly.

This blog post was inspired by Lucas Ceballos’ talk “Do it Later, Do it Better” as part of his workshop at leading tech conference, Nerdearla, that took place in Buenos Aires, Argentina. You can watch the full talk here.

  • Share This
  • Google plus
  • Linkedin
  • Twitter
  • Facebook
  • Whatsapp
  • Mail