When google analytics is not enough

One of our clients – a market leading telecommunication company – wanted to implement applications ( both web and mobile) for managing clients’ accounts and clients’ services accounts(self-service). The web application was to be available in many domains (multi-brand application).

A traditional approach based on Google Analytics, Funnel.io or Mixpanel did not fulfil the requirements since most of the platforms allow only to export raw event data while the company needed advanced reports which were to cover many aspects of the business, including:

  • customer transaction history (payments, transactions, due dates)
  • data used for segmentation (profile data, mobile phone)
  • omni-channel oriented solution (possible to join data from other communication sources)

This architecture was to collect events and predict customers subsequent possible actions on any channel.

The architecture

User events came from many sources (different web applications, mobile devices). Each source of events uses Javascript or native mobile technologies to send REST messages with user data.

The goal was to receive data without any delay and push messages for future transformation. The architecture based on the Amazon Web Services stack was built on following components:

  • Amazon API Gateway – the entry point for requests, which expose REST API for applications and web pages. The REST API can be executed from any software. The API Gateway supports simple anonymous authentication as well as advanced (tokens, API ID’s). The message resized by the API Gateway is passed to Amazon Kinesis Stream
  • Amazon Kinesis – this part of the system provides three different components:

Amazon Streams – The role of the component is to receive data messages and build the execution queue. The benefit of this component is scalability. Depending on the traffic the component could be scaled from one to 126 instances/

  • Amazon Kinesis Analytics – The component transforms messages and pass to data sinks. The idea behind Kinesis Analytics is to provide the quasi-SQL mechanism for stream transformation, filtering, and aggregations.
  • Amazon Firehouse – as a delivery method which pushes data into Amazon S3 and to Redshift in parallel,
  • Amazon Redshift – data warehouse which provides distributed SQL engine which can be scaled from one to 250 instances. The platform is supported by many commercial and Open Source business intelligence platforms,
  • Amazon QuickSight – Business Intelligence platform for data visualization and analysis.

Benefits

The overall architecture could be set up on open source stack (Apache Kafka, Postgresql or Green Plum). But you don’t need to care about many aspects of Amazon including:

  • hardware installation, configuration, and problems,
  • operating system licensing, setup and maintenance,
  • core software installation and support

The benefits come from the scalability of the whole solution, which covers many scenarios:

  • when your business is growing,
  • when your business needs periodic reports (monthly, annual),
  • when your business getting down (because you are very near to bankrupt or your business is seasonal),
  • when you have a brilliant idea how to use a data to create new business and you need more power to verify it.