Serverless Framework for Log Analysis: A Step by Step Guide

Post by Tanuj Mitra Apr 14, 2019

WHY IS LOG ANALYSIS IMPORTANT?

In the modern world, business applications continue to evolve; the log data generated becomes enormous. Log analytics tools help in extracting the data as desired. The analysis will also help in deriving the metrics about an application and its performance over a period.

Log Analysis is used to collect, index, and store massive amounts of data from any source deployed in the cloud. Since each log file includes audit information, and we can use the dashboards to analyze the collected log data and compare results specific to business needs.

Further, log analytics tools can help in identifying the root cause of an issue and consequently give the admins a chance to prevent such issues from occurring in the future. When a problem occurs, the critical concerns are:

1. Identifying the log file which contains the issue
2. Locating the server
3. Searching for the data (e.g., timestamp, version, etc.)

AWS SERVICES LEVERAGED FOR LOG ANALYTICS AND VISUALIZATION:

• Amazon Simple Server Storage (S3) is a storage service to store and retrieve any amount of data.
• Amazon Athena is a query service to analyze data directly from files stored in S3 using standard SQL statements.
• Amazon QuickSight helps build interactive visualizations, perform ad-hoc analysis, and get useful business insights from various data sources hosted on the AWS infrastructure.

HOW TO BUILD A SERVERLESS ARCHITECTURE FOR LOG ANALYSIS?

The following are the steps for building the solution for log analytics on AWS.

STEP 1: UPLOAD YOUR LOG FILES TO S3

The logs generated are uploaded to S3 for further processing. Create an S3 bucket in your AWS account.

STEP 2: CREATE TABLES IN ATHENA

Athena is used to analyze the data by querying the source datasets.

Open the AWS Management Console and type ‘Athena’ in the AWS Services search box. Once you find Athena, click on ‘Get Started.’

Using the Query Editor, run the command CREATE DATABASE to create a new database. You can save the command by clicking the ‘Save as’ option for future use.

Once the query is executed, the new database will appear in the drop-down menu on the left side of your screen. Now select the database that you created.

Create a new Table for the files in S3 as below:

Once you create the table, verify it by browsing for the table on the left-side panel.

To load all partitions of the table, run the command – MSCK REPAIR TABLE. After creating the table, you can run various queries to investigate your logs.
For, E.g., select * from

After receiving confirmation on data access via Athena, the next step is to visualize the data using QuickSight.

STEP 3: VISUALIZING DATA IN QUICKSIGHT

Select ‘Quicksight’ from the AWS search bar.

Select the relevant QuickSight edition based on your requirement. Here in this example, we will go with the standard version for the demo.

After selecting the QuickSight edition, click on ‘Continue.’ You will then be directed to the web page shown below. Fill the necessary details and click ‘Finish.’

After creating the QuickSight account from the QuickSight home page, click ‘Manage data.’

Select the ‘New data set’ option as below.

Now select the ‘Athena’ option from Data Sets.

For the data source name, enter the same name as the ‘Athena’ database and click ‘Validate’ to connect QuickSight to Athena. After validation, click ‘Create data source.’

Select the database and table from the following window. Click on ‘Edit/preview data.’

Here you can change the following variables as below:

1. ‘Data type‘ of the data field
2. ‘Rename‘ the data field
3. ‘Exclude‘ a data field if you don’t need it

After completing the changes, click ‘Save and Visualize.’ You can now view the QuickSight dashboard, as depicted in the below diagram. Here, you can create your dashboard by adding visuals.

Choose Add on the application bar, and then choose Add visual. Select the fields to use from the Fields list, pane at left. Then create a visual by selecting a visual type.

You can also customize the visuals per your requirements.

1. Creating
2. Renaming
3. Changing fields
4. Changing visual layout

This is how we can leverage the AWS services to process, analyze, and visualize the logs generated from different sources of log data. If you want to know more about effective cloud management, visit our page today.

About the Author

Tanuj Mitra

Tanuj is a storyteller whose ideas and snackable insights are in-sync with dynamic IT operations and modern networks. He likes to develop content that's smartly worded, clutter-breaking, and easy to digest.