Subscribe To Our Newsletter!

Stay up to date on the top trending threats as well as the top stories in Security, Networks, Cloud, IT Ops & AIOps.

Serverless Framework for Log Analysis: A Step by Step Guide

Log analysis is used to collect, index, and store massive amounts of data from any source deployed in the cloud.

Post by Tanuj Mitra In Clouds on Apr 14, 2019

WHY IS LOG ANALYSIS IMPORTANT?

In the modern world, business applications continue to evolve; the log data generated becomes enormous. Log analytics tools help in extracting the data as desired. The analysis will also help in deriving the metrics about an application and its performance over a period.

Log Analysis is used to collect, index, and store massive amounts of data from any source deployed in the cloud. Since each log file includes audit information, and we can use the dashboards to analyze the collected log data and compare results specific to business needs.

Further, log analytics tools can help in identifying the root cause of an issue and consequently give the admins a chance to prevent such issues from occurring in the future. When a problem occurs, the critical concerns are:

  1. Identifying the log file which contains the issue
  2. Locating the server
  3. Searching for the data (e.g., timestamp, version, etc.)

AWS SERVICES LEVERAGED FOR LOG ANALYTICS AND VISUALIZATION:

  • Amazon Simple Server Storage (S3) is a storage service to store and retrieve any amount of data.
  • Amazon Athena is a query service to analyze data directly from files stored in S3 using standard SQL statements.
  • Amazon QuickSight helps build interactive visualizations, perform ad-hoc analysis, and get useful business insights from various data sources hosted on the AWS infrastructure.

HOW TO BUILD A SERVERLESS ARCHITECTURE FOR LOG ANALYSIS?

The following are the steps for building the solution for log analytics on AWS.

Building the solution for log analytics on aws

STEP 1: UPLOAD YOUR LOG FILES TO S3

The logs generated are uploaded to S3 for further processing. Create an S3 bucket in your AWS account.

STEP 2: CREATE TABLES IN ATHENA

Athena is used to analyze the data by querying the source datasets.

Open the AWS Management Console and type ‘Athena’ in the AWS Services search box. Once you find Athena, click on ‘Get Started.’

Amazon Athena get started

Using the Query Editor, run the command CREATE DATABASE to create a new database. You can save the command by clicking the ‘Save as’ option for future use.

Amazon Athena create database

Once the query is executed, the new database will appear in the drop-down menu on the left side of your screen. Now select the database that you created.

Amazon Athena query editor

Create a new Table for the files in S3 as below:

Amazon Athena files in s3

Once you create the table, verify it by browsing for the table on the left-side panel.

MSCK REPAIR TABLE

To load all partitions of the table, run the command – MSCK REPAIR TABLE. After creating the table, you can run various queries to investigate your logs.
For, E.g., select * from

Athena QuickSight

After receiving confirmation on data access via Athena, the next step is to visualize the data using QuickSight.

STEP 3: VISUALIZING DATA IN QUICKSIGHT

Select ‘Quicksight’ from the AWS search bar.

Sign up for QuickSight

Select the relevant QuickSight edition based on your requirement. Here in this example, we will go with the standard version for the demo.

Create your QuickSight account

After selecting the QuickSight edition, click on ‘Continue.’ You will then be directed to the web page shown below. Fill the necessary details and click ‘Finish.’

QuickSight testing

After creating the QuickSight account from the QuickSight home page, click ‘Manage data.’

QuickSight manage data

Select the ‘New data set’ option as below.

QuickSight new data set

Now select the ‘Athena’ option from Data Sets.

Athena from data sets

For the data source name, enter the same name as the ‘Athena’ database and click ‘Validate’ to connect QuickSight to Athena. After validation, click ‘Create data source.’

New Athena data source

Select the database and table from the following window. Click on ‘Edit/preview data.’

Athena edit preview

Here you can change the following variables as below:

  1. Data type‘ of the data field
  2. Rename‘ the data field
  3. Exclude‘ a data field if you don’t need it

Athena data type

After completing the changes, click ‘Save and Visualize.’ You can now view the QuickSight dashboard, as depicted in the below diagram. Here, you can create your dashboard by adding visuals.

Save and visualize QuickSight

Choose Add on the application bar, and then choose Add visual. Select the fields to use from the Fields list, pane at left. Then create a visual by selecting a visual type.

You can also customize the visuals per your requirements.

  1. Creating
  2. Renaming
  3. Changing fields
  4. Changing visual layout

VPC flow logs

This is how we can leverage the AWS services to process, analyze, and visualize the logs generated from different sources of log data. If you want to know more about effective cloud management, visit our page today.

CONNECT WITH US

About Author

Tanuj is a storyteller whose ideas and snackable insights are in-sync with dynamic IT operations and modern networks. He likes to develop content that's smartly worded, clutter-breaking, and easy to digest.