Columnar Versus Row based databases

This year one of my goals was to create more big data content and learn to read more research papers an art that I had forgotten about after I finished grad school. I will try to read atleast 2 papers each month in the Data science space and blog about them. I thought I'd start with something fundamental in big data Row and columnar databases .

Title and Author of Paper

C-Store: A column-oriented DBMS. Stonebraker et al.

Quick synopsis and explanation of the important bits

As a quick recap the paper talks about traditional data bases which implement record-oriented storage attributes of a record are placed contiguosly in storage.When writing to disk, a single write pushes all fields of the record to disk.However, for querying data a read optmised system may be more suited. In comes C-store In C-Store, fast reads are accomplished by storing data organized by column instead of row.Each column value or attribute is stored as a contiguous block. With a column store architecture DBMS can only read the column values required for the query rather than reading the whole row and bringing irrelevant attributes into memory.Since all data within a column is of uniform type the data can be compressed to a more compact form

A visual understanding of Columnar versus Row Suppose I have a an employee table

Employee	Location	Department
Jac	NSW	IT
Sally	WA	Sales

In a row oriented db it will be stored like this

Jac

NSW

IT

Sally

WA

Sales

In a column oriented db it will be stored like this

Jac

Sally

NSW

WA

IT

Sales

To summarize the key differences between Columnar versus Row based DBs are

Columnar	Row
Columns Stored contiguosly	Rows stored contiguously
OLAP Usecase	OLTP Usecase
Reads easy	Writes easy
Compresses better since all columns are of same data type	Does not compress as well

AZURE DEVOPS TO AWS cloud formation

Recently I was tasked with a job to deploy cloud formation templates from Azure Devops . Its a relatively easy task Let me walk you through the steps 1)What are we trying to do?

2)How to do it?
1)Set up Service connection In azure devops go to Project Settings>>Service Connection and create a new service connection.You will need to provide some details about an AWS User you will have to configure on the AWS End. Setup the AWS User with the required privileges to spin up cloud formation stacks

2)Setup azure pipelines

Many teams prefer to define their build and release pipelines using YAML.This YAML file in AzureDevops is called azure-pipelines.yaml file. In azure devops create a AWSShellScript task that basically executes your cloudformation deployment You can build your stack and deploy it using the SAM CLI

- task: AWSShellScript@1 

  displayName: Network layer 

  inputs:

    awsCredentials: WHATEVER_YOU_SETUP

    regionName: WHATEVER_REGION

    scriptType: 'inline'

    inlineScript: |

      sam deploy \

        --template TEMPLATEl\

        --no-confirm-changeset \

        --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \

        --stack-name NetworkLayer

  

ML Project whats-cooking

After a brief hiatus from Machine learning generally ,this year I have decided to start looking at doing more ML.Learn from old kaggle competitions and participate in new ones.I plan to spend a few hours every week looking at old competitions just to hone my pretty rusty ML skills. For the first one this year I start out of with the WHATS COOKING competition.

In this competition we are asked to predict the category/cuisine of a dish from its ingredients. First step is EDA. I do a count of the ingredients to see if some cuisines are generally more verbose than others .Some do seem verbose potential here for a feature. I clean the data and stem it remove stop words remove commonly occuring words etc.On further investigation I find that when certain ingredients occur together than there probability of it being a certain cuisine increases drastically.So I find 2 words and 3 word combinations for every ingredient in the list.This could potentially be an interesting feature for the model.

After adding our 2 ingredient features I see that we have potentially too many features. We run a LInearSVC on this feature set since this is reasonably resistant to over-fitting. The final model gave me a score of 0.80882

Code