DATABRICKS CERTIFIED DATA ENGINEER

Data Engineering is a very in demand role even touted to be the "real sexiest" job in the 21st century. Databricks is a company at the forefront of innovation in this space especially with its unified LakeHouse paradigm. I spent the last two weeks gaining my Databricks Certified Data Analyst and Databricks Certified DataEngineer Learning Pathways

About the test
This is a simple exam aimed at someone with close to six months of experience working with Databricks platform Minimally Qualified Candidate The minimally qualified candidate should be able to:

• Understand how to use and the benefits of using the Databricks Lakehouse Platform and its tools, including:
• Data Lakehouse (architecture, descriptions, benefits)
• Data Science and Engineering workspace (clusters, notebooks, data storage)
• Delta Lake (general concepts, table management and manipulation, optimizations)
• Build ETL pipelines using Apache Spark SQL and Python, including:
• Relational entities (databases, tables, views)
• ELT (creating tables, writing data to tables, cleaning data, combining and reshaping tables, SQL UDFs)
• Python (facilitating Spark SQL with string manipulation and control flow, passing data between PySpark and Spark SQL)
• Incrementally process data, including:
• Structured Streaming (general concepts, triggers, watermarks)
• Auto Loader (streaming reads)
• Multi-hop Architecture (bronze-silver-gold, streaming applications)
• Delta Live Tables (benefits and features)
• Build production pipelines for data engineering applications and Databricks SQL queries and dashboards, including:
• Jobs (scheduling, task orchestration, UI)
• Dashboards (endpoints, scheduling, alerting, refreshing)
• Understand and follow best security practices, including:
• Unity Catalog (benefits and features)
• Entity Permissions (team-based permissions, user-based permissions)

Duration
Testers will have 90 minutes to complete the certification exam. Questions
There are 45 multiple-choice questions on the certification exam. The questions will be distributed by high-level topic in the following way: Breakdown

• Databricks Lakehouse Platform – 24% (11/45)
• ELT with Spark SQL and Python – 29% (13/45)
• Incremental Data Processing – 22% (10/45)
• Production Pipelines – 16% (7/45)
• Data Governance – 9% (4/45)

Cost
Each attempt of the certification exam will cost the tester $200. Testers might be subjected to tax payments depending on their location. Testers are able to retake the exam as many times as they would like, but they will need to pay $200 for each attempt. Study strategy

• To study for this course I used my companie's partner account for the Data Engineering with Databricks V2 in the DB Academy
• Good notebooks to prepare are in this github account
• Databricks community cloud is good for practice but it does not have unity catalog and Delta live tables or DB SQL End point etc to practice
• Practice test

How much time dedicated
I didnt spend much time since I was familiar with it but I think about 2 weeks of dedicated effort of 3-4 hours by someone who is relatively new to databricks should be sufficient.
With this preparation you should be able to easily pass this test
Happy learning My certification

The crazy time of the year again

Its that time of the year again. This year has been really fruitful both on a personal and a professional front. My dad was diagnosed with colorectal cancer last year and this year he finally finished up with his surgery and chemo therapy and is much better. On a professional front I have moved back into a new contract role where I am helping australian farmers access premium export markets faster Ive worked on many technologies this year and love learning and experimenting with new things.I want to try to commit doing more writing couple of times a week on my blog. I think the protégé effect will help me learn more deeply and further my career goals quicker. For 2023 my goals will be to

participate in more hackathons

More opensource projects

Write more

As always learn and grow

As always take care and thanks for visiting my blog

Azure data factory copy with stored procedures

I have used Azure Data Factory a lot and also the copy activity. For a particular usecase recently I had to use a stored procedure as a sink. This was because we needed to add some business logic before the data was written to the Azure SQL DB To do this we will need to set up the following

• User defined table type. This is what the source is mapped to and will be a parameter for the stored procedure
• Stored procedure in SQL DB
• Sink data set to point to the database required

The stored procedure in the sink step of copy activity takes as input the user defined table type . User-defined table types are the predefined tables . The schema definition is created by the users In our case these hold temporary data that is sent from the copy activity source .




In my stored proc I insert into a new table after masking some columns.We can pretty much add any SQL Functions into this stored procedure.

To run the data pipeline we need to give execute privileges to the ADF to run Stored procedures and access user defined table types using GRANT EXEC ON TYPE::[schema].[typename] to [User]
and GRANT EXEC ON spname to [User]

Databricks Certified Associate Developer for Apache Spark 3.0 - Python

This weekend I finally bit the bullet and gave the databricks Assosciate Spark Developer.This is one of their speciality badges Specialty badges represent an achievement in a focus area

The exam is a 60 question multiple choice The exam covered

• Questions on Spark Architecture Concepts
• Questions on Spark Architecture Application and heaps of questions on
• Spark Programming Dataframe API

The exam costed 200USD I think this exam was worth giving I learnt some new things that I had not used in my day job using databricks

Columnar Versus Row based databases

This year one of my goals was to create more big data content and learn to read more research papers an art that I had forgotten about after I finished grad school. I will try to read atleast 2 papers each month in the Data science space and blog about them. I thought I'd start with something fundamental in big data Row and columnar databases .

Title and Author of Paper

C-Store: A column-oriented DBMS. Stonebraker et al.

Quick synopsis and explanation of the important bits

As a quick recap the paper talks about traditional data bases which implement record-oriented storage attributes of a record are placed contiguosly in storage.When writing to disk, a single write pushes all fields of the record to disk.However, for querying data a read optmised system may be more suited. In comes C-store In C-Store, fast reads are accomplished by storing data organized by column instead of row.Each column value or attribute is stored as a contiguous block. With a column store architecture DBMS can only read the column values required for the query rather than reading the whole row and bringing irrelevant attributes into memory.Since all data within a column is of uniform type the data can be compressed to a more compact form

A visual understanding of Columnar versus Row Suppose I have a an employee table

Employee	Location	Department
Jac	NSW	IT
Sally	WA	Sales

In a row oriented db it will be stored like this

Jac

NSW

IT

Sally

WA

Sales

In a column oriented db it will be stored like this

Jac

Sally

NSW

WA

IT

Sales

To summarize the key differences between Columnar versus Row based DBs are

Columnar	Row
Columns Stored contiguosly	Rows stored contiguously
OLAP Usecase	OLTP Usecase
Reads easy	Writes easy
Compresses better since all columns are of same data type	Does not compress as well

AZURE DEVOPS TO AWS cloud formation

Recently I was tasked with a job to deploy cloud formation templates from Azure Devops . Its a relatively easy task Let me walk you through the steps 1)What are we trying to do?

2)How to do it?
1)Set up Service connection In azure devops go to Project Settings>>Service Connection and create a new service connection.You will need to provide some details about an AWS User you will have to configure on the AWS End. Setup the AWS User with the required privileges to spin up cloud formation stacks

2)Setup azure pipelines

Many teams prefer to define their build and release pipelines using YAML.This YAML file in AzureDevops is called azure-pipelines.yaml file. In azure devops create a AWSShellScript task that basically executes your cloudformation deployment You can build your stack and deploy it using the SAM CLI

- task: AWSShellScript@1 

  displayName: Network layer 

  inputs:

    awsCredentials: WHATEVER_YOU_SETUP

    regionName: WHATEVER_REGION

    scriptType: 'inline'

    inlineScript: |

      sam deploy \

        --template TEMPLATEl\

        --no-confirm-changeset \

        --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND \

        --stack-name NetworkLayer

  

ML Project whats-cooking

After a brief hiatus from Machine learning generally ,this year I have decided to start looking at doing more ML.Learn from old kaggle competitions and participate in new ones.I plan to spend a few hours every week looking at old competitions just to hone my pretty rusty ML skills. For the first one this year I start out of with the WHATS COOKING competition.

In this competition we are asked to predict the category/cuisine of a dish from its ingredients. First step is EDA. I do a count of the ingredients to see if some cuisines are generally more verbose than others .Some do seem verbose potential here for a feature. I clean the data and stem it remove stop words remove commonly occuring words etc.On further investigation I find that when certain ingredients occur together than there probability of it being a certain cuisine increases drastically.So I find 2 words and 3 word combinations for every ingredient in the list.This could potentially be an interesting feature for the model.

After adding our 2 ingredient features I see that we have potentially too many features. We run a LInearSVC on this feature set since this is reasonably resistant to over-fitting. The final model gave me a score of 0.80882

Code