What I learned while studying for the GCP PDE exam

 

GCP PDE Badge

I recently (June 22nd, 2024) appeared for and passed the GCP Professional Data Engineer certification. [1] In this blog, I'd like to share what I've learned in studying for and taking the exam.

First off, here are the main GCP offerings that are tested on the exam:

  1. BigQuery: This is a data warehouse that can hold structured data. It is SQL compliant and you can perform data analysis with this tool. Despite seeming like a regular relational database, it is actually a columnar database. This database can go up-to petabyte scale!
  2. BigTable: Despite have 'Table' in the name, it's a NoSQL database. It is a non-relational database used for things like receiving large amounts of IoT/factory sensors' data, user's application data, etc. Got to learn about BigTable schema design (& especially row-key optimization)
  3. Cloud Storage: This service was covered even in the Cloud Associate Engineer exam but it's just a blob (Binary Large OBject) storge like Amazon S3 that is designed to store unstructured data like images, videos, audio files, etc. It has different storage classes & availability choices.
  4. Firestore: NoSQL document database used for mobile, web, & server deployments.
  5. Memorystore: Fully managed Redis & Memcached for sub-milliseconds latency.
  6. CloudSQL: Managed service for MySQL, PostgreSQL, & SQL server.
  7. Spanner: SQL database when you need global consistency for use cases like gaming, global finances, supply chain/retail, etc.
  8. Dataprep (by Trifacta): This tool let's you explore different datasets and create 'recipes' that let you transform data. It can be used for cleaning & transforming raw data into a format ready for analysis.
  9. Cloud Data Fusion: It is a date integration service that helps users build & manage data pipelines with GUI. In the visual interface, you drag-and-drop pre-built connectors that have the logic to do things like read from sources such as databases, perform calculcation, write to sinks, etc.
  10. Dataproc: For managed Hadoop & Spark clusters.
  11. Dataflow: This is GCP's managed service for Apache Beam which is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream processing.
  12. Cloud Composer: This is GCP's manged service for Apache Airflow which is an open-source workflow management platform for data engineering pipelines.
  13. Pub/Sub: This is like Apache Kafka, there are publishers & subscribers and topics to which messages are published. It can be done is pull/push manner and at least once delivery is guaranteed.
You can find the exam guide here. 

Cheat Sheet:

Quick reference for GCP tool selection


BigQuery & GoogleSQL:

Learned how it's important to reduce to the amount of data scanned to get results faster (& be charged less!). This can be done with Partitioned tables (segmenting data). You can partition based on an integer range values such as customer_id, time-unit column, or ingestion time. It's also possible to optimize queries using clustered tables where there's a user-defined column sort order.

It's also imperetive to optimize JOINs which can otherwise be resource-intensive. This can be done by using FILTER clauses (reducing data processed) and using WITH clauses. Also, avoid CROSS JOINs which produces cartesian product which can end up being a lot unnecessary rows!

There's a feature called 'EXPLAIN' which provides detailed query execution plan. We can use this to identify bottlenecks & optimize queries.

See this link to learn more about query optimization for BigQuery.

Machine Learning using BigQuery

BigQuery also has built-in capabilities to build, train, and deploy machine learning models directly within the BigQuery environment. You can create models with the 'CREATE MODEL' clause, train, evaluate, & use for Prediction.


Got to learn about linear regression which predicts continuous outcomes based on input features such as sales prediction and logistic regressions which is used for binary outcomes such as malignant/bening tumors.

Learned briefly about model evaluation and AUC-ROC (Area under the curve - receiver operating characteristic).



Dataproc (Hadoop & Spark):

It was very interesting to learn about Hadoop which was invented to deal with large data and the computing & storage limitations. Hadoop enables distributed storage (through HDFS) and processing (throught MapReduce).




Hadoop Distributed FileSystem is a distributed file system that stores data across multiple machines & provides high throughput access to data.

MapReduce is a programming model & processing engine for large-scale data processing. It breaks down a large task into smaller ones and processes them in parallel.
Hadoop uses YARN (Yet Another Resource Negotiator) for scheduling jobs. This is what enables distributed processing.
Apache Spark is an analytics engine for big data processing with modules. It does in-memory computing which makes it very fast.
 


Dataproc is a managed-GCP service what this means is that you just select how much resources (computing, memory, & storage) that you need and the underlying tools will be installed and provided for you. This reduces the amount of time to get the cluster up & running. Also, since we would be able to leverage GCP's massive infrastructure, we would get great scaling capabilities.

It was also fascinating to learn about GCP's Colossus file system and Jupiter network which has petabit-scale bi-sectional bandwidth! 😮

Dataflow (data processing pipelines):

Data processing pipelines are used to convert raw data into meaningful, usable information. for analytics workflows. There are batch pipelines (regular intervals) & stream pipelines (continuous). There are different types of loading, ETL, ELT, or EL. ETL should be used when the transformation takes significant amount of work such as performing some computing/logic/appending other information to derive insights, etc. while ETL should be used if you can get what you want with a simple query like, loading all sales data and getting cumulative sales for the day. EL can be used when there isn't any transformation required.




If you don't have an on-premises Hadoop/Spark clusters that you are migrating to GCP, it's recommended to use dataflow pipelines instead. Dataflow uses Beam programming model. 



It's a fully-managed service with automatic scaling and integrated with other GCP services for sources/sinks. It also has built-in monitoring and logging.

Summary:

These are just some of the services that I learned about and had my interest peeked by. There are a lot more services in GCP that enable different workflows. I think these tools are great to store & process data, derive insights, enable automation, and most importantly solve business problems. 

Thank you for reading! Let me know your thoughts in the comments.

---
Resources:

[1] https://cloud.google.com/learn/certification/guides/data-engineer

Comments

Post a Comment

Popular posts from this blog

Playing around with Dell R520 server

Image
After I got my first major tech job, I learned that a major part of the job would be working with Dell servers, setting up ESXi hosts, vCenter installation, etc. so I thought it would be a good idea to practice this at home in my own lab environment so I went and bought a Dell R520 server from the Facebook marketplace for around 300 CAD$. Let me explain how I set up the server. We'll go over the installation of ESXi in detail in another post. For now, let's assume that our server comes with ESXi pre-installed. Let's start by ensuring that the server is plugged in. Most servers come with 2 different PSUs (Power Supply Units). These are meant to go on different Power Strips for power outage redundancy. Once the server is plugged in, try to get your hands on an old monitor with a VGA cable, should cost about 20 CAD$ used. We also need a keyboard. In my case, ESXi was pre-installed so all I had to do was set up the hostname and IP address. Since the internal IP address in my ne
Read more

Experience Interviewing for an Infrastructure Engineer - Continuous Delivery position

Been a while since I got to write to this blog. Recently, I had the pleasure of interviewing for an Infrastructure Engineer - Continuous Delivery position for a company based in Dallas, Texas. I wanted to pen down my experience while the memory is fresh and possibly help others in the same pursuits. The first round was the typical HR round where the interviewer requested to know more about my experience in the DevOps / Infrastructure engineering field and gauge my interest in the position / learn about my background. It included a lot of open-ended questions such as can you please explain your role in your current position. The next round was considerably more interesting and included more pointed questions about my experience in particular tools and working platforms. The interviewer also took the time to explain the role and the need for this position.  After passing this round, I received a take-home exam, the exam question is as follows: Hi Srikar, We're excited to continue our
Read more

2023 Summer Reading List Summary

Image
This blogpost is going to be quite different from my regular posts. I would like to write something down by myself instead of just being a proxy for chatGPT like I did in the past few posts. Books I've read this summer: Thinking in Systems by Donella Meadows Extreme Ownership by Jockco Wilink and Leif Basin Financial Intelligence by Karen Berman Zero to One by Peter Thiel Stillness Is The Key by Ryan Holiday Let's talk about my takeaways book by book: Thinking In Systems , after reading this book, I try to think of everything happening around me in terms of a simple flow chart, on the one side is a 'source', then the other side is the 'output', & in the middle is a box that is an effect which acts on the source to produce the output. Here's an example from the book: Most systems along with having a 'source' and an output, also have feedback loops, they can broadly be classified into 2 types. One is a corrective loop, e.g. a thermostat on a an Air
Read more