<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Certification on Fabio Lopes</title><link>https://fabiolopes.page/tags/certification/</link><description>Recent content in Certification on Fabio Lopes</description><generator>Hugo -- gohugo.io</generator><language>en-US</language><copyright>Fabio M. Lopes</copyright><lastBuildDate>Fri, 30 Oct 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://fabiolopes.page/tags/certification/index.xml" rel="self" type="application/rss+xml"/><item><title>Google Cloud Professional Data Engineer: A Practical Study Guide and Service Reference</title><link>https://fabiolopes.page/p/preparation-for-google-cloud-professional-data-engineer-pde-exam/</link><pubDate>Tue, 04 Feb 2025 00:00:00 +0000</pubDate><guid>https://fabiolopes.page/p/preparation-for-google-cloud-professional-data-engineer-pde-exam/</guid><description>&lt;p&gt;The &lt;a class="link" href="https://cloud.google.com/learn/certification/data-engineer" target="_blank" rel="noopener"&#10; &gt;Google Cloud Professional Data Engineer (PDE) certification&lt;/a&gt; covers a broad range of subjects: data storage, batch and streaming processing, orchestration, security, governance, machine learning, and operating data systems over time.&lt;/p&gt;&#10;&lt;p&gt;When I first put together my PDE preparation notes, I wanted a single place where I could compare services, review concepts, and revisit scenarios that could appear in the exam. The notes eventually became too long for the article about my personal preparation experience.&lt;/p&gt;&#10;&lt;p&gt;This companion guide keeps that original objective while making the reference easier to scan. It is organized around the decisions a data engineer might need to make, rather than a long sequence of product descriptions. It combines topics from my original 2025 notes with terminology and documentation updates reviewed in October 2026. &lt;strong&gt;It is an independent study aid, not an official exam guide or a guarantee of exam coverage.&lt;/strong&gt;&lt;/p&gt;&#10;&lt;p&gt;For my personal preparation process and the learning resources I used, see my companion article: &lt;a class="link" href="https://fabiolopes.page/p/preparing-for-the-google-cloud-professional-data-engineer-exam-my-study-approach-and-resources/" &gt;Preparing for the Google Cloud Professional Data Engineer Exam: My Study Approach and Resources&lt;/a&gt;.&lt;/p&gt;&#10;&#10; &lt;blockquote&gt;&#10; &lt;p&gt;&lt;strong&gt;How to use this guide:&lt;/strong&gt; Start with the &lt;a class="link" href="https://cloud.google.com/learn/certification/data-engineer" target="_blank" rel="noopener"&#10; &gt;official exam guide&lt;/a&gt;, then use the sections below to identify areas you cannot yet explain in your own words. Try a lab or small experiment, and finally practice answering &lt;em&gt;why&lt;/em&gt; one service might fit a given scenario better than another.&lt;/p&gt;&#10;&#10; &lt;/blockquote&gt;&#10;&lt;h2 id="1-start-with-the-exam-objectives"&gt;&lt;a href="#1-start-with-the-exam-objectives" class="header-anchor"&gt;&lt;/a&gt;1. Start with the exam objectives&#10;&lt;/h2&gt;&lt;p&gt;The official certification page is the source of truth for the current exam format, languages, eligibility, and links to the standard and renewal exam guides. Exam objectives and services change, so I would not treat any static checklist as exhaustive.&lt;/p&gt;&#10;&lt;p&gt;The main areas I would organize my study around are:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Designing data processing architectures, including batch and streaming.&lt;/li&gt;&#10;&lt;li&gt;Ingesting, storing, and processing data with appropriate services.&lt;/li&gt;&#10;&lt;li&gt;Maintaining reliable pipelines and orchestrating dependent workloads.&lt;/li&gt;&#10;&lt;li&gt;Security, governance, data quality, and access management.&lt;/li&gt;&#10;&lt;li&gt;Monitoring, troubleshooting, performance, and cost optimization.&lt;/li&gt;&#10;&lt;li&gt;Understanding where machine learning fits into data engineering workflows.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;As a study exercise, take any architecture question and identify its workload volume, latency requirements, consistency expectations, operational constraints, security requirements, and cost priorities &lt;strong&gt;before&lt;/strong&gt; picking a product.&lt;/p&gt;&#10;&lt;h2 id="2-a-quick-service-selection-map"&gt;&lt;a href="#2-a-quick-service-selection-map" class="header-anchor"&gt;&lt;/a&gt;2. A quick service-selection map&#10;&lt;/h2&gt;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Requirement&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Services worth evaluating&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Main question&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Analytical SQL over large datasets&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;BigQuery&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Is the primary workload analytical rather than transactional?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Large-scale key-based, low-latency access&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Bigtable&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Are access patterns known and row-key design appropriate?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Globally distributed relational transactions&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Spanner&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Are strong consistency and horizontal scale requirements sufficient to justify the complexity and cost?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Managed relational database&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cloud SQL&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Does a conventional relational engine satisfy the workload?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Object storage and data lake inputs&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cloud Storage&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Which storage class, lifecycle, and file format fit the access pattern?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Event ingestion and fan-out&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Pub/Sub&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;How will consumers handle duplicates, ordering, and backpressure?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Managed batch or streaming transformations&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Dataflow / Apache Beam&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Are event-time processing, windows, and autoscaling relevant?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Managed Spark or Hadoop workloads&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Dataproc&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Is there existing Spark/Hadoop code or ecosystem compatibility to preserve?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Scheduled, dependent workflows&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Cloud Composer / Apache Airflow&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Do dependencies, retries, and operational visibility justify orchestration?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Model training and inference&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Vertex AI&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Is the requirement feature preparation, training, evaluation, or prediction serving?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;SQL-based ML on warehouse data&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;BigQuery ML&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Can the model workflow stay close to the data in BigQuery?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Metadata, discovery, governance&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;BigQuery universal catalog&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;How will data assets be discovered, classified, and governed?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;These are starting points, not one-to-one equivalents. For example, Pub/Sub can replace some messaging patterns associated with Kafka or RabbitMQ, but the products do not offer identical execution or operational semantics.&lt;/p&gt;&#10;&lt;h2 id="3-data-storage-and-analytical-systems"&gt;&lt;a href="#3-data-storage-and-analytical-systems" class="header-anchor"&gt;&lt;/a&gt;3. Data storage and analytical systems&#10;&lt;/h2&gt;&lt;h3 id="bigquery-analytical-workloads"&gt;&lt;a href="#bigquery-analytical-workloads" class="header-anchor"&gt;&lt;/a&gt;BigQuery: analytical workloads&#10;&lt;/h3&gt;&lt;p&gt;BigQuery is central to many PDE scenarios. I would spend time on &lt;strong&gt;how data is modeled, loaded, queried, secured, and paid for&lt;/strong&gt;, rather than memorizing individual SQL examples.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Modeling.&lt;/strong&gt; Know the trade-offs between star and snowflake schemas, denormalization, nested and repeated fields, and how those choices influence analytical query patterns. A star schema can simplify common dimensional queries; further normalization can reduce duplication but require additional joins.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Partitioning and clustering.&lt;/strong&gt; Partitioning can reduce scanned data when queries filter on a supported partition key. Clustering can further help organize data for filters and aggregation. Neither feature is a universal optimization: examine real query predicates and data distribution before choosing keys.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Execution and query tuning.&lt;/strong&gt; Study query execution details through the BigQuery query plan and execution graph, along with bytes processed, shuffle behavior, joins, and partition pruning. My original notes referred to generic &lt;code&gt;EXPLAIN&lt;/code&gt; and index-based scans; that is not a useful mental model for BigQuery&amp;rsquo;s execution engine. &lt;a class="link" href="https://cloud.google.com/bigquery/docs/query-plan-explanation" target="_blank" rel="noopener"&#10; &gt;BigQuery query plan documentation&lt;/a&gt; is a better reference.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Derived results.&lt;/strong&gt; Views, materialized views, and scheduled queries solve different needs. Materialized views can accelerate supported repeated aggregations, but their refresh behavior, eligibility, and maintenance costs matter.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Pricing.&lt;/strong&gt; Compare on-demand query pricing with capacity-based compute pricing using BigQuery editions, reservations, and slots. The older “flat-rate versus on-demand” distinction in my initial notes is outdated. Storage, streaming, and other charges also need consideration. &lt;a class="link" href="https://cloud.google.com/bigquery/pricing" target="_blank" rel="noopener"&#10; &gt;Current pricing&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Data ingestion.&lt;/strong&gt; Know batch loads from Cloud Storage, managed transfers through BigQuery Data Transfer Service, and ingestion using the Storage Write API. Think about latency, at-least-once versus exactly-once semantics where applicable, cost, and operational complexity.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Security.&lt;/strong&gt; Understand project, dataset, table, row, and column access patterns; authorized views; policy tags; data masking; IAM; and customer-managed encryption keys (CMEK). A view that exposes selected fields is useful, but simply creating a view is not the same as enforcing all row- and column-level security requirements.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Other capabilities.&lt;/strong&gt; Understand BigQuery ML for SQL-centric model workflows and BigQuery Omni for certain cross-cloud analytics patterns. They are worth knowing conceptually, but always check their current limits and supported regions.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario to reason through:&lt;/strong&gt; A dashboard repeatedly aggregates sales by date and product category. What are the relative benefits of partitioning, clustering, a materialized view, and a capacity reservation? What evidence would you collect before choosing?&lt;/p&gt;&#10;&lt;h3 id="cloud-storage-objects-and-analytical-file-formats"&gt;&lt;a href="#cloud-storage-objects-and-analytical-file-formats" class="header-anchor"&gt;&lt;/a&gt;Cloud Storage: objects and analytical file formats&#10;&lt;/h3&gt;&lt;p&gt;Cloud Storage is often the landing zone for files before a processing or analytics stage.&lt;/p&gt;&#10;&lt;p&gt;Know Standard, Nearline, Coldline, and Archive classes, together with minimum storage durations, retrieval costs, and lifecycle policies. Infrequent access alone does not determine the optimal class; retrieval patterns and retention matter.&lt;/p&gt;&#10;&lt;p&gt;Review object versioning, retention controls, lifecycle management, signed URLs, IAM, Storage Transfer Service, and the &lt;code&gt;gcloud storage&lt;/code&gt; command family. Older materials may still use &lt;code&gt;gsutil&lt;/code&gt;; recognize it, but learn the current tooling as well.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;File formats matter:&lt;/strong&gt; CSV and JSON are common interchange formats; Avro is row-oriented and supports schemas; Parquet and ORC are columnar formats that can be efficient for analytical scanning. My original notes incorrectly grouped Avro with columnar formats. Compression and schema evolution support vary by format and integration.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; You ingest large daily event files and query only a few columns. Would you transform the files into Parquet before analysis, and how would that interact with your downstream BigQuery loading strategy?&lt;/p&gt;&#10;&lt;h3 id="cloud-sql-managed-relational-engines"&gt;&lt;a href="#cloud-sql-managed-relational-engines" class="header-anchor"&gt;&lt;/a&gt;Cloud SQL: managed relational engines&#10;&lt;/h3&gt;&lt;p&gt;Review MySQL, PostgreSQL, and SQL Server deployment options; backups; point-in-time recovery; high availability; read replicas; connection management; IAM-related access; networking; and the Cloud SQL Auth Proxy.&lt;/p&gt;&#10;&lt;p&gt;High availability and read replication serve different purposes. A read replica is not automatically equivalent to a high-availability failover configuration. For a latency-sensitive application, connection pooling and database limits may be as important as instance size.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; A transactional application needs automatic failover and recovery from accidental deletion. Explain why failover, backups, and point-in-time recovery address different failure modes.&lt;/p&gt;&#10;&lt;h3 id="spanner-relational-consistency-at-scale"&gt;&lt;a href="#spanner-relational-consistency-at-scale" class="header-anchor"&gt;&lt;/a&gt;Spanner: relational consistency at scale&#10;&lt;/h3&gt;&lt;p&gt;Study primary-key design, interleaving or table locality concepts, secondary indexes, transactions, replication, and the cost/performance implications of data distribution. Monotonically increasing keys may create hotspots, depending on the schema and workload.&lt;/p&gt;&#10;&lt;p&gt;Do not assume Spanner is preferable whenever a relational database must scale. Its value depends on geographic distribution, consistency requirements, write patterns, and the organization&amp;rsquo;s tolerance for operational and financial complexity.&lt;/p&gt;&#10;&lt;h3 id="bigtable-low-latency-wide-column-access"&gt;&lt;a href="#bigtable-low-latency-wide-column-access" class="header-anchor"&gt;&lt;/a&gt;Bigtable: low-latency, wide-column access&#10;&lt;/h3&gt;&lt;p&gt;Know row-key design, access patterns, hotspots, column families, replication, and why a key-value or wide-column workload differs from an analytical SQL warehouse.&lt;/p&gt;&#10;&lt;p&gt;A useful comparison: &lt;strong&gt;Bigtable&lt;/strong&gt; can fit high-throughput key-oriented lookups; &lt;strong&gt;BigQuery&lt;/strong&gt; can fit large analytical scans; &lt;strong&gt;Spanner&lt;/strong&gt; can fit relational transactions at distributed scale. They solve overlapping but materially different problems.&lt;/p&gt;&#10;&lt;h3 id="firestore-in-datastore-mode"&gt;&lt;a href="#firestore-in-datastore-mode" class="header-anchor"&gt;&lt;/a&gt;Firestore in Datastore mode&#10;&lt;/h3&gt;&lt;p&gt;Review entity kinds, keys, indexes, transactions, consistency behavior, namespaces, and query limitations. The original notes described Datastore as uniformly “eventually consistent,” which is too broad for an exam reference. Look at the guarantees of the specific database mode and operation in the &lt;a class="link" href="https://cloud.google.com/firestore/docs" target="_blank" rel="noopener"&#10; &gt;current Firestore documentation&lt;/a&gt;.&lt;/p&gt;&#10;&lt;h2 id="4-data-processing-batch-streaming-and-orchestration"&gt;&lt;a href="#4-data-processing-batch-streaming-and-orchestration" class="header-anchor"&gt;&lt;/a&gt;4. Data processing: batch, streaming, and orchestration&#10;&lt;/h2&gt;&lt;h3 id="dataflow-and-apache-beam"&gt;&lt;a href="#dataflow-and-apache-beam" class="header-anchor"&gt;&lt;/a&gt;Dataflow and Apache Beam&#10;&lt;/h3&gt;&lt;p&gt;Dataflow runs Apache Beam pipelines in a managed environment. Understand how &lt;code&gt;PCollection&lt;/code&gt;, &lt;code&gt;PTransform&lt;/code&gt;, and pipeline definitions relate to the processing graph.&lt;/p&gt;&#10;&lt;p&gt;The more challenging concepts usually involve &lt;strong&gt;event time, processing time, watermarks, windows, triggers, late data, and state&lt;/strong&gt;.&lt;/p&gt;&#10;&lt;table&gt;&#10;&#9;&lt;thead&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Concept&lt;/th&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;th&gt;Questions to ask&lt;/th&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/thead&gt;&#10;&#9;&lt;tbody&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Fixed window&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Is grouping by nonoverlapping intervals acceptable?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Sliding window&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Does a rolling view justify overlapping computations?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Session window&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Is activity separated by gaps, for example user inactivity?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Watermark&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;How does the system estimate event-time completeness?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Trigger&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;When should partial or final results be emitted?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Allowed lateness&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;What happens when data arrives after the watermark?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&#9;&#9;&lt;tr&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Deduplication&lt;/td&gt;&#10;&#9;&#9;&#9;&#9;&#9;&lt;td&gt;Can sources and retries produce duplicate events?&lt;/td&gt;&#10;&#9;&#9;&#9;&lt;/tr&gt;&#10;&#9;&lt;/tbody&gt;&#10;&lt;/table&gt;&#10;&lt;p&gt;Batch and streaming pipelines have different resource, latency, and operational characteristics. Review autoscaling, worker failures, shuffle, stateful processing, and templates. Automatic worker recovery does not mean application code is free from side effects or duplicate-processing risk.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; You must detect an abandoned shopping cart after a user becomes inactive. A session window may help represent inactivity, but you also need to define which events indicate checkout and how late events affect the decision.&lt;/p&gt;&#10;&lt;h3 id="dataproc-spark-and-hadoop"&gt;&lt;a href="#dataproc-spark-and-hadoop" class="header-anchor"&gt;&lt;/a&gt;Dataproc, Spark, and Hadoop&#10;&lt;/h3&gt;&lt;p&gt;Dataproc is especially relevant when an organization already uses Spark, Hadoop, Hive, or related ecosystem components.&lt;/p&gt;&#10;&lt;p&gt;Know Spark DataFrames, Spark SQL, RDDs, and the basics of execution graphs and partitioning. For Hadoop, understand HDFS and MapReduce at a conceptual level. Spark&amp;rsquo;s model can be advantageous for iterative computation, but performance depends on workload and configuration.&lt;/p&gt;&#10;&lt;p&gt;Operational topics from my notes worth reviewing include cluster sizing, autoscaling policies, worker decommissioning, job submission (&lt;code&gt;gcloud dataproc jobs submit spark&lt;/code&gt;), initialization actions, image versions, the component gateway, and integrations with Cloud Storage, BigQuery, and Cloud Logging.&lt;/p&gt;&#10;&lt;p&gt;Be careful with simplistic autoscaling statements: Dataproc autoscaling behavior depends on configuration, supported components, and relevant YARN metrics; a generic CPU-threshold example is not a reliable description of its autoscaling policy mechanism.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; Your company has existing Spark jobs and libraries. What would be gained or lost by moving them to Dataflow rather than running them on Dataproc?&lt;/p&gt;&#10;&lt;h3 id="pubsub-messaging-and-asynchronous-processing"&gt;&lt;a href="#pubsub-messaging-and-asynchronous-processing" class="header-anchor"&gt;&lt;/a&gt;Pub/Sub: messaging and asynchronous processing&#10;&lt;/h3&gt;&lt;p&gt;Review topics, subscriptions, publishers, pull versus push, filters, retention, dead-letter topics, retry policies, acknowledgment deadlines, and subscriber observability.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Default delivery is at least once, with no general ordering guarantee.&lt;/strong&gt; Ordering keys can provide ordering for messages with the same key under documented constraints. Exactly-once delivery is a configurable feature of pull subscriptions, scoped to a region and subject to its documented semantics. It does not automatically guarantee that arbitrary downstream business operations happen exactly once. &lt;a class="link" href="https://cloud.google.com/pubsub/docs/subscription-overview" target="_blank" rel="noopener"&#10; &gt;Subscription behavior&lt;/a&gt; and &lt;a class="link" href="https://cloud.google.com/pubsub/docs/exactly-once-delivery" target="_blank" rel="noopener"&#10; &gt;exactly-once delivery&lt;/a&gt;.&lt;/p&gt;&#10;&lt;p&gt;Build consumers to handle duplicate delivery, delayed events, backpressure, and dead-letter processing. Know when idempotency should be enforced in application logic.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; A payment event triggers fulfillment. How can the consumer avoid duplicate fulfillment when messages are redelivered? How would you monitor unacknowledged message backlog and oldest message age?&lt;/p&gt;&#10;&lt;h3 id="cloud-composer-and-apache-airflow"&gt;&lt;a href="#cloud-composer-and-apache-airflow" class="header-anchor"&gt;&lt;/a&gt;Cloud Composer and Apache Airflow&#10;&lt;/h3&gt;&lt;p&gt;Composer provides a managed Airflow environment for orchestrating dependent workloads. Learn DAGs, tasks, operators, scheduling, retries, connections, variables, XCom, idempotency, backfills, catchup, and operational logs.&lt;/p&gt;&#10;&lt;p&gt;I would focus on the distinction between an orchestrator and a processing engine. Airflow can coordinate BigQuery and Dataflow jobs without executing all their computation inside Airflow workers.&lt;/p&gt;&#10;&lt;p&gt;Operator names and providers evolve, so avoid memorizing older classes such as &lt;code&gt;BigQueryOperator&lt;/code&gt; without checking the current provider package. Prefer understanding what the operator does and how authentication, retries, and data dependencies are modeled.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; A daily pipeline loads Cloud Storage files, transforms data in BigQuery, validates results, and publishes a report. What happens when the transformation fails after the load has already succeeded? How will reruns avoid duplicate output?&lt;/p&gt;&#10;&lt;h3 id="event-driven-compute"&gt;&lt;a href="#event-driven-compute" class="header-anchor"&gt;&lt;/a&gt;Event-driven compute&#10;&lt;/h3&gt;&lt;p&gt;Cloud Functions, now commonly discussed within the Cloud Run functions product family, can run event-triggered and HTTP-triggered code without managing servers directly. Study event sources, service accounts, memory and timeout configuration, concurrency where supported, cold starts, retries, and observability.&lt;/p&gt;&#10;&lt;p&gt;Understand the distinction between using a function to process a lightweight event and using Dataflow or Dataproc for sustained large-scale processing. Service names, generation-specific capabilities, and limits need checking against the current documentation.&lt;/p&gt;&#10;&lt;h2 id="5-machine-learning-from-the-data-engineers-perspective"&gt;&lt;a href="#5-machine-learning-from-the-data-engineers-perspective" class="header-anchor"&gt;&lt;/a&gt;5. Machine learning from the data engineer&amp;rsquo;s perspective&#10;&lt;/h2&gt;&lt;p&gt;The PDE exam is not primarily an ML research exam. The practical concern is often whether data is reliable, properly transformed, governed, and ready for training or inference.&lt;/p&gt;&#10;&lt;h3 id="vertex-ai"&gt;&lt;a href="#vertex-ai" class="header-anchor"&gt;&lt;/a&gt;Vertex AI&#10;&lt;/h3&gt;&lt;p&gt;Know the roles of managed datasets, custom training, AutoML approaches, pipelines, model registry, endpoints, batch prediction, online prediction, monitoring, and feature management.&lt;/p&gt;&#10;&lt;p&gt;Concepts worth understanding include training versus serving, evaluation metrics (precision, recall, AUC where appropriate), hyperparameter tuning, distributed training, and data drift. Model and feature capabilities evolve frequently, so validate specific current product names before relying on old diagrams.&lt;/p&gt;&#10;&lt;h3 id="bigquery-ml"&gt;&lt;a href="#bigquery-ml" class="header-anchor"&gt;&lt;/a&gt;BigQuery ML&#10;&lt;/h3&gt;&lt;p&gt;Understand when SQL-based model building close to warehouse data makes sense. Practice recognizing supported workflows and the implications for data movement, model evaluation, and operationalization.&lt;/p&gt;&#10;&lt;h3 id="tensorflow-pytorch-gpus-and-tpus"&gt;&lt;a href="#tensorflow-pytorch-gpus-and-tpus" class="header-anchor"&gt;&lt;/a&gt;TensorFlow, PyTorch, GPUs, and TPUs&#10;&lt;/h3&gt;&lt;p&gt;Know these as ecosystem and compute choices rather than attempting to memorize every training API. Be able to explain why the size of a model or dataset, framework requirements, accelerators, and distributed training strategy might influence a Vertex AI workload design.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; Most features already live in BigQuery, and the goal is a straightforward regression model. Would BigQuery ML simplify the workflow? What would push you toward custom training on Vertex AI instead?&lt;/p&gt;&#10;&lt;h2 id="6-security-privacy-and-data-governance"&gt;&lt;a href="#6-security-privacy-and-data-governance" class="header-anchor"&gt;&lt;/a&gt;6. Security, privacy, and data governance&#10;&lt;/h2&gt;&lt;h3 id="iam-and-resource-hierarchy"&gt;&lt;a href="#iam-and-resource-hierarchy" class="header-anchor"&gt;&lt;/a&gt;IAM and resource hierarchy&#10;&lt;/h3&gt;&lt;p&gt;Review organizations, folders, projects, policies, inheritance, principal types, groups, service accounts, predefined roles, custom roles, and least privilege.&lt;/p&gt;&#10;&lt;p&gt;Remember that broad grants at higher levels in the resource hierarchy can affect downstream resources. IAM policies generally combine applicable permissions; a lower-level grant does not simply override a higher-level grant. Deny policies and other controls introduce additional rules.&lt;/p&gt;&#10;&lt;p&gt;Service account keys create long-lived credential risks. Prefer appropriately scoped workload identities and keyless authentication where suitable rather than treating periodic key rotation as the complete answer.&lt;/p&gt;&#10;&lt;h3 id="data-classification-discovery-and-governance"&gt;&lt;a href="#data-classification-discovery-and-governance" class="header-anchor"&gt;&lt;/a&gt;Data classification, discovery, and governance&#10;&lt;/h3&gt;&lt;p&gt;My original notes included &lt;strong&gt;Data Catalog&lt;/strong&gt;, which was deprecated in 2025 with a discontinuation date of January 30, 2026. Study the &lt;a class="link" href="https://cloud.google.com/dataplex/docs/catalog-overview" target="_blank" rel="noopener"&#10; &gt;BigQuery universal catalog&lt;/a&gt; direction instead, while recognizing older names when reading historical practice material.&lt;/p&gt;&#10;&lt;p&gt;Understand discovery, metadata, lineage, classification, policy tags, data quality, and auditability at the architectural level.&lt;/p&gt;&#10;&lt;h3 id="sensitive-data-and-cloud-kms"&gt;&lt;a href="#sensitive-data-and-cloud-kms" class="header-anchor"&gt;&lt;/a&gt;Sensitive data and Cloud KMS&#10;&lt;/h3&gt;&lt;p&gt;Review Sensitive Data Protection (formerly Cloud DLP) concepts such as inspection, infoTypes, masking, redaction, de-identification, and tokenization. Understand when to use templates and how sensitive data may appear in Cloud Storage, BigQuery, and streams.&lt;/p&gt;&#10;&lt;p&gt;For Cloud KMS, learn key rings, key versions, rotation, IAM roles, CMEK, envelope encryption, and operational implications of key unavailability. Rotation policies depend on compliance requirements and how keys are consumed; they are not a substitute for authorization controls.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Scenario:&lt;/strong&gt; Multiple customers share a data platform but should not see each other&amp;rsquo;s data. Evaluate dataset separation, row-level policies, service identities, and audit logging. Which controls enforce isolation and which only help detect violations?&lt;/p&gt;&#10;&lt;h2 id="7-operations-reliability-and-cost"&gt;&lt;a href="#7-operations-reliability-and-cost" class="header-anchor"&gt;&lt;/a&gt;7. Operations, reliability, and cost&#10;&lt;/h2&gt;&lt;h3 id="monitoring-and-troubleshooting"&gt;&lt;a href="#monitoring-and-troubleshooting" class="header-anchor"&gt;&lt;/a&gt;Monitoring and troubleshooting&#10;&lt;/h3&gt;&lt;p&gt;Distinguish metrics, logs, traces, and audit logs. Review Cloud Monitoring dashboards and alerting, Cloud Logging queries and structured entries, log-based metrics, and log sinks to Cloud Storage, BigQuery, or Pub/Sub.&lt;/p&gt;&#10;&lt;p&gt;For data pipelines, useful signals might include throughput, error rates, processing latency, consumer backlog, failed jobs, duplicate events, late data, and end-to-end freshness. A pipeline can be technically running while the delivered data is stale or incorrect.&lt;/p&gt;&#10;&lt;h3 id="reliability-patterns-worth-recognizing"&gt;&lt;a href="#reliability-patterns-worth-recognizing" class="header-anchor"&gt;&lt;/a&gt;Reliability patterns worth recognizing&#10;&lt;/h3&gt;&lt;ul&gt;&#10;&lt;li&gt;Idempotent processing and repeatable backfills.&lt;/li&gt;&#10;&lt;li&gt;Controlled retries with backoff and clear terminal-failure behavior.&lt;/li&gt;&#10;&lt;li&gt;Dead-letter handling and reprocessing workflows.&lt;/li&gt;&#10;&lt;li&gt;Checkpoints, recovery, and replay behavior.&lt;/li&gt;&#10;&lt;li&gt;Data validation at ingestion and transformation stages.&lt;/li&gt;&#10;&lt;li&gt;Service quotas, capacity constraints, and autoscaling limits.&lt;/li&gt;&#10;&lt;li&gt;Separation of source-system failures from downstream consumer failures.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h3 id="cost-and-performance"&gt;&lt;a href="#cost-and-performance" class="header-anchor"&gt;&lt;/a&gt;Cost and performance&#10;&lt;/h3&gt;&lt;p&gt;Understand how query scans, slots, storage classes, data transfer, streaming ingestion, worker capacity, and regional placement can affect costs. For each optimization, look at the workload before assuming the change is beneficial.&lt;/p&gt;&#10;&lt;h2 id="8-exam-style-reasoning-exercises"&gt;&lt;a href="#8-exam-style-reasoning-exercises" class="header-anchor"&gt;&lt;/a&gt;8. Exam-style reasoning exercises&#10;&lt;/h2&gt;&lt;p&gt;These are &lt;strong&gt;original study prompts&lt;/strong&gt;, not official or recalled exam questions. Instead of treating answers as fixed product-name mappings, write down assumptions and explain your choice.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;1. Analytical queries are getting expensive.&lt;/strong&gt; A large date-partitioned BigQuery table is scanned repeatedly for weekly reporting. Which evidence would you inspect before adjusting the partition key, clustering fields, query design, or reservation configuration?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;2. Events arrive out of order.&lt;/strong&gt; A Dataflow pipeline aggregates user activity. How do event time, watermarks, windows, and triggers interact? What happens to late events?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;3. Duplicate messages cause side effects.&lt;/strong&gt; A Pub/Sub subscriber sends notifications and may receive messages more than once. Where should idempotency be enforced, and what does enabling exactly-once delivery change?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;4. A database needs business continuity.&lt;/strong&gt; A Cloud SQL application requires automatic failover and the ability to recover from an accidental destructive update. Why are high availability and point-in-time recovery both relevant?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;5. A Spark migration has operational constraints.&lt;/strong&gt; A team needs to move existing Hadoop/Spark jobs to Google Cloud. What determines whether Dataproc or a redesigned Beam/Dataflow pipeline is a more reasonable first approach?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;6. Customer data requires isolation.&lt;/strong&gt; A shared analytical platform serves multiple tenants. How do datasets, IAM, row-level security, and audit logging address different aspects of the problem?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;7. A pipeline claims success but data is stale.&lt;/strong&gt; What metrics and validation checks would you add beyond worker health and job completion status?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;8. A model needs daily training.&lt;/strong&gt; What roles could Cloud Composer, BigQuery, Vertex AI, and Cloud Storage play? Which component owns orchestration, preparation, training, and artifacts?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;9. Global relational consistency is required.&lt;/strong&gt; Which workload characteristics justify evaluating Spanner rather than Cloud SQL or Bigtable?&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;10. An archive is rarely accessed.&lt;/strong&gt; How would you choose a Cloud Storage class after accounting for retrieval charges, minimum durations, and operational recovery requirements?&lt;/p&gt;&#10;&lt;h2 id="9-practice-questions-labs-and-references"&gt;&lt;a href="#9-practice-questions-labs-and-references" class="header-anchor"&gt;&lt;/a&gt;9. Practice questions, labs, and references&#10;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Start with the official material:&lt;/strong&gt;&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/learn/certification/data-engineer" target="_blank" rel="noopener"&#10; &gt;Google Cloud Professional Data Engineer certification and exam guide&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://www.cloudskillsboost.google/paths/16/" target="_blank" rel="noopener"&#10; &gt;Google Cloud Data Engineer learning path&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/learn/certification/data-engineer" target="_blank" rel="noopener"&#10; &gt;Google Cloud sample questions&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/bigquery/docs" target="_blank" rel="noopener"&#10; &gt;BigQuery documentation&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/dataflow/docs" target="_blank" rel="noopener"&#10; &gt;Dataflow documentation&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/dataproc/docs" target="_blank" rel="noopener"&#10; &gt;Dataproc documentation&lt;/a&gt;&lt;/li&gt;&#10;&lt;li&gt;&lt;a class="link" href="https://cloud.google.com/pubsub/docs" target="_blank" rel="noopener"&#10; &gt;Pub/Sub documentation&lt;/a&gt;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;&lt;strong&gt;Supplementary practice resources:&lt;/strong&gt;&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&#10;&lt;p&gt;&lt;a class="link" href="https://dailydebian.com/blogs/practice-questions/google-professional-data-engineer-practice-questions" target="_blank" rel="noopener"&#10; &gt;DailyDebian free PDE practice questions&lt;/a&gt;. Added during the October 2026 revision. The site advertises 87 free questions with explanations, alongside optional larger collections.&lt;/p&gt;&#10;&lt;/li&gt;&#10;&lt;li&gt;&#10;&lt;p&gt;&lt;a class="link" href="https://www.examprepper.co/exam/10/1" target="_blank" rel="noopener"&#10; &gt;ExamPrepper PDE questions&lt;/a&gt;. This was among the community resources in my original notes; verify question freshness and explanations.&lt;/p&gt;&#10;&lt;/li&gt;&#10;&lt;li&gt;&#10;&lt;p&gt;&lt;a class="link" href="https://www.udemy.com/" target="_blank" rel="noopener"&#10; &gt;Udemy PDE practice tests&lt;/a&gt;. Course offerings, names, and quality change, so prefer a current listing over an expired coupon or a dated exam-version claim.&lt;/p&gt;&#10;&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;Third-party question banks can help reveal knowledge gaps, but they can also contain outdated or oversimplified answers. I would use them to practice reasoning, then verify disputed details in the official documentation.&lt;/p&gt;&#10;&lt;h2 id="10-a-final-study-checklist"&gt;&lt;a href="#10-a-final-study-checklist" class="header-anchor"&gt;&lt;/a&gt;10. A final study checklist&#10;&lt;/h2&gt;&lt;p&gt;Before considering myself ready, I would want to be comfortable explaining, without looking at notes:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;When to choose BigQuery, Cloud SQL, Spanner, Bigtable, and Cloud Storage.&lt;/li&gt;&#10;&lt;li&gt;Why partitioning, clustering, and query design influence BigQuery cost and performance.&lt;/li&gt;&#10;&lt;li&gt;How Dataflow event-time processing handles windows, triggers, watermarks, and late data.&lt;/li&gt;&#10;&lt;li&gt;How Pub/Sub delivery, ordering, retries, and idempotency fit together.&lt;/li&gt;&#10;&lt;li&gt;When Dataproc is a good fit for existing Spark or Hadoop workloads.&lt;/li&gt;&#10;&lt;li&gt;How to orchestrate dependable pipelines using Cloud Composer.&lt;/li&gt;&#10;&lt;li&gt;How data engineers use BigQuery ML and Vertex AI without confusing training with serving.&lt;/li&gt;&#10;&lt;li&gt;How IAM, row/column controls, metadata, classification, and key management protect data.&lt;/li&gt;&#10;&lt;li&gt;How to investigate a pipeline that is available but producing incorrect or stale results.&lt;/li&gt;&#10;&lt;li&gt;How service selection changes when latency, scale, cost, governance, and operational maturity differ.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;This is the reference I wanted when assembling my initial study notes: broad enough to orient a full preparation plan, but organized so individual topics are easy to revisit. It is not a replacement for hands-on labs, official objectives, or experience with production data systems. The most useful preparation, in my view, is being able to explain a decision and its limitations rather than recalling a product name in isolation.&lt;/p&gt;&#10;</description></item><item><title>Preparing for the Google Cloud Professional Data Engineer Exam: My Study Approach and Resources</title><link>https://fabiolopes.page/p/preparing-for-the-google-cloud-professional-data-engineer-exam-my-study-approach-and-resources/</link><pubDate>Tue, 04 Feb 2025 00:00:00 +0000</pubDate><guid>https://fabiolopes.page/p/preparing-for-the-google-cloud-professional-data-engineer-exam-my-study-approach-and-resources/</guid><description>&lt;img src="https://fabiolopes.page/p/preparing-for-the-google-cloud-professional-data-engineer-exam-my-study-approach-and-resources/dall-e_hu_19d84e9ace505479.webp" alt="Featured image of post Preparing for the Google Cloud Professional Data Engineer Exam: My Study Approach and Resources" /&gt;&lt;p&gt;If you want to follow the full study guide, see my &lt;a class="link" href="https://fabiolopes.page/p/preparation-for-google-cloud-professional-data-engineer-pde-exam/" &gt;Google Cloud Professional Data Engineer: A Practical Study Guide and Service Reference&lt;/a&gt;. It covers service selection, core concepts, and practice scenarios in more detail. Here, I share my personal preparation approach and the resources I used.&lt;/p&gt;&#10;&lt;p&gt;After passing the Google Cloud Professional Cloud Architect (PCA) exam, I wanted to spend more time learning about the systems that run &lt;em&gt;on&lt;/em&gt; cloud infrastructure, especially data pipelines, analytics, and machine learning workloads.&lt;/p&gt;&#10;&lt;p&gt;Most of my professional experience had been on the infrastructure and architecture side: designing, provisioning, and operating environments used by other teams. Preparing for the &lt;strong&gt;Google Cloud Professional Data Engineer (PDE)&lt;/strong&gt; exam was an opportunity to look more closely at those teams&amp;rsquo; technical decisions and the services behind them.&lt;/p&gt;&#10;&lt;p&gt;I originally wrote this article in February 2025, while preparing for the PDE certification. I&amp;rsquo;m revisiting it in October 2026 to make the material easier to read and to update some of the resources and terminology. The preparation described below reflects what I did at that time; the additions are not meant to suggest I used newer resources back then.&lt;/p&gt;&#10;&#10; &lt;blockquote&gt;&#10; &lt;p&gt;&lt;strong&gt;In brief:&lt;/strong&gt; My preparation centered on the Google Cloud Data Engineer learning path, hands-on labs, learning when to use services such as BigQuery, Dataflow, Dataproc, and Pub/Sub, and working through practice questions. For me, understanding the trade-offs between services mattered more than memorizing their names.&lt;/p&gt;&#10;&#10; &lt;/blockquote&gt;&#10;&lt;h2 id="how-i-approached-the-preparation"&gt;&lt;a href="#how-i-approached-the-preparation" class="header-anchor"&gt;&lt;/a&gt;How I approached the preparation&#10;&lt;/h2&gt;&lt;p&gt;I began with the &lt;a class="link" href="https://www.cloudskillsboost.google/paths/16/" target="_blank" rel="noopener"&#10; &gt;Google Cloud Data Engineer learning path&lt;/a&gt; on Skills Boost (now part of Google Skills). Its labs gave me an environment where I could experiment with Google Cloud services instead of only watching demonstrations.&lt;/p&gt;&#10;&lt;p&gt;I found the hands-on exercises more valuable than the short video segments. Some explanations felt too brief for topics I wanted to understand in depth, so I spent additional time exploring the services beyond the instructions provided in the labs.&lt;/p&gt;&#10;&lt;p&gt;The end-of-module quizzes were also useful, although they weren&amp;rsquo;t a substitute for scenario-based practice exams.&lt;/p&gt;&#10;&lt;p&gt;My general approach was to study a service, try the relevant lab, and then ask myself a more practical question: &lt;em&gt;In what situation would I choose this service over another option?&lt;/em&gt;&lt;/p&gt;&#10;&lt;p&gt;That question helped connect my infrastructure background with data engineering decisions involving operational overhead, cost, scaling, and failure handling.&lt;/p&gt;&#10;&lt;h2 id="the-services-i-focused-on"&gt;&lt;a href="#the-services-i-focused-on" class="header-anchor"&gt;&lt;/a&gt;The services I focused on&#10;&lt;/h2&gt;&lt;p&gt;Rather than trying to memorize every Google Cloud product, I concentrated on a few areas that appeared repeatedly in the learning material.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Analytics and storage.&lt;/strong&gt; I spent time with &lt;a class="link" href="https://cloud.google.com/bigquery" target="_blank" rel="noopener"&#10; &gt;BigQuery&lt;/a&gt;, particularly SQL querying, partitioning, clustering, cost considerations, and the relationship between data layout and access patterns. I also reviewed &lt;a class="link" href="https://cloud.google.com/storage" target="_blank" rel="noopener"&#10; &gt;Cloud Storage&lt;/a&gt;, &lt;a class="link" href="https://cloud.google.com/bigtable" target="_blank" rel="noopener"&#10; &gt;Bigtable&lt;/a&gt;, &lt;a class="link" href="https://cloud.google.com/spanner" target="_blank" rel="noopener"&#10; &gt;Spanner&lt;/a&gt;, and &lt;a class="link" href="https://cloud.google.com/sql" target="_blank" rel="noopener"&#10; &gt;Cloud SQL&lt;/a&gt;. The important distinction was understanding their different data models and workload expectations.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Batch and streaming data processing.&lt;/strong&gt; &lt;a class="link" href="https://cloud.google.com/products/dataflow" target="_blank" rel="noopener"&#10; &gt;Dataflow&lt;/a&gt; and Apache Beam introduced concepts such as event time, windowing, triggers, and late-arriving data. &lt;a class="link" href="https://cloud.google.com/dataproc" target="_blank" rel="noopener"&#10; &gt;Dataproc&lt;/a&gt; was useful for understanding the managed Spark and Hadoop ecosystem, including when existing processing frameworks might influence a cloud architecture.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Messaging and orchestration.&lt;/strong&gt; With &lt;a class="link" href="https://cloud.google.com/pubsub" target="_blank" rel="noopener"&#10; &gt;Pub/Sub&lt;/a&gt;, I focused on publishers, subscriptions, acknowledgments, delivery behavior, and the design of consumers that can tolerate repeated messages. I also reviewed &lt;a class="link" href="https://cloud.google.com/composer" target="_blank" rel="noopener"&#10; &gt;Cloud Composer&lt;/a&gt; and Apache Airflow for coordinating tasks and managing dependencies across pipelines.&lt;/p&gt;&#10;&lt;p&gt;&lt;strong&gt;Machine learning and governance.&lt;/strong&gt; I explored &lt;a class="link" href="https://cloud.google.com/vertex-ai" target="_blank" rel="noopener"&#10; &gt;Vertex AI&lt;/a&gt; as part of the broader ML lifecycle, alongside IAM, encryption, monitoring, and data governance. These areas matter because data engineering includes operating and protecting pipelines, not just moving or transforming data.&lt;/p&gt;&#10;&lt;p&gt;One thing that helped me was comparing Google-managed offerings with technologies I already knew. For example, understanding the relationships between Apache Beam and Dataflow, Apache Airflow and Cloud Composer, or Apache Spark and Dataproc made the ecosystem easier to navigate. Those services aren&amp;rsquo;t interchangeable in every detail, but the comparisons provided useful starting points.&lt;/p&gt;&#10;&lt;h2 id="practice-tests-and-study-resources"&gt;&lt;a href="#practice-tests-and-study-resources" class="header-anchor"&gt;&lt;/a&gt;Practice tests and study resources&#10;&lt;/h2&gt;&lt;p&gt;Practice questions helped me identify topics I didn&amp;rsquo;t understand well enough, particularly when the options looked similar but carried different architectural implications.&lt;/p&gt;&#10;&lt;p&gt;Here are the resources I used originally, followed by one additional resource included in this 2026 update.&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a class="link" href="https://cloud.google.com/learn/certification/data-engineer" target="_blank" rel="noopener"&#10; &gt;Google Cloud official exam page and exam guide&lt;/a&gt;:&lt;/strong&gt; The starting point for understanding the current exam scope and finding Google&amp;rsquo;s official sample questions. I would consult this first, since the certification blueprint can change.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a class="link" href="https://www.cloudskillsboost.google/paths/16/" target="_blank" rel="noopener"&#10; &gt;Google Cloud Data Engineer learning path&lt;/a&gt;:&lt;/strong&gt; My main structured learning resource, especially for hands-on labs.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a class="link" href="https://dailydebian.com/blogs/practice-questions/google-professional-data-engineer-practice-questions" target="_blank" rel="noopener"&#10; &gt;DailyDebian&amp;rsquo;s free Professional Data Engineer practice questions&lt;/a&gt;:&lt;/strong&gt; &lt;em&gt;Added in October 2026.&lt;/em&gt; The site provides free questions with answer explanations and also offers larger question sets under separate terms, not only for this exam but also for many others.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a class="link" href="https://www.examprepper.co/exam/10/1" target="_blank" rel="noopener"&#10; &gt;ExamPrepper&lt;/a&gt;:&lt;/strong&gt; One of the free question banks I used during my preparation. I found community discussions helpful, while keeping in mind that third-party explanations can be incomplete or outdated.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;&lt;a class="link" href="https://www.udemy.com/" target="_blank" rel="noopener"&#10; &gt;Udemy practice exams&lt;/a&gt;:&lt;/strong&gt; I used additional practice tests found on Udemy. Course offerings and links change, so I would check the current syllabus and reviews rather than rely on a particular dated coupon link.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;For any third-party question bank, I recommend checking explanations against &lt;a class="link" href="https://cloud.google.com/docs" target="_blank" rel="noopener"&#10; &gt;Google Cloud documentation&lt;/a&gt;. A question can be useful even when its answer is debatable, as long as you use it to investigate the underlying service behavior rather than memorize a potentially outdated response.&lt;/p&gt;&#10;&lt;h2 id="what-i-found-most-useful-to-understand"&gt;&lt;a href="#what-i-found-most-useful-to-understand" class="header-anchor"&gt;&lt;/a&gt;What I found most useful to understand&#10;&lt;/h2&gt;&lt;p&gt;The exam preparation made more sense when I focused on decisions instead of product lists.&lt;/p&gt;&#10;&lt;p&gt;For &lt;strong&gt;data storage&lt;/strong&gt;, that meant asking about query patterns, consistency requirements, expected growth, operational overhead, and cost. An analytical workload suited to BigQuery is different from a transactional application that needs a relational database, even if both store structured data.&lt;/p&gt;&#10;&lt;p&gt;For &lt;strong&gt;streaming systems&lt;/strong&gt;, I tried to understand what happens when messages arrive late, are redelivered, or need to be processed in order. These details affect the design of pipelines much more than the ability to connect two services in a diagram.&lt;/p&gt;&#10;&lt;p&gt;For &lt;strong&gt;data processing&lt;/strong&gt;, I compared the trade-offs between managed processing services and frameworks such as Spark. Existing code, team skills, infrastructure responsibility, and workload characteristics can change the decision.&lt;/p&gt;&#10;&lt;p&gt;For &lt;strong&gt;security and operations&lt;/strong&gt;, I reviewed IAM, service accounts, logging, monitoring, and protecting sensitive data. My background in infrastructure was useful here, but data platforms introduce their own operational considerations, including data quality and governance.&lt;/p&gt;&#10;&lt;p&gt;These were study priorities, not a claim that every topic carries the same weight on the current examination. Google&amp;rsquo;s exam guide remains the better reference for coverage.&lt;/p&gt;&#10;&lt;h2 id="what-i-would-do-differently"&gt;&lt;a href="#what-i-would-do-differently" class="header-anchor"&gt;&lt;/a&gt;What I would do differently&#10;&lt;/h2&gt;&lt;p&gt;If I were starting this preparation again, I would spend less time building exhaustive lists of product features and more time constructing small, end-to-end exercises.&lt;/p&gt;&#10;&lt;p&gt;For example, I would consider building a pipeline that receives events through Pub/Sub, processes them with Dataflow, and makes the output available for analysis in BigQuery. I would then test how the design responds to duplicate events, delayed records, schema changes, and cost constraints.&lt;/p&gt;&#10;&lt;p&gt;I would also revisit the current certification blueprint before committing to a study plan. Some product names and recommended services have changed since I first wrote this article. For example, Google deprecated Data Catalog and moved its catalog capabilities toward &lt;strong&gt;BigQuery universal catalog&lt;/strong&gt;. BigQuery pricing discussions have also evolved beyond the older “on-demand versus flat-rate” comparison.&lt;/p&gt;&#10;&lt;p&gt;Finally, I would be more selective with practice questions. A smaller number of well-understood scenarios can teach more than rushing through hundreds of answers, particularly when the questions rely on older service behavior.&lt;/p&gt;&#10;&lt;h2 id="final-thoughts"&gt;&lt;a href="#final-thoughts" class="header-anchor"&gt;&lt;/a&gt;Final thoughts&#10;&lt;/h2&gt;&lt;p&gt;My PDE preparation gave me an opportunity to explore cloud services from a different perspective. Having spent years working on infrastructure, I found it valuable to think about the requirements of the teams that consume that infrastructure: how they ingest data, design pipelines, choose storage systems, and operate workloads over time.&lt;/p&gt;&#10;&lt;p&gt;For my learning style, the combination of labs, documentation, and scenario-based practice worked better than relying on a single course. I wouldn&amp;rsquo;t claim that the same balance is ideal for everyone. Someone already working extensively with BigQuery or Spark may need a different preparation path.&lt;/p&gt;&#10;&lt;p&gt;What I would carry into another certification is the emphasis on understanding &lt;em&gt;why&lt;/em&gt; one architectural choice fits a particular problem, including the trade-offs that come with it.&lt;/p&gt;&#10;&lt;p&gt;&lt;em&gt;This article was originally published in February 2025 and revised in October 2026. Links and service terminology can change; consult the official Google Cloud certification guide for the current exam requirements.&lt;/em&gt;&lt;/p&gt;&#10;</description></item></channel></rss>