Practice with concrete deliverables. Progress at the pace your projects allow.
Suggested study windows, not guarantees of job readiness or seniority. Checklist progress lasts for this page session.
Foundations → platform design
Modern data engineering
Self-paced · suggested 6–12 months
Build a small data product, then make it testable, recoverable and explainable.
0 / 4 milestones completed
Practice & deliverable
Practice joins, window functions, null handling and dimensional models.
Deliver: a model with documented grain, key checks and three analytical queries.
Practice & deliverable
Read, validate and transform data; inspect a distributed execution plan.
Deliver: an ingestion job with fixtures, rejected-record handling and a repeatable run.
Practice & deliverable
Choose one platform first: BigQuery or Azure Synapse. Define access, partitioning and spend controls.
Deliver: a sandbox dataset, an access model and a measured query experiment.
Practice & deliverable
Schedule dependencies, retry safely and rehearse a backfill.
Deliver: a workflow with freshness checks, an incident runbook and a recovery demonstration.
Specialization · data foundations required
AI & vector data infrastructure
Self-paced · suggested 3–6 months
Build an access-aware retrieval pipeline and evaluate it against a small question set.
0 / 4 milestones completed
Practice & deliverable
Track source identity, permissions, content versions and deletion requirements.
Deliver: a versioned document manifest and a replay plan.
Practice & deliverable
Compare chunk boundaries and keep the embedding model version with each experiment.
Deliver: a reproducible indexing job and a short comparison of retrieval results.
Practice & deliverable
Keep metadata filters and access checks close to retrieval. Plan index refresh and deletion.
Deliver: a search endpoint tested with allowed and disallowed documents.
Practice & deliverable
Evaluate retrieved evidence, answer support, latency and cost on a held-out question set.
Deliver: an evaluation report, failure examples and a decision about what to improve next.
Practice explaining the tradeoff.
Illustrative interview scenarios. Open each prompt to compare your reasoning.
A scheduled batch ran twice. What do you inspect?
Start with the output contract and business key. Determine whether append created duplicates, identify the affected batch and compare a corrected result before changing consumers. Explain how you would add a batch identity, idempotent write or controlled merge.
One Spark task dominates the stage. How do you investigate?
Compare partition sizes and task durations, inspect the executed plan and test the hot-key hypothesis. Consider built-in adaptive handling before adding a targeted salting experiment. Validate row multiplicity as well as runtime.
Retrieval returns a restricted document. What must change?
Trace the permission metadata from ingestion to the retrieval filter. Test access boundaries explicitly, refresh or remove stale indexed content and prevent answer generation from receiving unauthorized context.
READ / ADAPT / TEST
Code patterns you can inspect.
Small examples with explicit assumptions. No production-readiness claims.
"""Teaching example. Execute only against a sandbox dataset."""
import argparse
from pyspark.sql import SparkSession, functions as F
def load_batch(spark, source_path, target_table):
source = spark.read.parquet(source_path)
required = {"event_id", "event_type"}
missing = required - set(source.columns)
if missing:
raise ValueError(f"Missing required columns: {sorted(missing)}")
if source.filter(F.col("event_id").isNull()).limit(1).count():
raise ValueError("Null event_id found; batch rejected")
result = source.select("event_id", "event_type").withColumn(
"ingested_at", F.current_timestamp()
)
(result.write.format("bigquery")
.option("writeMethod", "direct")
.mode("append")
.save(target_table))
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("--source", required=True)
parser.add_argument("--target", required=True)
args = parser.parse_args()
session = SparkSession.builder.appName("SandboxBatchLoad").getOrCreate()
try:
load_batch(session, args.source, args.target)
finally:
session.stop()
Before running: PySpark → BigQuery append
Setup. Requires PySpark, a Spark/Scala-compatible BigQuery connector, Application Default Credentials and permissions for the source and target dataset. Supply a Parquet source and a project.dataset.table target.
Scope & limitations. Append is not replay-idempotent: rerunning a batch can duplicate rows. Add a batch ledger or a staging-and-merge design before recurring use. This example rejects missing fields and null IDs; it does not implement schema-drift recovery or a generic ingestion framework.
Teaching templates. Syntax and selected failure paths checked; not exercised against live cloud services.
Technologies and patterns to examine. Each tool brings a different tradeoff.
Distributed compute
› Apache Spark
› PySpark
› Databricks
Storage & lakehouse
› Delta Lake
› Apache Iceberg
› Snowflake
› AWS S3
Streaming & ingestion
› Google Cloud Pub/Sub
› Structured Streaming
› Event Hubs
Orchestration & delivery
› Apache Airflow
› Docker
› Kubernetes
› Terraform
02 / INSPECT THE SYSTEM
Follow the data.
A reference pipeline and code notebook. Examples for discussion, ready to inspect.
event-platform / simulation
01
Pub/SubIngest events
02
SparkTransform & validate
03
LakehouseStore curated data
04
AnalyticsServe the consumer
Simulation running · no external systems connected.
adaptive-query.pyPYTHON
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
# Inspect the plan before tuning the workload.
result = events.groupBy("event_type").count()
result.explain(mode="formatted")
Illustrative code · inspect and adapt to your environment.
REFERENCE ARCHITECTURE / INTERACTIVE
From source data to grounded AI.
A RAG pattern to inspect. Select a stage to explore its responsibilities.
Illustrative design, not a description of a personal or employer project.
01 / INGESTION
Versioned source records
Capture documents and event metadata with durable source identifiers. Make reprocessing a deliberate operation, with an owner and a record of the input version.
Check: source permissions, provenance and replay scope.
This path prepares and retrieves context for RAG. Model fine-tuning is a separate training workflow; it is not implied by creating embeddings or a vector index.