Data Engineer Roadmap

Data skew occurs when one partition has significantly more records than others, causing 99% of tasks to finish in seconds while one straggler task runs for hours or crashes with an OutOfMemory (OOM) error.

Resolution Strategies:
1. Salting: Append a random integer (e.g., 0 to 9) to the skewed join/grouping key to distribute records evenly across worker nodes, perform the computation, and then aggregate back.
2. Broadcast Join: If one table in the join is small (default <10MB, or configurable via spark.sql.autoBroadcastJoinThreshold), broadcast it to all executors to eliminate the shuffle phase completely.
3. Adaptive Query Execution (AQE): Enable spark.sql.adaptive.enabled = true and spark.sql.adaptive.skewJoin.enabled = true in Spark 3+, which automatically detects skewed partitions at runtime and splits them into smaller sub-partitions.

# Example: Key Salting in PySpark to fix join skew
import pyspark.sql.functions as F

# Add salt column (0 to 4) to large skewed dataset
df_large_salted = df_large.withColumn("salt", (F.rand() * 5).cast("int"))
df_large_salted = df_large_salted.withColumn("join_key_salted", F.concat(F.col("user_id"), F.lit("_"), F.col("salt")))

# Replicate dimension table 5 times with matching salts
df_small_exploded = df_small.withColumn("salt_array", F.array([F.lit(i) for i in range(5)])) \
                            .withColumn("salt", F.explode("salt_array")) \
                            .withColumn("join_key_salted", F.concat(F.col("user_id"), F.lit("_"), F.col("salt")))

# Perform balanced distributed join without straggler bottlenecks
df_result = df_large_salted.join(df_small_exploded, on="join_key_salted", how="inner")

Kafka achieves massive horizontal throughput via sequential disk I/O, OS page caching, and zero-copy data transfer (Linux sendfile system call). Each topic is subdivided into multiple partitions. For each partition, one broker acts as the Leader handling all read and write requests, while multiple Followers replicate the log in the background. The brokers keeping pace with the leader form the In-Sync Replicas (ISR) set. If the leader broker crashes, the cluster controller automatically elects a replacement leader from the ISR with zero data loss when producers use acks=all.

An idempotent pipeline guarantees that executing the pipeline multiple times with the same input produces the exact same output state as executing it once.
• Partition Overwrites: Write batch data using partition-level atomic overwrites (e.g., INSERT OVERWRITE TABLE transactions PARTITION (dt = '2026-09-24')) instead of simple appends.
• Upserts with Unique Keys: In lakehouses and warehouses, use MERGE INTO target USING source ON target.id = source.id WHEN MATCHED THEN UPDATE WHEN NOT MATCHED THEN INSERT.
• Deterministic Surrogate Keys: Generate primary keys using deterministic hashes of natural business keys (e.g., MD5(CONCAT(user_id, transaction_timestamp, amount))) rather than auto-incrementing integers.