--- ## Table 1 — Core Architecture & Execution Model | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Lazy Evaluation & DAG Optimization** | Spark defers computation of **transformations** until an **action** is called; it builds a **Directed Acyclic Graph (DAG)** of all operations and **optimizes the full chain** before execution. | Every Spark job — prevents partial, redundant computation | Enables plan-wide optimization; filters, projections, and stage merges applied before data touches the cluster | Excessively long lineage chains can cause memory pressure or stack overflows; use checkpointing to break them | `df.filter(col("status")=="active")`
`.select("id","name")`
`.count() # Action triggers DAG execution` | [1][2] | | **Catalyst Optimizer** | Spark SQL's rule-based query optimizer converts a **logical plan** through **analysis** → **logical optimization** → **physical planning** → **code generation**; applies rules like **predicate pushdown**, **column pruning**, and **constant folding** automatically. | SQL queries, DataFrame API | Automatic optimization with no user code changes; very broad coverage | Not always perfect for complex joins without statistics; supplement with CBO or hints | Run `df.explain(True)` to inspect Parsed, Analyzed, Optimized, and Physical Plans | [1][3] | | **Cost-Based Optimizer (CBO)** | Extends Catalyst with **data statistics** (row count, column NDV, min/max, histogram) to choose **join ordering and strategy** based on **estimated cost** rather than fixed heuristics. Enable with `spark.sql.cbo.enabled=true`. | Multi-join queries, star-schema analytics | Can reorder joins to minimize intermediate data; may produce dramatically better plans | Requires `ANALYZE TABLE … COMPUTE STATISTICS FOR COLUMNS …` to be run first; adds maintenance overhead | `spark.conf.set("spark.sql.cbo.enabled","true")`
`spark.sql("ANALYZE TABLE sales COMPUTE STATISTICS FOR COLUMNS id,amount")` | [3][4] | | **Tungsten Project (Memory & CPU Optimization)** | Low-level CPU/memory optimization engine: **off-heap binary encoding** of data (avoiding Java object overhead), **cache-aware sorting**, and **whole-stage code generation** (generates JVM bytecode for entire stage in one function). | All Spark SQL / DataFrame workloads automatically | Reduces GC pressure; improves CPU cache utilization; eliminates per-row virtual dispatch overhead | Enabled by default; manual tuning via `spark.sql.codegen.wholeStage=true`; some complex expressions fall back to interpreted mode | Code-gen fuses a filter+aggregate stage into a single tight loop — inspect via `explain()` looking for `*(n)` markers in physical plan | [5][6] | | **Whole-Stage Code Generation** | Part of Tungsten — **generates a single optimized Java function** per stage, eliminating the overhead of the **Volcano iterator model** (virtual function calls per row). | CPU-intensive aggregations, joins, filters at scale | 2–10× CPU speedup for tight loops; works best with primitive types | Disabled automatically for certain complex operations (non-serializable, python UDFs); check with `explain()` for `WholeStageCodegen` | Enabled by default since Spark 2.0; disable for debugging: `spark.sql.codegen.wholeStage=false` | [5] | | **Vectorized Read (Parquet/ORC)** | Spark's vectorized reader processes data in **columnar batches** instead of row-by-row, drastically reducing per-record overhead when reading **Parquet and ORC files**. Controlled by `spark.sql.parquet.enableVectorizedReader`. | Analytical workloads reading large Parquet/ORC tables | 2–5× faster reads; works synergistically with Photon on Databricks | Only available for supported types; complex nested schemas may fall back; requires Parquet/ORC format | `spark.conf.set("spark.sql.parquet.enableVectorizedReader","true")`
`df = spark.read.parquet("/data/sales/")` | [1][7] | --- ## Table 2 — Memory Management & Caching | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **`cache()` / `persist()` — MEMORY_AND_DISK** | Stores a **DataFrame or RDD in memory** (spilling to disk if needed) to avoid **re-computation** on subsequent actions. `cache()` is shorthand for `persist(MEMORY_AND_DISK)` for DataFrames. | Iterative algorithms (ML training), reused DataFrames in multi-step pipelines | Eliminates recomputation cost for frequently accessed data | Consumes executor memory; do NOT cache large DataFrames you use only once; call `unpersist()` when done | `df_clean = df.filter(...).persist()`
`df_clean.count(); df_clean.groupBy(...).agg(...)` | [8][9] | | **MEMORY_ONLY storage level** | Stores **deserialized Java objects** **entirely in JVM heap memory**; fastest access if data fits, but **evicts partitions silently** on memory pressure (those are recomputed on demand). | Small-to-medium RDDs accessed frequently | Lowest latency for re-reads; no disk I/O | Recomputes evicted partitions; not safe for expensive transformations if memory is tight | `from pyspark import StorageLevel`
`rdd.persist(StorageLevel.MEMORY_ONLY)` | [9][10] | | **MEMORY_AND_DISK_SER (serialized)** | Stores **serialized (compact binary) Java objects** in memory; excess spills to disk. Reduces **memory footprint** vs. deserialized at cost of CPU for ser/deser. | Large RDDs that barely fit in memory | Better memory efficiency than MEMORY_ONLY; prevents OOM evictions | Slight CPU overhead for serialization; use Kryo serializer for further gain | `rdd.persist(StorageLevel.MEMORY_AND_DISK_SER)` | [9][10] | | **OFF_HEAP Storage** | Stores data **outside JVM heap** using Tungsten's unsafe memory API; reduces **GC stop-the-world pauses** for large cached datasets. | Very large caches, GC-sensitive real-time workloads | Near-zero GC pressure for cached data | Requires `spark.memory.offHeap.enabled=true` and `spark.memory.offHeap.size` to be set; does not protect against container OOM | `spark.conf.set("spark.memory.offHeap.enabled","true")`
`spark.conf.set("spark.memory.offHeap.size","4g")` | [5][11] | | **Unified Memory Model Tuning** | Spark uses a **unified memory pool** split between **execution** (shuffle/sort/join buffers) and **storage** (cache). Tuned via `spark.memory.fraction` (default 0.6) and `spark.memory.storageFraction` (default 0.5). | Jobs mixing heavy caching AND expensive shuffles | Dynamic borrowing between storage and execution reduces OOM | Setting too high a storage fraction can starve execution memory, causing spills; measure both in Spark UI before tuning | `spark.conf.set("spark.memory.fraction","0.7")`
`spark.conf.set("spark.memory.storageFraction","0.4")` | [5][11] | | **Executor & Driver Sizing** | Right-sizing `spark.executor.memory`, `spark.executor.cores`, `spark.executor.memoryOverhead`; the **5-core rule** (5 cores per executor) balances **I/O throughput** and **GC efficiency**. | All cluster workloads | Reduces OOM errors, GC pressure, task starvation | Over-allocating executors reduces parallelism (fewer executors); `memoryOverhead` must cover off-heap and Python processes | `spark.executor.memory=8g`
`spark.executor.cores=5`
`spark.executor.memoryOverhead=2g` | [12][13] | | **GC Tuning (G1GC / ZGC)** | Switch from the default **Parallel GC** to **G1GC** (`-XX:+UseG1GC`) or Java 17+ **ZGC** for **lower GC pause times** in memory-heavy workloads. | Long-running Spark jobs with large heaps, streaming | G1GC reduces pause times for heaps >4 GB; ZGC provides near-pause-free collection | G1GC increases throughput overhead vs. Parallel GC; profiling needed to confirm benefit | Set in `spark.executor.extraJavaOptions=-XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=35` | [11][13] | | **Checkpointing (Reliable)** | Saves an RDD/DataFrame to **HDFS or cloud storage**, **truncating the lineage**. Used when lineage chains grow very long (e.g. iterative graph algorithms, streaming state). | GraphX, long iterative ML, complex streaming | Prevents stack overflows and recomputation of entire lineage on failure | I/O cost of writing to storage; triggers an action (eagerly materialized); requires `sc.setCheckpointDir(...)` | `sc.setCheckpointDir("dbfs:/checkpoints/")`
`rdd_iter.checkpoint(); rdd_iter.count()` | [14][15] | | **Local Checkpointing** | Saves RDD data **in executor's local storage** (faster than HDFS checkpoint) — trades **fault tolerance** for performance. Lineage is truncated but data is lost if executor fails. | Iterative jobs where fast truncation > fault tolerance | Much faster than reliable checkpointing; still breaks long lineages | Not fault-tolerant; not suitable for production pipelines where executor failures are likely | `rdd.localCheckpoint() # truncates lineage, stores locally` | [14][15] | --- ## Table 3 — Partitioning Strategies | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Shuffle Partition Tuning (`spark.sql.shuffle.partitions`)** | Default is **200** post-shuffle partitions — often far too many for small data or too few for large data. Tune based on: target **128–200 MB per partition**. AQE can auto-coalesce if enabled. | All jobs involving joins, aggregations, sorts (wide transforms) | Massive impact on performance; too many = overhead; too few = OOM / slow tasks | With AQE enabled, set a larger ceiling (e.g. 2000) and let AQE coalesce down; without AQE, calculate manually | `spark.conf.set("spark.sql.shuffle.partitions", "500")` | [16][17] | | **`repartition(n)` — Full Shuffle** | Performs a **full shuffle** to produce exactly `n` **evenly-distributed partitions**; can also **repartition by column** to co-locate related data. | Before expensive joins on large data, fixing skewed partitions | Guarantees even distribution; enables column-based locality for joins | Full shuffle is expensive; avoid just before write (use `coalesce` to reduce instead) | `df.repartition(200, col("customer_id")).write.parquet(...)` | [16][18] | | **`coalesce(n)` — Partition Reduction** | Reduces partitions **without a full shuffle** by **merging local partitions**; much cheaper than `repartition` when decreasing partition count. | After heavy filtering that reduces data volume, before writing fewer output files | No shuffle cost; fast partition reduction | Can only **decrease** partitions (without shuffle); produces uneven partitions if source data is skewed | `df.filter(col("active")==True).coalesce(50).write.parquet(...)` | [16][18] | | **Static Partition Pruning** | Writing data with Hive-style **directory partitioning** (`partitionBy("date","region")`) allows Spark to **skip entire directories** based on WHERE clause values known at query planning time. | Date/region partitioned tables, incremental ingestion patterns | Eliminates file listing and reading of irrelevant partitions entirely | Low-cardinality columns only (< ~10,000 distinct values); too many partitions → **small file problem**; partition column should align with query patterns | `df.write.partitionBy("year","month").parquet("/data/events/")` | [18][19] | | **Dynamic Partition Pruning (DPP)** | Spark 3.0+ feature: at **runtime**, builds a **subquery result from the dimension table** and uses it as a **filter on the fact table's partitions** — eliminating irrelevant partitions even when filter values are only known after a join. Enabled by `spark.sql.optimizer.dynamicPartitionPruning.enabled`. | Star-schema joins (large fact table + small dimension), data warehousing | Reduces I/O dramatically on partitioned fact tables; automatic (no code change) | Both tables must be partitioned on the join key; fact table must be large enough to justify the subquery overhead | `SELECT * FROM orders o JOIN dates d ON o.date_id=d.id WHERE d.year=2024` — Spark prunes `orders` partitions at runtime | [19][20] | | **Data Skew Handling — Salting** | When a join/group-by key has a **skewed distribution** (a few values dominate), "**salt**" the hot key by appending a random integer (0–N) to **spread records across `N` partitions**, then salt the other side by replicating `N` times. | Joins or aggregations with hot keys (e.g. top user IDs, null keys) | Eliminates stragglers; balances task duration | Doubles the data on the replicated side; post-salt aggregation required; complicates logic | `df.withColumn("salt", (rand()*10).cast("int"))`
`df.withColumn("salted_key", concat(col("key"),lit("_"),col("salt")))` | [18][21] | | **Bucketing** | **Pre-shuffles and sorts data** by a key **at write time** into a fixed number of "**buckets**"; future joins or aggregations on the same bucketed key can **skip the shuffle entirely**. | Repeated joins on the same large tables, Hive-interop scenarios | Eliminates shuffle for matching-bucket joins; saved as metadata in metastore | Requires same bucket count and key for both tables; only effective for repeated queries; not supported in all file formats equally | `df.write.bucketBy(64,"customer_id").sortBy("customer_id").saveAsTable("customers_bucketed")` | [18][22] | | **Custom Partitioner (RDD API)** | Implement a **`Partitioner` class** to define exactly which partition each key maps to; useful for **domain-specific co-location requirements**. | Graph processing, custom aggregation frameworks, low-level RDD pipelines | Full control over data locality; can prevent shuffle for specific patterns | Requires RDD API (not DataFrame); partition logic must be carefully balanced to avoid skew | `rdd.partitionBy(HashPartitioner(200))` | [2] | --- ## Table 4 — Join Strategies & Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Broadcast Hash Join (BHJ)** | The **smaller table is broadcast** (copied in full) to every executor; larger table is scanned locally and hash-joined with **no shuffle**. Triggered when small table is below `spark.sql.autoBroadcastJoinThreshold` (default 10 MB). | Joining a large table with a small lookup/dimension table | Eliminates shuffle entirely for the large table; usually the fastest join strategy | Small table must fit in executor memory (broadcast threshold); can cause driver OOM if threshold is too high; not supported for full outer joins | `from pyspark.sql.functions import broadcast`
`result = large_df.join(broadcast(small_df), "id")` | [1][23][24] | | **Sort-Merge Join (SMJ)** | Both tables are **shuffled on join key**, **sorted**, then **merged**; Spark's **default strategy for large-large joins**. Supports all join types and handles data larger than memory (disk spill). | Large-large table joins where neither side fits in memory | Robust, handles all join types, disk-safe via spill | Requires 2 shuffles + sort; slow for skewed data; overtaken by BHJ when possible | Default when both sides exceed `autoBroadcastJoinThreshold`; confirm with `explain()` showing `SortMergeJoin` | [23][24] | | **Shuffle Hash Join (SHJ)** | **Shuffles both tables** on join key then builds a **hash table from the smaller side** in each partition; no sort required. Faster than SMJ when smaller side fits in partition memory. | Medium-sized joins where smaller side fits per partition; hash-friendly data distributions | Avoids sort overhead; can be faster than SMJ for non-skewed data | Smaller side must fit in per-partition memory or will OOM; Spark prefers SMJ by default unless forced via hint | `df_large.hint("shuffle_hash").join(df_medium, "id")` | [23][24] | | **Cartesian / Broadcast Nested Loop Join** | **Full cross-join fallback** when no join key exists; the **worst-case strategy** — O(m×n). May also appear for non-equi join conditions. | Cross joins (explicitly intended), non-equi-joins | Handles any join condition | Catastrophically expensive at scale; always check `explain()` if you see this unexpectedly | Avoid; explicitly use `.crossJoin()` only when intentional | [23] | | **Join Hints** | Explicitly tell the optimizer which strategy to use via **SQL hints or DataFrame API hints**: `BROADCAST`, `MERGE`, `SHUFFLE_HASH`, `SHUFFLE_REPLICATE_NL`. **Overrides optimizer decisions**. | Correcting suboptimal optimizer choices; forcing broadcast when stats are stale | Provides user control over plan; useful when CBO/AQE can't detect the right strategy | Misuse (e.g. broadcasting a large table) causes OOM; not portable across platforms | `df.hint("broadcast").join(other, "key")` or SQL: `SELECT /*+ BROADCAST(t1) */ ...` | [23][24] | | **Skew Join with AQE** | AQE automatically **detects and handles skewed partitions** in Sort-Merge Joins by **splitting large skewed partitions** into smaller sub-partitions. Controlled by `spark.sql.adaptive.skewJoin.enabled`. | Any SMJ where one or a few keys have disproportionate row counts | Automatic; no code change needed in Spark 3.x+ | AQE must be enabled; skew threshold tunable via `spark.sql.adaptive.skewJoin.skewedPartitionFactor` and `skewedPartitionThresholdInBytes` | `spark.conf.set("spark.sql.adaptive.skewJoin.enabled","true")`
`spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor","5")` | [17][25] | | **Skew Hints (Manual)** | Inform Spark about **known skewed keys** using **`SKEW` hint in SQL**, allowing targeted **split-and-replicate treatment** of those specific values. | Known hot keys (e.g. `NULL`, a dominant customer_id) | Precise; avoids overhead of AQE threshold detection | Requires knowledge of skewed keys; less maintainable than AQE auto-detection | `SELECT /*+ SKEW('orders', 'customer_id', (0, 1234)) */ ...` | [25] | | **Range Partitioning for Ordered Joins** | For **joins/merges on ordered/range keys** (dates, numeric IDs), **range-partition both sides identically** to ensure co-located partitions, avoiding full cross-partition shuffle. | Time-series joins, range-key joins | Reduces shuffle volume for range-key patterns | Requires knowledge of data distribution for even range splits; skewed ranges still cause skew | `df.repartitionByRange(200, col("event_date")).write.parquet(...)` | [18] | --- ## Table 5 — Adaptive Query Execution (AQE) | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **AQE — Overview & Enable** | **AQE** (**Spark 3.0+**, default-on in Spark 3.2+) **re-optimizes the query plan at runtime** using actual **shuffle statistics** — resolves the fundamental weakness of static planning. Umbrella config: `spark.sql.adaptive.enabled`. | All SQL/DataFrame jobs on Spark 3.2+; Databricks always-on | Automatic improvement with no code changes; subsumes many manual tuning tasks | Adds minor overhead for statistics collection at shuffle boundaries; can be disabled for debugging | `spark.conf.set("spark.sql.adaptive.enabled","true")` (already default) | [17][25][26] | | **AQE — Dynamic Partition Coalescing** | **Post-shuffle**, AQE **merges small partitions** into larger ones to avoid the overhead of **thousands of tiny tasks**. Controlled by `spark.sql.adaptive.coalescePartitions.enabled` and `spark.sql.adaptive.advisoryPartitionSizeInBytes` (default 64 MB). | Any job producing many small shuffle partitions after aggregation/filtering | Reduces task scheduling overhead; eliminates "too many small tasks" problem | Set `spark.sql.adaptive.coalescePartitions.initialPartitionNum` large (e.g. 2000) to give AQE room to work | `spark.conf.set("spark.sql.adaptive.advisoryPartitionSizeInBytes","128m")` | [17][25] | | **AQE — SMJ→BHJ Conversion** | AQE monitors **actual data size** after shuffle; if one side is **smaller than broadcast threshold**, it **dynamically converts Sort-Merge Join to the cheaper Broadcast Hash Join** at runtime. | Any query where the static size estimate was wrong (stale stats, complex subqueries) | Automatic broadcast promotion without manual hints; saves a full shuffle | Broadcast threshold applies (`spark.sql.adaptive.autoBroadcastJoinThreshold`); very small tables were already broadcast statically | `spark.conf.set("spark.sql.adaptive.autoBroadcastJoinThreshold","30m")` | [17][26] | | **AQE — Skew Join Optimization** | **Detects and splits skewed partitions** in Sort-Merge Joins into **sub-partitions**, replicating the corresponding non-skewed side as needed to balance work. | All SMJ with hot keys — the most impactful AQE feature for production pipelines | Eliminates **long-tail tasks** without code change; adapts to per-job data characteristics | Can increase total data read (replication overhead); threshold parameters require tuning for very extreme skew | `spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes","256m")` | [17][25] | | **AQE — Runtime Bloom Filter (Spark 3.3+)** | Generates a **Bloom filter** from the **build side of a join** at runtime and applies it to the **probe side** to **pre-filter rows before the actual join**, reducing data movement. Enable with `spark.sql.optimizer.runtime.bloomFilter.enabled`. | Large-small joins, semi-joins where BHJ threshold is too low | Reduces shuffle and join cost; complements DPP | Bloom filters have false positives; most effective for high-selectivity joins; Spark 3.3+ required | `spark.conf.set("spark.sql.optimizer.runtime.bloomFilter.enabled","true")` | [27][28] | --- ## Table 6 — Shuffle & Network Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Minimize Wide Transformations** | **Wide transformations** (join, groupBy, distinct, repartition) **cause shuffles** — data movement across the network. Design pipelines to push **narrow transformations** (filter, select, map) **before wide ones** to reduce data shuffled. | Pipeline design / query architecture | Reduces network I/O, the primary bottleneck in distributed Spark | Sometimes unavoidable; use AQE and predicate pushdown to reduce shuffle data size automatically | Apply `filter()` and `select()` before `join()` or `groupBy()` to reduce rows/columns early | [1][16] | | **Shuffle Service Tuning** | Configure the **External Shuffle Service** (on YARN/Kubernetes) to **serve shuffle files** even when executors are removed during **dynamic allocation**. Tunable via `spark.shuffle.service.enabled`. | Dynamic allocation enabled clusters; long-running jobs | Allows safe deallocation of idle executors during shuffle reads; prevents data loss | Adds infrastructure complexity; shuffle data is still ephemeral (no replication); large shuffle files can saturate disk I/O | Enable on Databricks clusters by default when using enhanced autoscaling | [12][29] | | **Shuffle Compression** | **Compress shuffle files** using **LZ4** (default) or **Snappy** to reduce **disk write/read and network transfer**. Controlled by `spark.shuffle.compress` and `spark.io.compression.codec`. | All shuffles on CPU-abundant clusters | Reduces I/O and network cost; LZ4 is fast enough to not add CPU bottleneck | Compression adds CPU overhead; avoid for already-compressed data; ZSTD offers better ratio at slight CPU cost | `spark.conf.set("spark.io.compression.codec","lz4")`
`spark.conf.set("spark.shuffle.compress","true")` | [29][30] | | **Reduce Shuffles via Broadcast** | **Proactively broadcast small DataFrames/lookup tables** to avoid **shuffle-intensive joins**. Even for tables slightly above default threshold, manually broadcasting can be worthwhile. | Any join where one side is small (< hundreds of MB) | Eliminates the shuffle for the probe side entirely | Risk of OOM if table is larger than available executor memory; measure actual size, not estimate | `large.join(broadcast(lookup), "id") # explicit broadcast` | [23][24] | | **Sort Shuffle Manager (Default)** | Spark's default shuffle manager: **SortShuffleManager** writes **sorted shuffle files** with an **index** for efficient fetch. Uses a **bypass mechanism** for small number of output partitions (< `spark.shuffle.sort.bypassMergeThreshold` = 200). | General-purpose; default for all Spark jobs | Stable, well-tested; supports disk spill; efficient index-based fetch | Sorting step adds latency; for jobs not requiring sort, the bypass mechanism helps, but avoid setting `bypassMergeThreshold` too high | `spark.conf.set("spark.shuffle.sort.bypassMergeThreshold","400")` — only for small-partition jobs | [29][30] | | **Push-Based Shuffle (Spark 3.2+)** | Executors **push shuffle blocks to remote shuffle service** during map phase rather than having reducers pull them; allows **merge of shuffle blocks** on the service side, reducing **random I/O** during reduce. | Large-scale jobs on Databricks / YARN with External Shuffle Service | Reduces random I/O for reducers; improves tail latency | Requires Databricks Runtime or configured External Shuffle Service; not available in all environments | Enabled automatically in Databricks Runtime 11.0+; configure via `spark.shuffle.push.enabled=true` | [29] | | **Tungsten Unsafe Shuffle (UnsafeShuffleManager)** | Legacy (Spark 1.4–2.x): **binary-encoded, off-heap-friendly sort shuffle** that bypasses **Java object overhead** during sort. Now absorbed into the default SortShuffleManager via Tungsten. | Large-scale sorts, Tungsten-enabled workloads | Reduces memory footprint of sort buffer; GC-friendly | No longer a separate option in modern Spark; understand its legacy when reading old documentation | Historical: was enabled via `spark.shuffle.manager=tungsten-sort` (no longer valid in Spark 3+) | [29][30] | --- ## Table 7 — Delta Lake & Storage-Level Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **OPTIMIZE (File Compaction)** | Databricks/Delta Lake command that **compacts small Parquet files** in a Delta table into larger (default ~1 GB) files via **bin-packing**. Eliminates the **"small file problem"** from streaming writes or frequent inserts. | Tables with many small files; scheduled maintenance jobs | Faster reads (fewer files to list and open); improves data skipping; reduces metadata overhead | Write-intensive tables re-accumulate small files; run on a schedule; not compatible with Liquid Clustering (use `OPTIMIZE FULL` instead) | `OPTIMIZE sales_delta`
`-- or partition-scoped:`
`OPTIMIZE sales_delta WHERE event_date >= '2024-01-01'` | [31][32] | | **Z-Ordering (ZORDER BY)** | **Co-locates related data** within Parquet files by **multi-dimensional sorting** on high-cardinality columns; when a query filters on those columns, **file-level statistics** allow **skipping whole files**. | Tables queried with equality/range filters on high-cardinality columns (user_id, product_id) | **10–100× speedup** for selective queries via **data skipping**; complements OPTIMIZE compaction | Must be re-run periodically (not incremental); cannot be combined with Liquid Clustering; cardinality matters — low-cardinality columns get little benefit | `OPTIMIZE orders ZORDER BY (customer_id, order_date)` | [31][32][33] | | **Liquid Clustering (Databricks Runtime 13.3+)** | Next-generation replacement for Z-Ordering: **incremental**, **online re-clustering** that runs automatically via `OPTIMIZE`; columns can be changed without **rewriting all data**. No partitioning required. | New tables on DBR 13.3+; columns frequently used in filters; replaces static partitioning+ZORDER | Incremental (only newly added data is clustered on each run); flexible column changes; no manual partition management | Not backward-compatible with ZORDER or manual partitioning; requires Delta Lake 3.1+ and Databricks Runtime 13.3+ | `CREATE TABLE events CLUSTER BY (customer_id, event_date) ...`
`OPTIMIZE events -- incremental recluster` | [32][34] | | **Data Skipping (Min/Max Statistics)** | Delta Lake automatically collects **min/max/null statistics** for the first 32 columns of each Parquet file at write time. Queries with **WHERE clauses** on those columns **skip irrelevant files** without reading them. | Any analytical query with selective filters on statistics-covered columns | Automatic; zero-cost at query time; complements ZORDER/Liquid Clustering | Only covers first 32 columns (configurable); string columns are expensive to collect stats for; stats become stale if columns change | Controlled via `delta.dataSkippingNumIndexedCols=32` table property | [31][32] | | **Bloom Filters (Delta Lake / Spark 3.3+)** | **Probabilistic index** that quickly determines if a file **definitely does NOT contain** a given value for a column — ideal for **high-cardinality string columns (UUIDs, emails)** where min/max statistics are ineffective. | Point-lookup queries on string/hash columns; complement to ZORDER | Near-zero false-negative rate; space-efficient; query-time I/O reduction | False positives (files may be read unnecessarily); must be explicitly created and maintained; adds write overhead | `CREATE BLOOMFILTER INDEX ON TABLE users FOR COLUMNS (user_uuid OPTIONS (fpp=0.01))` | [32][35] | | **VACUUM** | **Removes Parquet files no longer referenced** by Delta Lake (deleted, overwritten, or compacted files). Retention period default: 7 days (for time travel). | Scheduled table maintenance; storage cost management | Reduces storage costs; improves metadata performance; prevents stale file accumulation | Too-aggressive VACUUM (< 7 days) breaks time travel; never VACUUM below active streaming reader checkpoint horizon | `VACUUM sales_delta RETAIN 168 HOURS` | [31][32] | | **Auto Compaction & Optimized Writes** | Databricks-specific: **Optimized Writes** dynamically **coalesces shuffle partitions before writing** to produce right-sized files; **Auto Compaction** runs **lightweight compaction after writes** asynchronously. Both reduce small file accumulation from streaming/micro-batch writes. | Delta tables written by streaming jobs or frequent micro-batches | Reduces manual OPTIMIZE frequency; zero-code improvement | Not a full substitute for scheduled OPTIMIZE+ZORDER; auto compaction is asynchronous (may lag under heavy write load) | `ALTER TABLE events SET TBLPROPERTIES ("delta.autoOptimize.optimizeWrite"="true", "delta.autoOptimize.autoCompact"="true")` | [32][36] | | **Predictive Optimization (Databricks)** | **ML-driven background service** that **automatically schedules OPTIMIZE and VACUUM** for Unity Catalog managed tables based on query patterns, write frequency, and cost-benefit analysis. | Unity Catalog managed tables on Databricks; reduces operational toil | Eliminates manual maintenance scheduling; uses ML to determine optimal timing; reduces TCO | Only for Unity Catalog managed tables; limited observability currently; Databricks proprietary feature | Enable at account level: `ALTER TABLE t SET TBLPROPERTIES ("delta.enablePredictiveOptimization"="auto")` | [36][37] | | **Delta Change Data Feed (CDF)** | Enables capturing **row-level changes** (INSERT, UPDATE, DELETE) from Delta tables for **incremental processing downstream** — avoids full table scans for downstream pipelines. | CDC pipelines, incremental ETL, feature stores | Orders-of-magnitude cheaper than full re-reads for change detection | Small overhead on write path; must be enabled at table creation or via ALTER TABLE; requires Delta Lake 2.0+ | `ALTER TABLE events SET TBLPROPERTIES ("delta.enableChangeDataFeed"="true")`
`spark.read.format("delta").option("readChangeFeed","true").load(...)` | [32] | | **Delta Time Travel Optimization** | Reading **specific historical versions** of a table (via `VERSION AS OF` or `TIMESTAMP AS OF`) for audit, debugging, or rollback — without **duplicating data outside Delta**. | Audit requirements, debugging data quality issues, experiment reproducibility | Zero additional storage for recent history; fast for recent versions | Performance degrades for very old versions with many intervening changes; VACUUM removes access to old versions beyond retention | `spark.read.format("delta").option("versionAsOf","5").load("/delta/orders/")` | [32] | --- ## Table 8 — Data Formats & Compression | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Parquet (Columnar)** | Apache Parquet is the **de-facto standard columnar format** for Spark analytics; stores data column-by-column, enabling **column pruning** and highly efficient compression per column. Native **vectorized reader** in Spark. | Analytical workloads, data lake storage, intermediate results | Best balance of read speed, compression, and ecosystem compatibility; excellent **predicate pushdown support** | Write performance slightly worse than row formats; not ideal for streaming where rows are appended incrementally; schema evolution requires care | `df.write.format("parquet").option("compression","snappy").save("/data/")` | [38][39] | | **Delta Lake (Parquet + Transaction Log)** | Delta Lake wraps Parquet files with an **ACID transaction log** (JSON/Parquet `_delta_log`), enabling **ACID transactions, time travel, schema enforcement**, and all Delta optimizations (OPTIMIZE, ZORDER, etc.). | Production data lake tables; lakehouse architecture; tables requiring ACID | ACID guarantees; schema evolution; **DML (UPDATE, DELETE, MERGE)**; time travel; full Databricks optimization ecosystem | Slightly larger metadata footprint; not portable to non-Delta readers without conversion; primarily Databricks/open-source Delta | `df.write.format("delta").mode("append").save("/delta/events/")` | [31][32][33] | | **ORC (Optimized Row Columnar)** | Columnar format optimized for **Hive ecosystem** with built-in indexes (Bloom filter, min/max), lightweight compression, and ACID support. Often achieves **better compression than Parquet**. | Hive-centric workloads, Hadoop ecosystem, ACID table requirements outside Delta | Superior compression in Hive-native environments; built-in index structures; ACID support | Less popular in Spark-native/Databricks environments; lower community momentum vs Parquet; Databricks ecosystem prefers Delta/Parquet | `df.write.format("orc").option("compression","zlib").save("/data/")` | [38][39] | | **Avro (Row-Based)** | **Row-based serialization format** with **rich schema evolution**; ideal for streaming/write-heavy pipelines and Kafka integration. Schema is embedded in files. | Kafka message serialization, write-heavy streaming, schema-evolution-heavy ETL | Excellent schema evolution (forward/backward compatible); fast writes; Kafka native | Row-based → **poor column pruning** for analytics; larger files than Parquet for analytical reads; requires schema registry for Kafka | `df.write.format("avro").save("/data/events/avro/")` | [38][40] | | **Snappy Compression** | Default compression codec for Parquet in Spark; **fast compression/decompression** with moderate compression ratio (~2:1). CPU-light. | General-purpose Parquet/ORC; default for most Spark workloads | Fast; low CPU overhead; **splittable** when used with block-level codecs | Lower compression ratio than GZIP or ZSTD; splittability depends on file format context | `spark.conf.set("spark.sql.parquet.compression.codec","snappy")` | [30][38] | | **ZSTD Compression (recommended in modern Spark)** | **Zstandard codec** provides **better compression ratio than Snappy** at similar or better decompression speed; supported in Parquet, ORC, and Spark shuffle since Spark 2.3. **Recommended** for Databricks and large-scale workloads. | Large Parquet tables, shuffle compression, Delta Lake | **~30% better compression** than Snappy; fast decompression; tunable compression levels | Slightly higher compression CPU cost than Snappy at max levels; use level 1–3 for streaming | `spark.conf.set("spark.io.compression.codec","zstd")`
`spark.conf.set("spark.sql.parquet.compression.codec","zstd")` | [30][38] | | **LZ4 Compression** | **Ultra-fast compression** optimized for **low latency** (shuffle buffers, in-memory caching); lower compression ratio than Snappy but higher throughput. Default for shuffle. | Shuffle compression, intermediate result caching | Extremely fast; minimal CPU overhead | Lowest compression ratio of common codecs; use where speed > storage is the priority | `spark.conf.set("spark.io.compression.codec","lz4")` | [30] | | **GZIP / DEFLATE Compression** | High compression ratio codec; **best for archival / cold storage** where read frequency is low and storage cost is the priority. Much slower than Snappy/ZSTD for decompression. | Archival Parquet files, raw data landing zone | Smallest file sizes (~3–5:1 ratio); good for rarely-queried data | **Not splittable at file level** (not an issue for columnar Parquet row groups); slow decompression degrades query performance on hot data | `df.write.option("compression","gzip").format("parquet").save(...)` | [30][38] | --- ## Table 9 — Serialization, UDFs & Python/PySpark Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Kryo Serialization** | Faster, more compact alternative to Java's default serialization for **RDD serialization and network transfer**. Register classes upfront to avoid class name overhead. Enable via `spark.serializer=org.apache.spark.serializer.KryoSerializer`. | RDD-heavy workloads, custom Java/Scala objects in Spark | **10× faster and ~10× more compact** than Java serialization for user classes | Requires registering custom classes; not automatic for all types; DataFrame/SQL paths use **Tungsten encoding** (not Kryo) | `spark.conf.set("spark.serializer","org.apache.spark.serializer.KryoSerializer")`
`spark.conf.set("spark.kryo.registrationRequired","false")` | [41][42] | | **Prefer Built-in SQL Functions over UDFs** | Spark's `pyspark.sql.functions` library implements operations **natively in the JVM/Catalyst** — **always prefer these over Python UDFs** which require Python process round-trips. | Any transformation expressible with built-in functions | **No Python-JVM serialization overhead**; fully optimized by Catalyst; vectorizable by Photon | Built-in functions are extensive but don't cover all custom business logic; when unavoidable, use Pandas UDF | Replace `udf(lambda x: x.upper(), StringType())` with `col("name").cast(StringType())` or `upper(col("name"))` | [41][43] | | **Standard Python UDF** | Classic `@udf` decorator: executes Python function **row-by-row**, requiring **serialization/deserialization through Pickle** for each row via Py4J. **Slowest option**. | Simple custom logic; legacy code | Easy to write; supports arbitrary Python logic | Significant overhead: **row-by-row Pickle serialization + Python process switch**; 10–100× slower than built-ins; blocks Catalyst optimization | `@udf(returnType=StringType())`
`def clean(x): return x.strip().lower()` | [41][43][44] | | **Pandas UDF (Vectorized UDF)** | Processes data as **Pandas Series/DataFrame batches** using **Apache Arrow for zero-copy transfer** between JVM and Python; dramatically faster than row-based UDFs. Enabled with `@pandas_udf` decorator. | Custom ML inference at scale, complex vectorizable transformations, apply external Python libraries to Spark data | **10–100× faster** than standard Python UDF; enables NumPy/Pandas operations; **Arrow batches minimize serialization** | Pandas UDF requires Arrow dependencies and Pandas installed; function must handle Series input/output; memory usage is higher per batch | `from pyspark.sql.functions import pandas_udf`
`@pandas_udf("double")`
`def normalize(s: pd.Series) -> pd.Series:`
` return (s - s.mean()) / s.std()` | [43][44][45] | | **Arrow-Optimized Python UDF (Spark 3.5+)** | Python UDFs annotated with `useArrow=True` or enabled globally via `spark.sql.execution.pythonUDF.arrow.enabled=true`; **uses Arrow for serialization** instead of Pickle — faster than classic UDF with minimal code change. | Upgrading legacy UDFs without full rewrite; type-safe transformations | Better performance than cloudpickle UDF without requiring Pandas Series API; automatic type coercion | Requires Spark 3.5+; Arrow type mappings differ from Python native types; test output types carefully | `@udf(returnType="string", useArrow=True)`
`def upper(s: str) -> str: return s.upper()` | [44][46] | | **Scala/Java UDF via JVM** | Register UDFs written in **Scala or Java** from PySpark using `spark.udf.register()` with a Jar; these run **natively in the JVM** — no Python-JVM boundary at all. | Performance-critical custom logic for PySpark workflows; legacy Scala functions | **Runs entirely in JVM** — no serialization boundary; Catalyst can potentially inline; fastest custom logic | Requires JVM language expertise; adds deployment complexity (Jar management); less accessible for Python-first teams | `spark.udf.register("my_java_udf", callJavaFunc)` (after uploading Jar via `spark.addJar(...)`) | [43][44] | | **Spark Connect / Client-Server Separation (Spark 3.4+)** | Spark Connect separates the driver from the cluster with a **gRPC API**, enabling **lightweight Python clients**. Reduces driver memory; enables remote Spark usage from IDEs / notebooks efficiently. | Databricks Notebooks, multi-tenant environments, remote development | **Decouples client from cluster**; enables multiple simultaneous client connections; reduces driver OOM risk | New API; some features (RDD API, StreamingQuery management) not yet fully supported in all versions; network latency between client and server | Available in Databricks Runtime as default for notebook connection mode | [47] | | **`mapInPandas` / `mapInArrow`** | Apply a Python function to each **partition as a Pandas/Arrow DataFrame** — more flexible than Pandas UDF (can change schema, return different number of rows per group). | Complex per-partition transformations, Python ML model inference per partition | **Full DataFrame flexibility** per partition; Arrow-based for efficiency | Requires careful schema management; no automatic type inference; best for non-standard transforms | `df.mapInPandas(lambda iter: (process(pdf) for pdf in iter), schema)` | [43][45] | --- ## Table 10 — Query Plan & I/O Push-Down Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Predicate Pushdown (Filter Pushdown)** | Moves **WHERE-clause filters as close to the data source as possible** — ideally into the data source reader itself — so only matching rows are loaded into Spark. Automatic via Catalyst for Parquet, Delta, JDBC, and many other sources. | Any query with WHERE filters; critical for large tables with selective filters | Reduces **bytes read and I/O dramatically**; automatic for supported sources; no code change needed | Verify it's working via `explain()` — look for `PushedFilters` in physical plan; complex expressions may not be pushed | `df.filter(col("region")=="EMEA").explain() # Should show PushedFilters: [EqualTo(region,EMEA)]` | [1][48][49] | | **Column Pruning (Projection Pushdown)** | Reads **only the columns needed** by the query instead of all columns — critical for wide Parquet/columnar tables where most columns are unused. Catalyst does this automatically when using `.select()`. | Analytical queries on wide tables; feature engineering pipelines | Dramatically reduces **I/O for columnar formats**; reduces deserialization cost; free performance gain | Automatic for Parquet/Delta/ORC; row formats (CSV, Avro) must read all columns regardless; always `select()` only needed columns | `df.select("id","amount","date").filter(...) # Only 3 columns read from Parquet` | [48][49] | | **JDBC Source Partitioning** | **Parallelize reads from relational databases** by specifying `partitionColumn`, `lowerBound`, `upperBound`, and `numPartitions` — causes Spark to issue **multiple queries in parallel**, each reading a range slice. | Ingesting from RDBMS (Postgres, MySQL, SQL Server) at scale | **Parallelizes** what would otherwise be a single serial read; saturates available network and DB threads | `partitionColumn` must be numeric, date, or timestamp; choose evenly distributed column to avoid skew; can overload source DB | `spark.read.jdbc(url, "orders",`
` column="id", lower=0, upper=10000000,`
` numPartitions=20, properties=props)` | [50][51] | | **Aggregate Pushdown to Source** | For JDBC and some connectors (Iceberg, Delta Sharing), Spark can **push COUNT, SUM, MIN, MAX aggregations into the source engine** to avoid full data transfer. Enabled via `spark.sql.sources.aggregate.enabled`. | Simple aggregate queries on JDBC tables | Reduces network transfer to just the aggregate result; leverages DB-side indexing | Only works for certain aggregations and connectors; complex aggregations fall back to Spark; verify via `explain()` | `spark.conf.set("spark.sql.optimizer.aggPushDown.enabled","true")` | [48] | | **Partition Elimination (Static, File System Level)** | For Hive/Parquet tables partitioned by directory (`partitionBy("year","month")`), Spark's metastore and file system listing combined with query filters **eliminate entire directory scans** before file reads begin. | Date-partitioned event/log tables; region-partitioned datasets | **Zero bytes read** for pruned partitions; scales to petabyte-level tables | Requires filtering on the partition column; must use correct column type (string vs int); too many partitions slow down metastore listing | `spark.read.parquet("/data/events/").filter(col("year")==2024).explain() # Shows partition pruning` | [18][19] | | **Higher-Order Functions (Arrays/Maps)** | Spark 2.4+ built-in functions (`transform`, `filter`, `aggregate`, `map_filter`, etc.) operate on **array/map columns natively inside the JVM** — much faster than explode+groupBy patterns and **eliminates the need for UDFs** on complex types. | Complex type manipulation (arrays of structs, nested maps); recommendation systems; feature engineering | **Fully native JVM execution**; no explode overhead; supports vectorization | SQL syntax can be complex; less familiar to developers from non-FP backgrounds | `df.withColumn("doubled", transform(col("amounts"), lambda x: x * 2))`
or SQL: `SELECT transform(amounts, x -> x * 2) FROM t` | [52][53] | | **Runtime Bloom Filter (Row-Level)** | Beyond join acceleration, Spark 3.3+ can use **runtime Bloom filters as row-level filters** during table scans when joined with a small dimension — reduces probe-side rows before join. Enabled with `spark.sql.optimizer.runtime.bloomFilter.enabled`. | Selective star-schema joins; large fact table scans | Reduces rows transferred in shuffle; complements predicate pushdown | False positives mean some rows pass through unnecessarily; most impactful for joins with selectivity > 90% | `spark.conf.set("spark.sql.optimizer.runtime.bloomFilter.enabled","true")`
`spark.conf.set("spark.sql.optimizer.runtime.bloomFilter.creationSideThreshold","10m")` | [27][28] | --- ## Table 11 — Cluster & Infrastructure Configuration | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Cluster Auto-Scaling** | Databricks/Spark clusters **dynamically add/remove worker nodes** based on pending tasks and idle time. Reduces cost by **shrinking during idle periods and expanding under load**. | Variable-load workloads, ETL jobs, interactive analytics | Can **reduce costs by 60–80%** vs. fixed clusters; zero manual intervention; handles spikes automatically | Auto-scaling adds node spin-up latency; poorly tuned min/max bounds cause thrashing; enhanced autoscaling (Databricks) is smarter than standard | Set `min_workers=2, max_workers=20` in cluster config; use Databricks Enhanced Autoscaling for streaming | [12][54][55] | | **Spot / Preemptible Instances** | Use **interruptible cloud instances** (AWS Spot, Azure Spot, GCP Preemptible) for worker nodes — typically **60–90% cheaper than on-demand**. Combine with auto-scaling for maximum savings. | Batch ETL jobs tolerant of interruption; non-SLA-critical analytics | **Massive cost reduction** (up to 80% combined with autoscaling); Databricks handles Spot interruptions automatically for workers | Not suitable for driver node (kills job); streaming jobs need careful checkpoint configuration; prices fluctuate | Use Spot instances for workers only; keep driver on on-demand; set `spot_bid_max_price` or use capacity-optimized strategy | [54][55] | | **Cluster Pools (Instance Pools)** | Pre-allocate a pool of **idle, ready-to-use VMs** that clusters draw from instantly — **eliminates the 2–5 minute VM provisioning latency** for new clusters. | Interactive notebooks, short-lived jobs, CI/CD testing | **Near-instant cluster start times**; eliminates cold-start latency | Idle VMs in pool still incur VM costs (no DBU cost though); size pool carefully to actual demand | Create pool with min_idle=5 instances; attach cluster policies to force pool use | [54][55] | | **Auto-Termination** | Configure clusters to **automatically shut down after N minutes of inactivity** to prevent paying for idle compute. | Interactive / ad-hoc exploration clusters | Prevents forgotten clusters from running indefinitely; **significant cost saving** | Set appropriately for use case — too short interrupts long-running interactive sessions | `auto_termination_minutes: 30` in cluster policy | [54][55] | | **Instance Type Selection (Memory vs Compute vs Storage)** | Choose VM family based on workload: **memory-optimized** (r-series, Lseries) for cache-heavy / large shuffle; **compute-optimized** (c-series) for CPU-heavy aggregations; **storage-optimized** for local disk shuffle | Different workload profiles require different hardware | Right-sizing hardware can **improve performance 2–5×** vs. generic instances; cost-efficiency improves significantly | Over-provisioning memory for CPU-bound jobs wastes money; use Spark UI to identify whether CPU, memory, or I/O is the bottleneck first | Analyze Spark UI: high GC time → memory-optimized; high task CPU → compute-optimized; high shuffle spill → storage-optimized | [12][55] | | **Cluster Policies** | Databricks admin feature to **enforce configurations** (max DBUs, allowed instance types, auto-termination, required libraries) across user-created clusters; prevents misconfigurations and cost overruns. | Enterprise multi-tenant Databricks deployments; cost governance | Ensures best practices are applied by default; **prevents runaway costs**; security/compliance enforcement | Requires Databricks Premium tier; overly restrictive policies frustrate developers | Define policy with `spark.sql.shuffle.partitions` default, max cluster size, and auto-termination enforced | [55] | | **Databricks Runtime (DBR) Version Selection** | Always use the **latest LTS Databricks Runtime** unless pinned by dependency constraints; newer versions include **Photon improvements, AQE enhancements, bug fixes, and security patches**. | All Databricks workloads | Free performance improvements; latest optimization features automatically available | Breaking changes between major DBR versions; ML-specific runtimes (DBR ML) include additional libraries | Use DBR 15.4 LTS (current LTS) or latest unless constrained; use DBR ML for ML workloads | [56] | | **Driver Configuration & Sizing** | The driver aggregates all task results and manages the DAG scheduler; it must be sized to handle the **largest broadcast table, collect() results, and Python session data**. **Driver OOM** causes full job failure. | Any job using `collect()`, large broadcasts, or complex scheduling | **Prevents cascading OOM failures**; critical for interactive notebooks | Driver is a single point of failure; avoid running on the driver anything that scales with data size; monitor driver heap in Spark UI | Driver memory should be ≥ largest broadcast table + overhead; typically 2× a single executor for most jobs | [12][13] | --- ## Table 12 — Databricks Platform-Specific Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Photon Engine (Vectorized Execution)** | Databricks-native **C++ vectorized query engine** that processes **columnar batches** instead of rows; runs SQL/DataFrame workloads at **native CPU speed using SIMD instructions**. Automatically used when available. | SQL queries, DataFrame operations, Delta Lake reads/writes on Databricks | Reported **2–8× speedup** for SQL workloads vs. Java Spark; lower cost-per-query; handles NULL-heavy data efficiently | Databricks proprietary (not open-source); not all operations supported (complex Python UDFs fall back to JVM Spark); requires Photon-enabled cluster type | Enable "Photon Acceleration" in Databricks cluster creation UI; verify in query plan with `PhotonResultStage` | [57][58] | | **Databricks SQL Warehouses (Serverless)** | Fully managed SQL compute with **sub-10-second cold start** (vs. minutes for classic clusters); auto-scales, auto-tunes, and includes Photon; optimized specifically for BI/SQL workloads. | BI dashboards, ad-hoc SQL analytics, concurrent user workloads | No cluster management; intelligent caching (disk cache); automatic query optimization; fast startup | Higher per-DBU price than classic compute; less flexibility for custom Spark configuration; Python/streaming not supported | Use serverless warehouse for Databricks SQL; use job compute for PySpark ETL jobs | [56][59] | | **Disk Cache (Databricks Local SSD Cache)** | Databricks-specific: **caches Parquet/Delta remote file data to local SSD (NVMe)** on each executor node; subsequent reads **skip cloud storage entirely**. Enabled by default on SSD-equipped instance types. | Iterative analytics over the same data; dashboards reading same table repeatedly | **Near-disk-speed reads** for cached data; transparent to user; no code change | Only caches remote reads (not in-memory transformation results); cache is populated on first access; cache is per-cluster (evicted on termination) | Use cache-accelerated instance types (e.g. `i3`, `i4i` on AWS, `Lseries` on Azure); monitor via Spark UI's Storage tab | [56][57] | | **Delta Live Tables (DLT)** | Databricks declarative framework for building **reliable, testable data pipelines** with **automatic data quality enforcement (expectations)**, incremental processing, and managed orchestration. | Production ETL/ELT pipelines, medallion architecture implementations, streaming+batch unified pipelines | **Automatic lineage tracking**; built-in data quality expectations; incremental processing by default; simplified orchestration | Higher abstraction = less flexibility for non-standard patterns; some performance overhead vs. hand-tuned Spark; Databricks proprietary | `@dlt.table`
`def silver_orders():`
` return dlt.read("bronze_orders")`
`.filter(col("amount") > 0)` | [60] | | **Unity Catalog Optimization** | Databricks' unified governance layer: column-level lineage, fine-grained access control (row/column filters, data masking), and **predictive optimization integration** — all with **minimal performance overhead** when designed correctly. | Enterprise data governance; multi-workspace data sharing | Centralized governance; automatic lineage; enables **Predictive Optimization** | Row-level security and column masking add per-query overhead; tag-based ABAC (Attribute-Based Access Control) is more performant than per-table row filter functions | Use ABAC tags for bulk access control instead of per-table row filter functions to minimize overhead | [37][61] | | **Databricks Workflows (Jobs) Optimization** | Use **job clusters** (not all-purpose clusters) for production jobs — they start fresh, use latest DBR, **auto-terminate on completion**, and cost **35–40% less DBUs** than all-purpose clusters. Chain tasks in a DAG for dependency management. | Production ETL, scheduled analytics pipelines | **Cheaper DBU rates**; guaranteed clean environment; DAG-based task dependencies enable parallelism | No interactive exploration during job run; cold-start latency (use cluster pools to mitigate) | Create job with `job_clusters` config; use `depends_on` to express task DAG; use `cluster_pools` for fast start | [55] | | **MLflow Integration for Experiment Tracking** | Databricks integrates MLflow for **tracking experiments, logging parameters, and registering models**; enables reproducibility and comparison of ML optimization attempts. | ML model training, hyperparameter tuning, A/B experiment comparison | **Automatic experiment tracking** in Databricks notebooks; model registry for deployment promotion; no infrastructure overhead | Logging too frequently in tight loops can add I/O overhead; use `mlflow.autolog()` for common frameworks | `import mlflow`
`mlflow.autolog()`
`with mlflow.start_run(): model.fit(X_train, y_train)` | [62] | | **DBSQL Caching (Result Cache)** | Databricks SQL warehouses **cache query results and intermediate scan results**; identical or similar queries **reuse cached data** without re-reading from storage. Automatic and transparent. | BI tools (Tableau, Power BI) running repeated queries; dashboards | **Orders-of-magnitude speedup** for repeated identical queries; reduces cloud storage I/O | Cache invalidated when underlying Delta table is modified; cache is per-warehouse; can be disabled if stale results are unacceptable | Automatic; monitor cache hit rates in Databricks SQL query history | [59] | --- ## Table 13 — Structured Streaming Optimizations | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Micro-Batch Trigger Interval Tuning** | Set `trigger(processingTime="N seconds")` to control how frequently Spark **polls for new data and processes batches**. Too short → high overhead; too long → increased latency. Balance latency vs. throughput needs. | Kafka-to-Delta streaming pipelines; event processing with latency SLAs | Tuning trigger interval can **reduce cluster load 2–5×** vs. default (0 = as fast as possible) | Default trigger (ASAP) maximizes throughput at cost of overhead; for batch-like workloads, use `trigger(availableNow=True)` | `df.writeStream.trigger(processingTime="30 seconds").start()` | [63][64] | | **`trigger(availableNow=True)` (Spark 3.3+)** | Processes **all available data** as of trigger time in one or more micro-batches, then stops the stream — **hybrid between streaming and batch**; replaces the deprecated `trigger(once=True)`. | Incremental batch processing patterns; cost-conscious pipelines | Processes all backlog efficiently; **releases cluster resources immediately after completion**; Databricks Workflows-friendly | Not truly continuous streaming; introduces latency equal to job schedule interval; newer API (3.3+) | `df.writeStream.trigger(availableNow=True).start().awaitTermination()` | [63][64] | | **Watermarking for State Management** | Define a **watermark** on an event-time column (`withWatermark("event_time","10 minutes")`) to tell Spark to **drop state** for events older than the watermark — **prevents unbounded state growth**. | Windowed aggregations (tumbling/sliding), stream-stream joins with time conditions | **Bounds state store memory**; enables late data handling without infinite retention | Watermark setting is a latency-correctness trade-off; too tight → missed late events; too loose → unbounded state; always required for stateful streaming in production | `df.withWatermark("ts","1 hour").groupBy(window("ts","15 min")).agg(count("*"))` | [63][64][65] | | **State Store Backend Tuning** | Default state store is **HDFS/Rocks-DB backed** (depending on Spark version/Databricks Runtime); Databricks Runtime offers **RocksDB-based state store** for lower memory footprint and faster state access at scale. | Stateful streaming (joins, aggregations, arbitrary stateful processing) | **RocksDB state store scales to billions of state rows** without OOM; incremental checkpointing reduces checkpoint time | Requires enabling `spark.sql.streaming.stateStore.providerClass=org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider` | `spark.conf.set("spark.sql.streaming.stateStore.providerClass","...RocksDBStateStoreProvider")` | [63][65] | | **Kafka Source Optimization** | Tune Kafka consumer parallelism via **minPartitions, maxOffsetsPerTrigger, and fetchOffset.numRetries**; use `startingOffsets="latest"` for low-latency production; set `kafka.fetch.min.bytes` for batching efficiency. | Kafka-to-Delta/Parquet streaming ingestion; real-time event processing | Prevents overloading Kafka brokers; allows controlled throughput; prevents micro-batches from being too large or too small | `maxOffsetsPerTrigger` limits throughput — set high enough to avoid falling behind; too many Spark partitions from small Kafka partitions → overhead | `df = spark.readStream.format("kafka").option("maxOffsetsPerTrigger","100000").load()` | [63] | | **Checkpointing for Fault Tolerance (Streaming)** | Every streaming query must have a **checkpoint location** (`checkpointLocation`) in cloud storage to **persist query state, offsets, and committed batch IDs** for **exactly-once semantics** and restart recovery. | All production streaming jobs | Enables **exactly-once end-to-end guarantees**; fast recovery from failure without reprocessing from beginning | Checkpoint location is tied to query identity — changing schema or logic may require a new checkpoint; checkpoint I/O adds latency to micro-batches | `df.writeStream.option("checkpointLocation","dbfs:/checkpoints/stream1/").start()` | [63][64] | | **Continuous Processing Mode (Spark 3.3+)** | An alternative to micro-batch: processes records **continuously with ~1ms latency** (vs. seconds for micro-batch); uses a fixed number of long-running tasks. Only supports simple non-stateful transformations. | Ultra-low-latency stateless streaming (filter, project, map) | **Sub-millisecond latency**; low overhead for simple pipelines | Very limited operation support (no aggregations, no joins currently); experimental feature; not yet production-ready for complex workloads | `df.writeStream.trigger(continuous="1 second").start() # Spark 3.3+ continuous mode` | [64] | --- ## Table 14 — Monitoring, Profiling & Performance Diagnostics | Technique / Approach | Description | Main Usages | Advantages | Considerations | Example | Ref | |---|---|---|---|---|---|---| | **Spark UI (Web UI / History Server)** | Spark's built-in web interface exposes **Jobs, Stages, Tasks, Storage, Executors, and SQL tabs** — the primary tool for diagnosing benchmarks, **skew, spills, and inefficient plans**. | Every optimization investigation; post-job analysis via History Server | Free, built-in, comprehensive; **physical execution plan visualization**; per-task metrics including shuffle read/write, spill, and GC time | Requires knowing what to look for; UI disappears after cluster termination unless History Server is configured | Check: long stages (bottlenecks), task duration distribution (skew), GC time %, shuffle spill volume | [66][67] | | **Explain Plans (`explain()`)** | Call `df.explain(True)` or `EXPLAIN EXTENDED` in SQL to print **all four execution plan stages** (Parsed, Analyzed, Optimized, Physical); inspect for **missing pushdowns, unexpected shuffles, or wrong join strategies**. | Pre-run query tuning; verifying predicate pushdown; confirming AQE / DPP activation | Zero overhead; **instant insight**; essential step before submitting large jobs | Reading execution plans requires practice; physical plan notation (`*`, `+`, `Exchange`) takes time to learn | `df.explain("formatted") # Most readable format in Spark 3.0+` | [1][66] | | **Event Listener / SparkMeasure** | Open-source library (`spark-measure`) that instruments Spark jobs with **fine-grained stage and task metrics** (CPU time, JVM GC time, shuffle bytes, task deserialization time) — easily added to any job. | Detailed performance profiling; CI/CD performance regression testing | Much richer than Spark UI alone; programmatic access to metrics; can be used to **automate performance assertions** | Adds slight overhead during metric collection; requires adding JAR to cluster; learning curve for interpreting metrics | `%pip install sparkmeasure`
`from sparkmeasure import StageMetrics`
`sm = StageMetrics(spark)`
`sm.runandmeasure(globals(), "df.count()")` | [66][67] | | **Databricks Query History & Query Profiler** | Databricks SQL records **all query executions** with duration, rows processed, cluster used, and a **visual query profile** — shows operator-level breakdown of time (scan, filter, join, aggregate). | SQL warehouse optimization; identifying slow queries in BI workloads; multi-user concurrency analysis | Persistent history (unlike ephemeral Spark UI); visual operator tree; **latency percentile analysis** | Only available for SQL warehouse queries; job compute queries use Spark UI / History Server instead | Access via Databricks SQL → Query History → click any query → View Profile | [56][59] | | **Ganglia / Prometheus + Grafana** | Cluster-level infrastructure metrics: **CPU utilization, network I/O, memory pressure, disk I/O** per node over time — complements Spark UI's task-level view with **system-level view**. | Identifying infrastructure bottlenecks (network saturation, CPU throttling, disk IOPS limits) | Reveals **I/O-bound vs CPU-bound vs memory-bound bottleneck**; historical trends; alerting | Requires setup/configuration; Ganglia is legacy (Databricks uses it internally); modern deployments use Prometheus/Grafana or Datadog | Databricks exposes Ganglia at `https:///ganglia/` during job run | [66][67] | | **Spark DAG Visualization** | Spark UI's **graphical DAG display** per stage shows which tasks ran in parallel, where barriers are (shuffle stages), and which stage is the **critical path**. | Identifying unnecessary sequential dependencies; verifying stage-level parallelism | **Intuitive visual representation** of execution plan; quick identification of bottlenecks in multi-stage jobs | Only available during active job or in History Server; reading DAG requires familiarity with Spark stages/tasks/barriers | Access via Spark UI Jobs tab → click Job → click Stage → DAG Visualization | [66] | --- ## References ### Official Documentation 1. [Apache Spark SQL Performance Tuning (Official Docs)](https://spark.apache.org/docs/latest/sql-performance-tuning.html) 2. [Apache Spark RDD Programming Guide](https://spark.apache.org/docs/latest/rdd-programming-guide.html) 3. [Cost-Based Optimizer — Azure Databricks Microsoft Learn](https://learn.microsoft.com/en-us/azure/databricks/optimizations/cbo) 4. [Cost-Based Optimizer in Apache Spark 2.2 — Databricks Blog](https://www.databricks.com/blog/2017/08/31/cost-based-optimizer-in-apache-spark-2-2.html) 5. [Deep Dive into Spark Memory Management — luminousmen](https://luminousmen.com/post/dive-into-spark-memory/) 6. [Spark Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html) 7. [Databricks Best Practices for Performance Efficiency (AWS)](https://docs.databricks.com/aws/en/lakehouse-architecture/performance-efficiency/best-practices) 8. [Spark DataFrame Cache and Persist Explained — Spark By Examples](https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/) 9. [Spark Persistence Storage Levels — Spark By Examples](https://sparkbyexamples.com/spark/spark-persistence-storage-levels/) 10. [Spark Caching Explained: What Really Happens Under the Hood — luminousmen](https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching/) 11. [Memory Management in Apache Spark — Medium](https://somnath-dutta.medium.com/memory-management-in-apache-spark-optimizing-performance-for-big-data-processing-c827e5aef82c) 12. [Databricks Cluster Configuration Best Practices — dateonic](https://dateonic.com/databricks-cluster-configuration-best-practices/) 13. [Decoding Memory in Spark — Walmart Global Tech Blog](https://medium.com/walmartglobaltech/decoding-memory-in-spark-parameters-that-are-often-confused-c11be7488a24) 14. [Concept of Checkpoints in Spark — NashTech Blog](https://blog.nashtechglobal.com/concept-of-checkpoints-in-spark/) 15. [RDD Checkpointing — The Internals of Apache Spark Core](https://books.japila.pl/apache-spark-internals/rdd/checkpointing/) 16. [Spark Tips: Partition Tuning — luminousmen](https://luminousmen.com/post/spark-tips-partition-tuning/) 17. [Adaptive Query Execution — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/optimizations/aqe) 18. [Data Partitioning and Distribution Strategies in Spark — Reintech](https://reintech.io/blog/data-partitioning-distribution-strategies-spark) 19. [Dynamic Partition Pruning — Spark 3.x Medium](https://medium.com/geekculture/dynamic-partition-pruning-baf3270694b4) 20. [Pruning Dynamic Partitions — AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/spark-tuning-glue-emr/pruning-dynamic-partitions.html) 21. [Mastering Data Partitioning for Efficient Scaling in Spark — moldstud](https://moldstud.com/articles/p-mastering-data-partitioning-for-efficient-scaling-in-spark-yourwebsite) 22. [Bucketing (Reddit r/dataengineering discussion)](https://www.reddit.com/r/dataengineering/comments/1d2d1xz/real_time_question/) 23. [Spark Join Strategies Explained — Canadian Data Guy](https://www.canadiandataguy.com/p/spark-join-strategies-explained-broadcast) 24. [Using Join Hints in Spark SQL — AWS Prescriptive Guidance](https://docs.aws.amazon.com/prescriptive-guidance/latest/spark-tuning-glue-emr/using-join-hints-in-spark-sql.html) 25. [Adaptive Query Execution: Speeding Up Spark SQL at Runtime — Databricks Blog](https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html) 26. [Performance Tuning — Spark 4.1.1 Docs](https://spark.apache.org/docs/latest/sql-performance-tuning.html) 27. [Spark Release 3.3.0 — Runtime Bloom Filters](https://spark.apache.org/releases/spark-release-3-3-0.html) 28. [What's New in Apache Spark 3.3 — Joins & Runtime Filters](https://www.waitingforcode.com/apache-spark-sql/what-new-apache-spark-3.3-joins/read) 29. [SPARK Shuffle Tuning — blog.imaginea.com](https://blog.imaginea.com/spark-shuffle-tuning/) 30. [Spark Storage Levels: Optimize Data Persistence — sparkcodehub.com](https://www.sparkcodehub.com/spark-storage-levels) 31. [Delta Lake Optimizations (Official Delta Lake Docs)](https://docs.delta.io/latest/optimizations-oss.html) 32. [Optimize Data File Layout — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/delta/optimize) 33. [Optimizing Delta Tables: OPTIMIZE and Z-ORDER — Conduktor](https://www.conduktor.io/glossary/optimizing-delta-tables-optimize-and-z-order) 34. [Use Liquid Clustering for Tables — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/delta/clustering) 35. [Delta Lake Data Skipping with Z-Order and Bloom Filter — Stack Overflow](https://stackoverflow.com/questions/74721211/delta-lake-data-skipping-with-z-order-and-bloom-filter-index) 36. [Control Data File Size (Auto Compaction) — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/delta/tune-file-size) 37. [Predictive Optimization Automatically Delivers Faster Queries — Databricks Blog](https://www.databricks.com/blog/predictive-optimization-automatically-delivers-faster-queries-and-lower-tco) 38. [Big Data File Formats Explained — Towards Data Science](https://towardsdatascience.com/big-data-file-formats-explained-275876dc1fc9/) 39. [The Data Engineer's Guide to File Formats — Medium Towards Data Engineering](https://medium.com/towards-data-engineering/the-data-engineers-guide-to-file-formats-parquet-vs-orc-vs-avro-470e1d7f7643) 40. [Avro vs. Parquet: A Complete Comparison — DataCamp](https://www.datacamp.com/blog/avro-vs-parquet) 41. [Why Your PySpark UDF Is Slowing Everything Down — Canadian Data Guy](https://www.canadiandataguy.com/p/why-your-pyspark-udf-is-slowing-everything) 42. [Spark Off-Heap Memory Config and Tungsten — Stack Overflow](https://stackoverflow.com/questions/43330902/spark-off-heap-memory-config-and-tungsten) 43. [How to Use PySpark UDFs and Pandas UDFs Effectively — DataCamp](https://www.datacamp.com/tutorial/pyspark-udfs) 44. [Everything You Ever Wanted to Know about Pandas / PyArrow UDFs — Databricksters](https://www.databricksters.com/p/everything-you-ever-wanted-to-know) 45. [Introducing Pandas UDF for PySpark — Databricks Blog](https://www.databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html) 46. [Arrow-Optimized Python UDFs in Apache Spark 3.5 — Databricks Blog](https://www.databricks.com/blog/arrow-optimized-python-udfs-apache-sparktm-35) 47. [Chapter 5: UDFs & UDTFs — PySpark 4.1.1 Documentation](https://spark.apache.org/docs/latest/api/python/user_guide/udfandudtf.html) 48. [Why Predicate and Projection Pushdown Matter in PySpark — Medium](https://medium.com/@anowerhossain97/why-predicate-and-projection-pushdown-matter-in-pyspark-dcd5dd241c64) 49. [PySpark Partition Pruning, Predicate, and Projection Pushdown — Medium](https://medium.com/@davisjustin42/pyspark-partition-pruning-predicate-and-projection-pushdown-a939c1b49c10) 50. [Spark Tips: Optimizing JDBC Data Source Reads — luminousmen](https://luminousmen.com/post/spark-tips-optimizing-jdbc-data-source-reads/) 51. [Increasing Apache Spark Read Performance for JDBC — Mercedes-Benz Tech Innovation](https://medium.com/mercedes-benz-techinnovation-blog/increasing-apache-spark-read-performance-for-jdbc-connections-a028115e20cd) 52. [An Introduction to Higher-Order Functions in Spark SQL — SlideShare](https://www.slideshare.net/slideshow/an-introduction-to-higher-order-functions-in-spark-sql-with-herman-van-hovell/119151768) 53. [Optimization Recommendations on Databricks (AWS)](https://docs.databricks.com/aws/en/optimizations/) 54. [Best Practices for Cost Optimization — Databricks Docs (AWS)](https://docs.databricks.com/lakehouse-architecture/cost-optimization/best-practices.html) 55. [Best Practices for Performance Efficiency — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/lakehouse-architecture/performance-efficiency/best-practices) 56. [Databricks Performance Optimization (Training Course Overview)](https://www.databricks.com/training/catalog/databricks-performance-optimization-1809) 57. [What is Photon? — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/compute/photon) 58. [Photon: A Fast Query Engine for Lakehouse Systems — SIGMOD 2022 Paper](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf) 59. [SQL Warehouse Types — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-types) 60. [Getting Started with Delta Live Tables — Databricks](https://www.databricks.com/discover/pages/getting-started-with-delta-live-tables) 61. [What is Unity Catalog? — Databricks Docs (AWS)](https://docs.databricks.com/aws/en/data-governance/unity-catalog/) 62. [MLOps Gym: Harnessing the Power of Apache Spark in ML Workflows — Databricks Community](https://community.databricks.com/t5/technical-blog/mlops-gym-harnessing-the-power-of-apache-spark-in-data-science/ba-p/68712) 63. [Structured Streaming Programming Guide — Spark 3.5](https://spark.apache.org/docs/3.5.1/structured-streaming-programming-guide.html) 64. [Feature Deep Dive: Watermarking in Apache Spark Structured Streaming — Databricks Blog](https://www.databricks.com/blog/feature-deep-dive-watermarking-apache-spark-structured-streaming) 65. [Structured Streaming: A Declarative API for Real-Time — SIGMOD 2018 Paper](https://people.eecs.berkeley.edu/~matei/papers/2018/sigmod_structured_streaming.pdf) 66. [A Deep Dive into Spark UI for Job Optimization — Microsoft Tech Community](https://techcommunity.microsoft.com/blog/microsoftmissioncriticalblog/a-deep-dive-into-spark-ui-for-job-optimization/4442229) 67. [Monitor Performance in Apache Spark — tsmatz.wordpress.com](https://tsmatz.wordpress.com/2019/09/19/sparkmeasure-metrics-in-apache-spark/) --- ### Key Technical Blog Posts & Guides - [Spark Performance Optimization in Databricks — A Complete Guide (Medium/kaviprakash)](https://medium.com/@kaviprakash.2007/spark-performance-optimization-in-databricks-a-complete-guide-ab57280a8260) - [Comprehensive Guide to Optimize Data Workloads — Databricks](https://www.databricks.com/discover/pages/optimize-data-workloads-guide) - [5 Critical Databricks Performance Hacks Most Engineers Miss — DZone](https://dzone.com/articles/databricks-performance-hacks-faster-queries) - [Delta Lake Optimisation Guide — LinkedIn Pulse](https://www.linkedin.com/pulse/delta-lake-optimisation-guide-deenar-toraskar) - [Boosting Spark and Delta Lake Performance on Databricks — Medium](https://medium.com/@arvindpant/boosting-spark-and-delta-lake-performance-on-databricks-a-practical-guide-d14e5d2dafd8) - [Apache Spark Performance Tuning: 7 Optimization Tips — Chaos Genius](https://www.chaosgenius.io/blog/spark-performance-tuning/) - [How to Optimize Databricks Performance: Complete Query Tuning Guide 2025 — e6data](https://www.e6data.com/query-and-cost-optimization-hub/databricks-performance-optimization-complete-query-tuning-guide-2025) --- ### Academic Papers - [Photon: A Fast Query Engine for Lakehouse Systems — SIGMOD 2022](https://people.eecs.berkeley.edu/~matei/papers/2022/sigmod_photon.pdf) - [Structured Streaming: A Declarative API for Real-Time Applications — SIGMOD 2018](https://people.eecs.berkeley.edu/~matei/papers/2018/sigmod_structured_streaming.pdf) - [Predicate Pushdown in Parquet and Apache Spark — CWI Master Thesis](https://homepages.cwi.nl/~boncz/msc/2018-BoudewijnBraams.pdf) - [A Comparative Analysis of AVRO, Parquet, and ORC Formats — JETA](https://www.espjeta.org/Volume4-Issue3/JETA-V4I3P117.pdf) --- ### Popular YouTube Videos (Most Viewed & Rated) - **Apache Spark Core – Practical Optimization** by Daniel Tomes (Databricks) — referenced in multiple top posts as the go-to video for hands-on tuning - **PySpark - Zero to Hero | PySpark Tutorial 2024** — high-view introductory series - **Apache Spark Architecture 101: How Spark Works** — foundational concepts, widely referenced - Search: *"Databricks Performance Optimization"* on YouTube — filter by "Most Viewed" on the Databricks official channel for updated talks from Data + AI Summit - Search: *"Tech Talk: Top Tuning Tips for Spark 3.0 and Delta Lake"* — Databricks Tech Talk series, highly referenced in community posts [35]

Spark / PySpark & Databricks Optimization Techniques

Spark / PySpark & Databricks Optimization Techniques

Spark / PySpark & Databricks Optimization Techniques

Spark / PySpark & Databricks Optimization Techniques