Spark performance:
a field guide.

The fastest way to waste an afternoon is to raise executor.memory on a job that's actually skewed. Open the Spark UI, find the slow stage, and look at two numbers: the number of tasks and the max task duration vs. the median.

Aim for partitions of roughly 128–256 MB. Too few and you can't parallelise; too many and the scheduler drowns in overhead. Repartition on a high-cardinality key before any wide join.

# too few partitions = no parallelism; too many = scheduler overhead
df = df.repartition(400, "user_id")

# check the spread
df.rdd.glom().map(len).collect()

Skew shows up as one or two tasks that never finish. Confirm it by counting rows per key — if a handful of keys dominate, that's your problem.

# which keys are hot?
(df.groupBy("user_id")
   .count()
   .orderBy(col("count").desc())
   .show(20))

For the join itself, enable adaptive query execution — it splits skewed partitions automatically on recent Spark versions.

spark.conf.set("spark.sql.adaptive.enabled", True)
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True)

When AQE isn't enough, salt manually: spread each hot key across N buckets, join on the salted key, then drop the salt. It trades a little shuffle for a lot of balance.

# spread hot keys across 16 buckets
df = df.withColumn("salt", (rand() * 16).cast("int"))
keyed = df.withColumn("jkey",
    concat_ws("_", col("user_id"), col("salt")))

Only after partitioning is sane should you touch memory — and raise overhead before heap. Most surprise kills are off-heap: shuffle buffers and Python workers, not the JVM heap.

--conf spark.executor.memory=8g
--conf spark.executor.memoryOverhead=2g   # raise this first
--conf spark.sql.shuffle.partitions=400

Before escalating to "we need a bigger cluster," run this in order:

# 1. partitions sane?     ~128–256MB each
# 2. skew?                max/median task time
# 3. AQE on?              adaptive + skewJoin
# 4. salt hot keys        if AQE not enough
# 5. overhead before heap memoryOverhead first
# 6. THEN scale the cluster

Nine times out of ten you never reach step six. The cluster was big enough; the work just wasn't balanced.