Video summary

PySpark Tutorial | Full Course (From Zero to Pro!)

Main summary

Key takeaways

Educational

Main ideas & lessons from the video (PySpark Master Class)

1) Course promise / goal

The video positions itself as a “zero to pro” PySpark course for beginners, aiming to help viewers:

  • Learn Spark concepts + PySpark (Python) API in depth
  • Practice reading data from multiple file formats (CSV, JSON)
  • Learn common interview-relevant scenarios
  • Build mastery through many transformations and utilities

2) Prerequisites / Spark fundamentals

What is Spark?

Spark is presented as a distributed computing engine that:

  • Splits data across multiple machines in a cluster
  • Processes data in parallel for scalability (contrasted with a single machine’s resource limits)

Spark architecture (interview-important)

The architecture is explained as a master/driver + workers model:

  • A cluster manager manages resources
  • A driver program/node:
    • Receives the submitted code
    • Breaks it into transformation stages, jobs, and tasks
    • Requests worker nodes from the cluster manager
  • Worker nodes:
    • Execute the transformations and processing

Why Spark benefits vs Hadoop (as stated)

Key benefits mentioned:

  • In-memory computation (faster than Hadoop’s disk-based behavior)
  • Lazy evaluation (transformations are not executed immediately)
  • Fault tolerance
  • Partitioning (distributing data across machines)

Lazy evaluation is also highlighted as a core interview topic.

Lazy evaluation (core concept)

  • Transformations build a logical plan
  • Execution happens only when an action is triggered

Examples of actions referenced:

  • show, display, collect (and similar notebook concepts)

Jobs, stages, tasks hierarchy

A submitted workload becomes:

  • Job
    • multiple stages
      • multiple tasks

This is used to explain how Spark executes work internally.

3) DataBricks setup (workflow for free account)

Steps described to create a DataBricks Community Edition account:

  • Go to Google → “DataBricks Community sign up”
  • Choose Try DataBricks / Community Edition
  • Complete verification and use the email to activate

Workspace navigation explained:

  • Workspace: folders for notebooks
  • Recent, Search
  • Catalog: upload/store datasets
  • Workflows: orchestrate notebooks (or related tasks)
  • Compute: create a free cluster/compute

4) Building your first notebook in DataBricks

  • Create a notebook inside a folder
  • Create a cluster (Community Edition) and attach it to the notebook
  • Use notebook UI features:
    • Markdown headings (%md, hashtags like ###)
    • Code cells
    • Run cells with Shift+Enter (sometimes Alt+Enter to create the next cell)

PySpark methodology & code concepts covered

A) Reading data (DataFrameReader API)

CSV reading (example approach)

Typically uses:

  • spark.read.format("csv")
  • option("inferSchema", "true")
  • option("header", "true")
  • .load(<path>)

Data path retrieval method shown:

  • Use dbutils.fs.ls(...) to list files under a container/folder and construct the final file path

JSON reading (example approach)

Typically uses:

  • spark.read.format("json")
  • option("inferSchema", "true")
  • option("header", "true")
  • option("multiLine", <true/false>) depending on JSON structure
  • .load(<path>)

Emphasis:

  • Correctly setting single-line vs multi-line JSON via multiLine is important.

B) Schema management (high interview relevance)

Why schema matters

  • Default schema inference can be overridden to enforce types
  • The example requirement includes converting an inferred numeric column (e.g., “double”) into a string

Two schema definition methods

  1. DDL schema (string-based)

    • Define schema as SQL-like type declarations in a multi-line string
    • Pass into reader using: .schema(<ddlSchemaVariable>)

  2. StructType / StructField (programmatic)

    • Import from:
      • pyspark.sql.functions
      • pyspark.sql.types
    • Build schema as a list of StructField(name, dataType, nullable) and pass into .schema(...)

C) Core transformations taught

Select (column projection)

  • Purpose: keep only specific columns
  • Methods shown:
    • df.select("col1", "col2", ...)
    • df.select(col("col1"), ...) (column object approach)

Alias (rename columns)

  • Example pattern:
    • df.select(col("oldName").alias("newName"))

Filter / Where (row filtering)

  • Simple condition: col("itemFatContent") == "low fat" (conceptual)
  • Combined conditions:
    • AND: (condition1) & (condition2)
    • OR and null handling (conceptual example):
      • col("outletSize").isNull() & col("outletLocationType").isin("Tier 1","Tier 2")

Emphasis:

  • Use correct boolean logic operators and .isNull().

WithColumnRenamed

  • Rename at the DataFrame level:
    • df.withColumnRenamed("old", "new")

WithColumn (create/modify columns)

  • Create a new column:
    • df.withColumn("flag", lit("value")) (uses lit for constants)
  • Create computed columns:
    • df.withColumn("multiply", col("a") * col("b"))
  • Modify values in an existing column:
    • Example concept: replace string values using regex replace

Type casting

  • Example concept:
    • df = df.withColumn("item_weight", col("item_weight").cast("string"))

Sorting (orderBy / sort)

  • Sort by one column:
    • descending: .desc() (or equivalent)
    • ascending: .asc() or default
  • Sort by multiple columns:
    • use lists of columns + matching direction flags

Limit

  • Example:
    • df.limit(10)

Drop

  • Drop one column:
    • df.drop("colName")
  • Drop multiple columns:
    • df.drop("col1","col2")

Drop duplicates & Distinct

  • Remove duplicate rows:
    • df.dropDuplicates()
  • Also mentioned:
    • df.distinct()
  • Subset-based duplicates:
    • df.dropDuplicates(subset=["colA", ...])

Union and Union by name

  • df1.union(df2):
    • concatenates rows positionally; can mismatch if column order differs
  • df1.unionByName(df2):
    • aligns columns by name to prevent “mixed columns” issues

Function topics taught beyond basic transformations

1) String functions

  • initcap (proper case)
  • lower
  • upper
  • alias used to name derived columns

2) Date functions

  • current_date
  • date_add
  • date_sub explained, then workaround using date_add(..., -N)
  • datediff (difference between start and end columns)
  • date_format (including token case sensitivity, e.g., y vs Y)

3) Handling nulls

Dropping nulls

  • dropna() options:
    • how="any": drop if any column in the row is null
    • how="all": drop only if all columns are null
  • Prefer subset-based drops to avoid losing too much data:
    • df.dropna(subset=["colName"])

Filling nulls

  • fillna(value) to replace nulls across all columns
  • Use subset to fill only selected columns:
    • df.fillna(value, subset=["colName"])

4) Split, Indexing, Explode (array/list operations)

  • split(col, delimiter): converts string to an array/list
  • Indexing:
    • arrays start at index 0
  • explode(arrayCol):
    • turns array elements into multiple rows

5) array_contains

  • Checks whether a value exists in an array column
  • Used to create boolean/flag columns for membership

Aggregations & reshaping

GroupBy (aggregation)

  • GroupBy aggregates values per key(s)
    • Example: sum MRP per item type
    • Example: average MRP per item type
  • Multi-key groupBy:
    • group by item_type and outlet_size
  • Multiple aggregations:
    • compute both sum and avg in one grouped result

collect_list (groupBy alternative to group_concat)

  • Collects grouped values into a list:
    • groupBy("user").agg(collect_list("book"))

Pivot

  • Produces a matrix-like result (similar to Excel pivot tables)
  • Structure:
    • Rows: one key (e.g., item type)
    • Columns: pivot key (e.g., outlet size)
    • Values: aggregated metric (sum/avg/etc.)

Conditional logic

when / otherwise (case-when equivalent)

  • Create conditional flag columns:
    • if condition then X else Y
  • Advanced scenario:
    • chain multiple when clauses:
      • when(cond1).then(val1)
      • when(cond2).then(val2)
      • otherwise(valOther)

Joins (core relational operations)

The video explains join behavior conceptually and provides PySpark-style code patterns:

  • Inner join
    • Output only keys present in both tables
  • Left join
    • Keep all rows from left table; unmatched right columns become null
  • Right join
    • Keep all rows from right table; unmatched left columns become null
  • Anti join
    • Output rows from left table that do NOT match right table on key

Special note:

  • Anti join is described as a “special” join compared to SQL lacking a direct keyword.

Implementation detail mentioned:

  • When join keys share the same name across datasets:
    • use qualified column references (e.g., df1["Department ID"] == df2["Department ID"]) to avoid ambiguity.

Window functions (advanced analytics patterns)

Concepts taught

Window functions enable row-level calculations without collapsing rows like GroupBy.

Covered:

  • row_number
  • rank
  • dense_rank
  • cumulative sum

Also covered:

  • over(...) with partitionBy/orderBy logic
  • frame clauses for cumulative sum

Rank vs dense_rank differences

  • rank:
    • gaps appear after ties
  • dense_rank:
    • no gaps after ties

Cumulative sum via window + frame clause

  • Example pattern:
    • sum(col).over(Window.orderBy(...).rowsBetween(...))
  • Need a frame definition like:
    • from “unbounded preceding” to “current row”
  • Mentioned:
    • “unbounded following” can produce totals across future rows too

User Defined Functions (UDF)

  • Purpose:
    • use Python logic when built-in Spark functions can’t handle the case (or it’s too complex)
  • Warning:
    • UDFs can hurt performance because Python needs to be interpreted and execution patterns involve JVM-side conversion

UDF workflow shown:

  • write a normal Python function
  • register it as a Spark UDF
  • apply using withColumn

Data writing & storage formats

Data writing layer + storage concepts

  • Pipeline stages: 1) Read data 2) Transform 3) Write (serving layer for downstream consumers)
  • Storage location differs by environment:
    • Azure: ADLS Gen2 / Data Lake
    • AWS: S3
    • Community demo: DataBricks default storage (file store)

File formats demonstrated

  • CSV writing via DataFrameWriter
  • Parquet (columnar)
  • Delta Lake:
    • built on Parquet
    • uses transaction log for metadata/history

Write modes (critical methodology)

Common .mode("<mode>") values:

  • append
    • add new data files without removing existing data
  • overwrite
    • replace/truncate existing data at the destination
  • error / errorifexists
    • fail if destination already has data
  • ignore
    • do nothing if destination already has data

Creating tables

  • Mentioned “save as table” style to create a managed table

Managed vs external tables

  • Managed table
    • Data lifecycle managed by DataBricks
    • Dropping the table can remove underlying data
  • External table
    • Data stored in a user-controlled location
    • Dropping the table removes schema but not underlying data
  • Guidance:
    • Prefer external when data is critical and should not be deleted automatically

Spark SQL integration

  • Convert DataFrame → create temporary view:
    • df.createOrReplaceTempView("my_view")
  • Run SQL using notebook SQL magic (e.g., %sql)
  • Convert SQL result back to DataFrame:
    • spark.sql("...")

Speakers / sources featured

  • Primary speaker/host: The course instructor (self-identified as an Azure / “AOR” data engineer)
    • Certifications mentioned include:
      • Microsoft Certified: Azure Data Engineer Associate
      • Spark Developer Associate
      • and others
  • No other named speakers or external sources are featured in the subtitles beyond:
    • references to commonly used tools/docs (DataBricks UI, PySpark APIs, SQL equivalents) and general concepts

Original video