Video summary

Docker for Data Engineering: Postgres, Docker Compose, and Real-World Workflows - Alexey Grigorev

Main summary

Key takeaways

Technology

What the video workshop covers (Module 1 update)

  • Introduces an updated Data Engineering course module focused on Docker (with a brief mention of Terraform, though the walkthrough is primarily Docker-related).
  • Goal: provide an end-to-end working version of the module using:
    • updated Postgres
    • newer workflow steps that address common issues from older course runs.
  • Workshop materials are linked externally, and the course is scheduled to start in January.

Setup / prerequisites for the workshop

Requirements

  • Docker installed
  • Python installed

Strong recommendation: GitHub Codespaces

Use GitHub Codespaces because it already provides:

  • a remote environment
  • Python + Docker
  • avoids local setup issues, especially on Windows

Workshop workflow

  1. Create a GitHub repository (public)
  2. Add a README and .gitignore (Python-focused)
  3. Open Codespaces via the GitHub UI
  4. Use Visual Studio Code Desktop with the Codespaces extension (browser is possible but less convenient)

Docker 101 concepts demonstrated

Containers are isolated and (mostly) stateless

  • Docker is explained as running software in isolated environments (containers) separate from the host.
  • Demonstrations:
    • docker run hello-world to verify Docker works
    • docker run ubuntu downloads an image and runs commands inside the container
    • installing packages inside a container does not affect the host
  • Key point: containers created from images are stateless—exiting and re-entering typically returns to the image’s original state.

Docker images vs containers

  • Docker image: a snapshot of an OS/environment
  • Container: a running instance created from that image

Entry points and command behavior

  • Demonstrates overriding container entry behavior:
    • Example: run a Python image but switch to a bash entry point to get a shell

Cleaning up unused containers

  • Shows listing and removing exited containers:
    • docker ps -a
    • docker ps -aq and docker rm ... to remove exited containers and keep the environment clean

Persisting data from host ↔ container

Volumes (host files usable inside container)

  • Shows bind mounts (-v) to map a host directory into a container directory.
  • Example pipeline support:
    • create a small script on the host
    • map test/ into the container so the script/files are visible inside Docker
  • Without volumes: created files disappear after container re-creation.

Pipeline concept: CSV/processing → Postgres

  • Defines a data pipeline as:
    • taking input (e.g., a CSV dataset)
    • producing output (e.g., loading into Postgres)
  • Today’s ingestion target: NYC Taxi dataset
  • Approach:
    • start with a simple parameterized Python pipeline using CLI arguments
    • later evolve it into a Dockerized ingestion workflow

Parameterized Python pipeline + pandas + parquet

CLI parameterization

  • Uses command-line arguments (sys.argv / parsing concept shown) so the pipeline can process specific partitions (e.g., month number).

Data processing with pandas

  • Demonstrates transformations (e.g., adding a month column).

Saving to parquet

  • Explains parquet as a binary format optimized compared to CSV.
  • Shows installing missing dependencies (e.g., pyarrow) to write parquet.

Dependency isolation: virtual environments + UV

Why UV is introduced

  • Avoid installing dependencies into the host Python environment.
  • Uses virtual environments managed by UV (UV is described as extremely fast, written in Rust, and faster than alternatives like Conda/mamba for this context).

UV workflow shown

  • Create a project environment with a chosen Python version (example uses a Python version different from system Python).
  • Install dependencies into that environment:
    • uv add pandas
    • uv add pyarrow
  • Configure the editor to use the correct interpreter (VS Code interpreter switching concept).
  • Reproducibility:
    • uses uv.lock
    • later in Docker builds runs uv sync --locked to ensure exact dependency versions

Dockerizing the pipeline (custom Dockerfile)

Build process

Introduces a Dockerfile that:

  • starts from a base Python image
  • installs required packages (pandas/pyarrow)
  • copies pipeline source code into the image
  • sets execution behavior using ENTRYPOINT or a docker run command override

Efficient rebuild behavior

  • Emphasizes Docker layer caching:
    • rebuilding may skip repeated steps if Dockerfile inputs haven’t changed

Improving the Docker build with UV inside the container

  • Shows building in a UV-managed environment by:
    • copying pyproject.toml, uv.lock, and the UV-created Python
    • running uv sync --locked
  • Advantage:
    • reproducible dependencies + faster installs
    • avoids pip installing into the container without locking

Running Postgres with Docker (and persistent storage)

Postgres container setup

  • Uses docker run to start a specific Postgres version (shown as Postgres 18).
  • Configured via environment variables:
    • user/password/database (example uses root/root and a database name like ny_taxi)
  • Uses:
    • volumes to persist database state across restarts
    • port mapping to access Postgres from the host

Access from outside the container

  • Uses pgcli (installed as a dev dependency) to connect via the host mapping.
  • Demonstrates:
    • creating a table
    • inserting/selecting rows
    • stopping and restarting Postgres while data remains (because of volume persistence)

Ingestion workflow via Jupyter → script

Interactive exploration with Jupyter Notebook

  • Installs jupyter as a dev dependency.
  • Starts Jupyter in the project environment from within Docker.
  • Demonstrates loading the NYC Taxi dataset:
    • dataset is remote and compressed (.csv.gz)
    • loaded directly using pandas via URL
  • Notes:
    • dataset size is large (mentions 1,330,000+ records).

Handling pandas dtype issues (important analysis/detail)

  • pandas warns about mixed-type columns (example: vendor_id inferred as float due to missing values).
  • Fix described:
    • provide explicit dtype / schema guidance
    • parse pickup/dropoff timestamps as datetime
  • Framed as especially important because CSV is schemaless, while parquet includes schema.

Performance / reliability pattern: chunked loading

  • Uses pandas chunk iteration (chunksize, iterator mode).
  • Loads to Postgres incrementally (example chunk size: ~100,000 rows).
  • Uses tqdm for progress reporting.
  • Avoids “load everything at once” to reduce memory/time uncertainty and improve debuggability.

Converting notebook to production script

  • Converts notebook to a Python script using jupyter nbconvert.
  • Adds parameterization:
    • year, month
    • Postgres connection parameters (PG user, PG password, PG host, PG database, PG port)
    • chunk size
    • target table naming
  • Demonstrates table replacement logic using if_exists='replace' when writing to SQL.

CLI improvement using Click

  • Uses Click to generate a proper command-line interface.
  • Result: ingestion script runs like a standard CLI tool with options.

Orchestration: connecting containers correctly (Docker networking)

The “localhost” problem

  • When running ingestion container separately from Postgres:
    • inside ingestion container, localhost points to itself (not the Postgres container)

Fix: use Docker networks

  • Create a shared Docker network (example name: pg_network):
    • attach Postgres container to the network
    • attach ingestion container to the same network
    • use the Postgres container name (service hostname) as the PG host

UI: Postgres Admin in Docker

  • Runs pgAdmin in a separate container.
  • Uses port mapping so pgAdmin is accessible via browser.
  • Connects pgAdmin to Postgres using the shared network hostname.
  • Demonstrates checking inserted data via pgAdmin (table lists, row counts).

Docker Compose: final end-to-end workflow

Why Docker Compose is used

  • Replaces manual start/stop sequences with a single docker compose up.
  • A compose file (docker-compose.yml) is created (partly via an AI-assisted/editing step).
  • Compose services:
    • Postgres
    • pgAdmin
  • Notes:
    • Docker Compose provides a default internal network so services can reach each other by service name.
    • DB state depends on volumes:
      • restarting compose “from scratch” may clear database contents if volume persistence isn’t maintained as expected.

Running ingestion using the compose network

  • After starting compose, the ingestion script container is run and attached to the compose default network (example network name: pipeline_default).
  • Verifies results by querying row counts (database fills incrementally).

Key “review / guide / tutorial” style takeaways

  • Codespaces-first recommendation to avoid OS-specific Docker installation pain (Windows especially).
  • Docker basics:
    • isolation, images vs containers, statelessness
    • entrypoint and how containers behave on run
  • Data engineering practicality:
    • use explicit schemas when reading CSV with pandas (avoid dtype inference problems)
    • use chunked loading for large datasets
    • ensure reproducibility with UV + uv.lock and uv sync --locked
  • Production-style container connectivity:
    • prefer Docker networks over localhost for inter-container communication
  • Operational UX:
    • use pgAdmin as a convenient Postgres interface when running in Docker
  • Orchestration:
    • consolidate Postgres + pgAdmin startup with Docker Compose
    • run ingestion into the Postgres service within the same compose network

Main speakers / sources

  • Speaker: Alexey Grigorev (implied by the video title)
  • Other sources/tools referenced: GitHub Codespaces, Docker, pandas, UV, Jupyter, pgcli, Click, pgAdmin, Docker Compose, NYC Taxi dataset (via remote CSV URLs)

Original video