Docker for Data Engineering: Postgres, Docker Compose, and Real-World Workflows - Alexey Grigorev

What the video workshop covers (Module 1 update)

• Introduces an updated Data Engineering course module focused on Docker (with a brief mention of Terraform, though the walkthrough is primarily Docker-related).
• Goal: provide an end-to-end working version of the module using:
  • updated Postgres
  • newer workflow steps that address common issues from older course runs.
• Workshop materials are linked externally, and the course is scheduled to start in January.

---

Setup / prerequisites for the workshop

Requirements

• Docker installed
• Python installed

Strong recommendation: GitHub Codespaces

Use GitHub Codespaces because it already provides:

• a remote environment
• Python + Docker
• avoids local setup issues, especially on Windows

Workshop workflow

1. Create a GitHub repository (public)
2. Add a README and .gitignore (Python-focused)
3. Open Codespaces via the GitHub UI
4. Use Visual Studio Code Desktop with the Codespaces extension (browser is possible but less convenient)

---

Docker 101 concepts demonstrated

Containers are isolated and (mostly) stateless

• Docker is explained as running software in isolated environments (containers) separate from the host.
• Demonstrations:
  • docker run hello-world to verify Docker works
  • docker run ubuntu downloads an image and runs commands inside the container
  • installing packages inside a container does not affect the host
• Key point: containers created from images are stateless—exiting and re-entering typically returns to the image’s original state.

Docker images vs containers

• Docker image: a snapshot of an OS/environment
• Container: a running instance created from that image

Entry points and command behavior

• Demonstrates overriding container entry behavior:
  • Example: run a Python image but switch to a bash entry point to get a shell

Cleaning up unused containers

• Shows listing and removing exited containers:
  • docker ps -a
  • docker ps -aq and docker rm ... to remove exited containers and keep the environment clean

---

Persisting data from host ↔ container

Volumes (host files usable inside container)

• Shows bind mounts (-v) to map a host directory into a container directory.
• Example pipeline support:
  • create a small script on the host
  • map test/ into the container so the script/files are visible inside Docker
• Without volumes: created files disappear after container re-creation.

---

Pipeline concept: CSV/processing → Postgres

• Defines a data pipeline as:
  • taking input (e.g., a CSV dataset)
  • producing output (e.g., loading into Postgres)
• Today’s ingestion target: NYC Taxi dataset
• Approach:
  • start with a simple parameterized Python pipeline using CLI arguments
  • later evolve it into a Dockerized ingestion workflow

---

Parameterized Python pipeline + pandas + parquet

CLI parameterization

• Uses command-line arguments (sys.argv / parsing concept shown) so the pipeline can process specific partitions (e.g., month number).

Data processing with pandas

• Demonstrates transformations (e.g., adding a month column).

Saving to parquet

• Explains parquet as a binary format optimized compared to CSV.
• Shows installing missing dependencies (e.g., pyarrow) to write parquet.

---

Dependency isolation: virtual environments + UV

Why UV is introduced

• Avoid installing dependencies into the host Python environment.
• Uses virtual environments managed by UV (UV is described as extremely fast, written in Rust, and faster than alternatives like Conda/mamba for this context).

UV workflow shown

• Create a project environment with a chosen Python version (example uses a Python version different from system Python).
• Install dependencies into that environment:
  • uv add pandas
  • uv add pyarrow
• Configure the editor to use the correct interpreter (VS Code interpreter switching concept).
• Reproducibility:
  • uses uv.lock
  • later in Docker builds runs uv sync --locked to ensure exact dependency versions

---

Dockerizing the pipeline (custom Dockerfile)

Build process

Introduces a Dockerfile that:

• starts from a base Python image
• installs required packages (pandas/pyarrow)
• copies pipeline source code into the image
• sets execution behavior using ENTRYPOINT or a docker run command override

Efficient rebuild behavior

• Emphasizes Docker layer caching:
  • rebuilding may skip repeated steps if Dockerfile inputs haven’t changed

Improving the Docker build with UV inside the container

• Shows building in a UV-managed environment by:
  • copying pyproject.toml, uv.lock, and the UV-created Python
  • running uv sync --locked
• Advantage:
  • reproducible dependencies + faster installs
  • avoids pip installing into the container without locking

---

Running Postgres with Docker (and persistent storage)

Postgres container setup

• Uses docker run to start a specific Postgres version (shown as Postgres 18).
• Configured via environment variables:
  • user/password/database (example uses root/root and a database name like ny_taxi)
• Uses:
  • volumes to persist database state across restarts
  • port mapping to access Postgres from the host

Access from outside the container

• Uses pgcli (installed as a dev dependency) to connect via the host mapping.
• Demonstrates:
  • creating a table
  • inserting/selecting rows
  • stopping and restarting Postgres while data remains (because of volume persistence)

---

Ingestion workflow via Jupyter → script

Interactive exploration with Jupyter Notebook

• Installs jupyter as a dev dependency.
• Starts Jupyter in the project environment from within Docker.
• Demonstrates loading the NYC Taxi dataset:
  • dataset is remote and compressed (.csv.gz)
  • loaded directly using pandas via URL
• Notes:
  • dataset size is large (mentions 1,330,000+ records).

Handling pandas dtype issues (important analysis/detail)

• pandas warns about mixed-type columns (example: vendor_id inferred as float due to missing values).
• Fix described:
  • provide explicit dtype / schema guidance
  • parse pickup/dropoff timestamps as datetime
• Framed as especially important because CSV is schemaless, while parquet includes schema.

Performance / reliability pattern: chunked loading

• Uses pandas chunk iteration (chunksize, iterator mode).
• Loads to Postgres incrementally (example chunk size: ~100,000 rows).
• Uses tqdm for progress reporting.
• Avoids “load everything at once” to reduce memory/time uncertainty and improve debuggability.

Converting notebook to production script

• Converts notebook to a Python script using jupyter nbconvert.
• Adds parameterization:
  • year, month
  • Postgres connection parameters (PG user, PG password, PG host, PG database, PG port)
  • chunk size
  • target table naming
• Demonstrates table replacement logic using if_exists='replace' when writing to SQL.

CLI improvement using Click

• Uses Click to generate a proper command-line interface.
• Result: ingestion script runs like a standard CLI tool with options.

---

Orchestration: connecting containers correctly (Docker networking)

The “localhost” problem

• When running ingestion container separately from Postgres:
  • inside ingestion container, localhost points to itself (not the Postgres container)

Fix: use Docker networks

• Create a shared Docker network (example name: pg_network):
  • attach Postgres container to the network
  • attach ingestion container to the same network
  • use the Postgres container name (service hostname) as the PG host

---

UI: Postgres Admin in Docker

• Runs pgAdmin in a separate container.
• Uses port mapping so pgAdmin is accessible via browser.
• Connects pgAdmin to Postgres using the shared network hostname.
• Demonstrates checking inserted data via pgAdmin (table lists, row counts).

---

Docker Compose: final end-to-end workflow

Why Docker Compose is used

• Replaces manual start/stop sequences with a single docker compose up.
• A compose file (docker-compose.yml) is created (partly via an AI-assisted/editing step).
• Compose services:
  • Postgres
  • pgAdmin
• Notes:
  • Docker Compose provides a default internal network so services can reach each other by service name.
  • DB state depends on volumes:
    • restarting compose “from scratch” may clear database contents if volume persistence isn’t maintained as expected.

Running ingestion using the compose network

• After starting compose, the ingestion script container is run and attached to the compose default network (example network name: pipeline_default).
• Verifies results by querying row counts (database fills incrementally).

---

Key “review / guide / tutorial” style takeaways

• Codespaces-first recommendation to avoid OS-specific Docker installation pain (Windows especially).
• Docker basics:
  • isolation, images vs containers, statelessness
  • entrypoint and how containers behave on run
• Data engineering practicality:
  • use explicit schemas when reading CSV with pandas (avoid dtype inference problems)
  • use chunked loading for large datasets
  • ensure reproducibility with UV + uv.lock and uv sync --locked
• Production-style container connectivity:
  • prefer Docker networks over localhost for inter-container communication
• Operational UX:
  • use pgAdmin as a convenient Postgres interface when running in Docker
• Orchestration:
  • consolidate Postgres + pgAdmin startup with Docker Compose
  • run ingestion into the Postgres service within the same compose network

---

Main speakers / sources

• Speaker: Alexey Grigorev (implied by the video title)
• Other sources/tools referenced: GitHub Codespaces, Docker, pandas, UV, Jupyter, pgcli, Click, pgAdmin, Docker Compose, NYC Taxi dataset (via remote CSV URLs)