Video summary

Finance Domain End-to-End Data Engineering Project in Databricks (Free Edition) with Source Code

Main summary

Key takeaways

Technology

Summary

Narendra Kumar demonstrates an end-to-end, metadata-driven banking data engineering project built in Databricks Free Edition. The example bank, “Neo Bank,” combines data from its core banking system, payment gateways, and credit bureaus to support unified analytics and self-service access for business users.

Architecture and data model

The project uses a medallion architecture:

  • Sources: Four Azure SQL Server tables—branches, customers, accounts, and transactions—and two CSV-based sources—credit bureau reports and payment gateway logs.
  • Bronze: Raw source data is ingested and appended, preserving incoming data.
  • Silver: Data is refined and kept current using full loads, appends, or key-based merges, depending on each table’s metadata.
  • Gold: Business-focused aggregate tables are prepared for analytics and dashboards.

The project uses JDBC connections for SQL Server and Auto Loader for CSV files. Databricks Volumes provide file storage for the demonstration; the presenter notes that cloud storage can also be mapped for a practical deployment.

Metadata-driven framework and governance

The framework is controlled by metadata rather than table-specific pipeline code. Metadata tables define source and target details, table processing order and active status, load strategy, keys, and watermark columns. A watermark table supports incremental SQL Server loads, while an audit table records run IDs, layers, timestamps, status, record counts, and error messages.

The presenter creates a banking catalog with schemas for metadata, bronze, silver, and gold. Unity Catalog manages the project’s tables and volume. Credentials are stored in Databricks secret scopes rather than hard-coded in notebooks. A Gmail app password is also stored as a secret for email delivery.

Ingestion, transformation, and orchestration

Reusable PySpark notebooks read metadata and process tables according to their configuration:

  • SQL Server ingestion uses JDBC and adds a condition on the watermark column for incremental loads.
  • Auto Loader reads CSV files, tracks processed files with checkpoint data, and manages schema information. Its rescued-data column can capture records that do not fit the expected schema.
  • Bronze ingestion appends data. Silver processing applies the configured full-load, append, or merge behavior, updates insert and update timestamps, and advances watermarks when applicable.
  • Gold notebooks create business aggregates, including branch performance, customer 360, daily bank KPIs, risk-customer summaries, and transaction-channel summaries.

Databricks Jobs orchestrate the notebooks. The tutorial builds a nested-job pattern: an outer job iterates over table metadata, while an inner job runs the parameter-reading, bronze-ingestion, and silver-processing steps for each table. A separate job handles gold transformations. The presenter also covers task dependencies, concurrency, retries, task-value passing, and run IDs.

Notifications and business consumption

A custom email notebook joins audit records with table metadata and sends an HTML summary through Gmail SMTP. The email reports overall success or failure, along with table-level status and record counts. The presenter contrasts this with built-in job notifications, which are simpler but less informative for tracking individual tables.

For analytics, the project creates a multi-page Databricks dashboard with executive metrics, customer and branch insights, and payment gateway and device breakdowns. The tutorial demonstrates counters, bar and pie charts, tables, filters, and AI-assisted visualization creation. The presenter notes that AI-generated charts may need correction.

A Genie workspace lets users ask questions in natural language and see generated SQL and visualizations. Configuration options include instructions, table relationships, reusable SQL expressions, example queries, monitoring, and benchmark questions. Databricks One is shown as a simplified interface for business users to access dashboards and Genie.

The dashboard can be scheduled directly or refreshed as part of the master job after gold processing.

Guides and demonstrations covered

  • Setting up Azure SQL Server tables and loading historical and incremental sample data.
  • Creating Databricks catalog schemas, metadata tables, a volume, and secret scopes.
  • Building reusable SQL Server and Auto Loader ingestion notebooks.
  • Implementing bronze-to-silver load strategies and watermark updates.
  • Creating nested Databricks Jobs and a master end-to-end job.
  • Building gold-layer banking aggregates and audit logging.
  • Sending custom HTML email summaries through Gmail SMTP.
  • Creating and publishing dashboards, configuring Genie, and scheduling dashboard refreshes.
  • Testing historical and incremental end-to-end runs.

The presenter says the project’s notebooks and data files are available in a repository for viewers to try.

Main speaker/source

  • Narendra Kumar — presenter and tutorial instructor.

Rate this summary

Your feedback will help improve summaries.

Improve this summary

Reprocess with a stronger model when the summary feels incomplete or inaccurate.

Pro

Translate summary in another language

Pro

Ask questions to this video

Chat for follow-up questions, clarifications, and source-backed answers.

Coming soon

Share this summary

Original video