Databricks Job: End‑to‑End Demo with Loops, Parameters, Failure Handling & Alerts

Summary

Narendra Kumar demonstrates how to build an end-to-end Databricks job that ingests multiple tables, transforms them, refreshes a dashboard, and sends notifications.

Job Creation and Parameters

• The tutorial focuses on Databricks Jobs, which orchestrate notebooks and other tasks. It briefly mentions ingestion pipelines and UI-based ETL as alternatives but does not cover them in depth.
• A notebook task copies tables from a sample schema into a training schema. The job can supply a table_name parameter, overriding the notebook’s default.
• The demo shows how to correct a failed notebook run and retry it using Repair run.

Ingesting Multiple Tables

• A For each task runs the same parameterized notebook for multiple table names. Inputs can be configured to run sequentially or concurrently.
• To avoid hard-coding the list, a metadata table stores the table names. A separate notebook reads the table, formats the names as a JSON array, and passes the result to the loop using Databricks task values.
• Task dependencies control execution order, and dependency conditions can be adjusted for different success or failure scenarios.

Failure Handling

• The demo intentionally causes one table’s ingestion to fail while another succeeds.
• After the code is fixed, Repair task reruns only the failed loop item rather than repeating work for the successful item.
• The walkthrough also shows how a misconfigured parameter caused an unexpected failure and how correcting it resolved the issue.

Transformation, Dashboard, and Notifications

• A transformation notebook joins the ingested customer and transaction tables, groups transactions by country, and creates a summary table.
• A Databricks dashboard visualizes transaction counts by country. A dashboard task refreshes it using a selected warehouse.
• Dashboard subscribers can receive an email when the dashboard refreshes. The email includes a dashboard screenshot and link.
• Job-level notifications can also be configured for selected outcomes, such as success and failure.

Run Details and Auditing

• Task run logs and query history are available for inspecting execution.
• Databricks system tables contain job and task run information, including run times and parameters. The speaker also notes that usage tables provide job cost information.

Tutorial Topics Covered

1. Creating a notebook-based Databricks job.
2. Passing task and job parameters.
3. Looping over a list of tables, including parallel execution.
4. Building loop inputs dynamically from a metadata table.
5. Repairing failed runs and handling partial loop failures.
6. Transforming data and refreshing a dashboard.
7. Setting up job and dashboard email notifications.
8. Finding run and cost information in system tables.

Main speaker/source: Narendra Kumar, from the DataBeli channel.