50 Data Engineering Terms EXPLAINED For Your Interview

Summary

The video is a beginner-oriented, interview-focused tour of data engineering concepts. Its central message is that understanding how data systems fit together—and why a design is used—is more useful than memorizing definitions. The presenter moves from databases and storage through data architecture, processing, modeling, and operational tooling.

Databases and storage

• Database: A system used by applications to store and query structured records, often through CRUD operations. The video emphasizes transactions and database constraints.
• ACID properties:
  • Atomicity: A transaction completes entirely or is rolled back.
  • Consistency: Transactions preserve defined rules and constraints.
  • Isolation: Concurrent transactions should not interfere with one another in unintended ways.
  • Durability: Once committed, a change should survive a crash.
• Write-ahead logging (WAL): Changes are recorded in a log before they are safely reflected in the main data files. The log can help recover committed changes after a failure.
• Data warehouse: A system and design approach for organizing data for analytical queries. A warehouse can also serve as a centralized place to bring together information from multiple organizational sources.
• Data lake: Object storage—such as S3 or Azure Data Lake Storage—is a relatively low-cost place to store structured, semi-structured, and unstructured data. The video contrasts this flexibility with database-style constraints and transaction features.
• Data lakehouse: Combines lake storage with selected warehouse or database capabilities. The presenter focuses on open table formats such as Delta Lake and Apache Iceberg, which add metadata and transaction-log mechanisms to files in a lake.
• Lakebase: Separates database compute from storage, so multiple processing engines can potentially work with shared data without duplicating it.

Organizing and governing data

• Data mart: A focused subset of an organization’s larger analytical repository, intended for a department or business area such as HR or finance.
• Data mesh: An organizational approach in which domain teams own and maintain their data products, while a central platform team supports shared infrastructure, standards, and governance. The video’s shorthand is centralized platform, decentralized ownership.
• Data fabric: A cross-system layer for capabilities such as governance, lineage, discovery, and observability. Its purpose is to help organizations find and understand data and its dependencies across different tools and platforms.

Pipelines and data loading

• Pipeline: A series of steps that moves data from source systems to destinations, often including cleaning or transformation.
• ETL and ELT:
  • ETL: Extract, transform, then load.
  • ELT: Extract and load first, then transform within or near the destination platform.
  • The presenter says ELT is common for larger data volumes because raw data can be landed before transformations are applied.
• Incremental loading: Loads only newly arrived or changed data rather than reprocessing the full historical dataset on every run.
• Backdated refresh (backfill): Lets a pipeline rerun for a specified date or time range. This is important when data is missing, incorrect, late, or changed after its original load.
• Idempotency and exactly-once processing: Design goals that prevent rerunning a pipeline from creating duplicate or inconsistent results. The presenter discusses watermarks—stored progress markers such as the last successfully processed date or event—as one way to track what has already been processed.
• Upsert: Updates a matching existing record or inserts it if it does not already exist. It is described as the equivalent of a merge-style operation.

Data modeling and analytics

• OLTP modeling: Associated with operational databases and normalized tables. Normalization helps reduce duplication and enforce structure; the video highlights primary keys, avoiding multiple values in a column, and avoiding certain dependencies.
• OLAP dimensional modeling: Common in analytics. It organizes measures in fact tables and descriptive context in dimension tables.
• Star schema: A fact table connects directly to dimension tables.
• Snowflake schema: Some dimensions are further divided into related tables, creating a more branched design.
• Slowly changing dimensions (SCDs): Techniques for handling changes to descriptive data, such as a customer’s address:
  • Type 1: Overwrite the old value; history is not retained.
  • Type 2: Add a new version or row so historical values can be retained.
  • The presenter also mentions other types but says Types 1 and 2 are the common approaches.
• One big table: A precombined table intended to reduce downstream joins and give analysts a convenient starting point. The video notes that this approach can be useful in some cases but may also have drawbacks.
• Medallion architecture: A common progression of data refinement:
  • Bronze: Land raw data.
  • Silver: Clean, transform, and apply business logic.
  • Gold: Prepare curated, modeled data for analytics and reporting.

Processing and architecture patterns

• Batch processing: Processes accumulated data on a schedule, such as hourly or daily. The presenter says it is widely used when results are not needed immediately.
• Stream processing: Processes events as they arrive, supporting use cases such as live dashboards or recommendations.
• Lambda architecture: Combines a low-latency streaming path with a separate batch path for broader historical analysis.
• Kappa architecture: A streaming-centered alternative that aims to use one processing path rather than separate batch and speed layers. The video notes that replaying or recovering historical data still needs to be considered.
• Change data capture (CDC): Captures source inserts, updates, and deletes so downstream systems can reflect those changes. The presenter distinguishes CDC, which can record details of what changed, from change tracking (CT), which can identify that a record changed without necessarily retaining the same level of detail.
• Schema evolution: Allows a destination table’s structure to adapt when the source schema changes, for example when a new column is added.
• Schema on write vs. schema on read: The video contrasts systems that validate or apply schema when data is written with data-lake-style approaches that interpret schema when data is read.
• Row-based and column-based formats:
  • Row-based formats store fields from each record together, which can suit record-oriented access and updates.
  • Column-based formats store values by column, which can make analytical queries that read only selected columns more efficient.
  • CSV and JSON are discussed as row-oriented examples; Parquet is presented as a columnar example.

Distributed systems and tools

• Distributed processing: Instead of scaling one machine indefinitely, a cluster spreads storage or computation across multiple machines. The presenter contrasts vertical scaling (making one machine larger) with horizontal scaling (adding machines).
• HDFS: Hadoop’s distributed file system, which uses a virtual, cluster-wide view of files stored across multiple machines.
• Hadoop MapReduce: Moves computation toward where data resides, reducing the need to move large datasets over a network.
• Apache Spark: A distributed processing framework that can use memory to speed up computation while spilling data to disk when needed.
• Orchestration: Coordinates pipeline tasks, dependencies, schedules, conditions, and alerts. The video names Apache Airflow and Azure Data Factory as examples.
• DBT (data build tool): Primarily organizes and runs data transformations—often written in SQL—across different data platforms while handling some platform-specific boilerplate. The presenter also notes support for Python models.
• Apache Kafka: A technology for handling event streams and messages used in streaming architectures.
• CI/CD and DataOps: Continuous integration and delivery help test, update, and deploy data solutions reliably. Data operations must account for data and pipelines as well as code. The presenter mentions Terraform, Azure templates, and Databricks Asset Bundles.

Speakers and sources featured

• Speaker: Ansh Lamba, the video’s solo presenter. No guest speakers are featured.
• Sources and organizations explicitly mentioned or consulted in the subtitles: Google’s File System (GFS) paper, the University of California, Berkeley, LinkedIn, and a Wikipedia page about slowly changing dimensions. The presenter also refers to browser searches for storage-cost and technology-history information.
• Examples and technologies discussed: Amazon, Netflix, AWS S3, Azure Data Lake Storage, PostgreSQL, SQL Server, Snowflake, Databricks, Delta Lake, Apache Iceberg, Trino, Spark, Kafka, Airflow, DBT, and several other cloud and data platforms.