Databricks Interview Questions and Answers: 25 Questions on Spark, Delta Lake, and Pipelines

Databricks interviews usually mix Apache Spark (how jobs run, why they're slow), Delta Lake (what makes tables reliable), and the platform (clusters, jobs, governance). The strongest answers tie each concept to a real problem such as skewed joins, small files, or a pipeline that must be re-run safely.

Databricks renames and reorganizes products fairly often, so say which names you've used (for example Delta Live Tables, now part of the Lakeflow family) and check the job description for the stack.

Below are 25 questions with answers.

Platform and architecture

1. What is Databricks, and what is a lakehouse?

Databricks is a cloud data and AI platform built around Apache Spark. A lakehouse keeps data in open formats on cloud object storage and adds the reliability and governance of a warehouse on top, so one copy of data can serve engineering, analytics, and machine learning.

2. What is the difference between all-purpose clusters, job clusters, and SQL warehouses?

All-purpose clusters are for interactive work and development. Job clusters start for a scheduled job and shut down afterward, which costs less. SQL warehouses are compute tuned for SQL and BI queries.

3. How do you control cost?

Use job clusters for production, auto-termination on interactive clusters, autoscaling with sensible limits, right-sized instance types, spot instances for fault-tolerant work, cluster policies to limit what people can create, and tags for cost reporting.

4. What is Unity Catalog?

The governance layer for data and AI assets. It gives a three-level namespace (catalog, schema, table), central access control, lineage, and auditing across workspaces.

Delta Lake

5. What is Delta Lake, and why use it instead of plain Parquet?

Delta Lake is a table format that adds a transaction log to Parquet files. It gives ACID transactions, schema enforcement and evolution, time travel, and efficient updates and deletes. Plain Parquet has no transactions, so a failed write can leave partial data, and readers can see inconsistent state.

6. How does the Delta transaction log work?

Each commit adds a JSON entry to the _delta_log folder, with periodic checkpoints. The log lists which data files are in the table at each version. Readers use it to see a consistent snapshot, and writers use optimistic concurrency, retrying or failing if another commit conflicts.

7. What is time travel?

Querying an earlier version of a table by version number or timestamp, for audit, debugging, or rollback.

SELECT * FROM sales VERSION AS OF 12;
RESTORE TABLE sales TO VERSION AS OF 12;

8. How do you upsert data with Delta?

With MERGE INTO:

MERGE INTO customers t
USING updates s ON t.id = s.id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

Make sure the source has one row per key, or the merge fails or produces wrong results.

9. What do OPTIMIZE and VACUUM do?

OPTIMIZE compacts many small files into larger ones, which speeds up reads. VACUUM deletes files no longer referenced by the table and older than the retention period, which frees storage but limits time travel. Don't set retention very short while other jobs may still read older versions.

10. How do you lay out data for fast queries?

Partition only on low-cardinality columns that queries often filter on, and keep partitions large enough. Over-partitioning creates many small files. For other columns, use Z-ordering or liquid clustering (the newer approach), which cluster data so queries can skip files.

Spark fundamentals

11. What are transformations and actions, and what is lazy evaluation?

Transformations such as filter and select build a plan but don't run. Actions such as count and write trigger execution. Because Spark sees the whole plan first, it can optimize it.

12. What's a shuffle, and why is it expensive?

A shuffle moves data between executors across the network so that rows with the same key land together, as in joins, groupBy, and distinct. It involves disk and network I/O, so reducing shuffles is the main route to faster jobs.

13. How does Spark join two tables, and how do you speed it up?

By default large joins use a shuffle (sort-merge) join. If one table is small, a broadcast join sends it to every executor and avoids shuffling the large table.

from pyspark.sql.functions import broadcast
df = big.join(broadcast(small), "id")

14. What is data skew, and how do you fix it?

A few keys hold most of the rows, so a few tasks take far longer than the rest. Signs are one task running long in the Spark UI. Fixes are adaptive query execution's skew handling, salting the key, broadcasting the smaller side, or filtering out null or default keys that cause the skew.

15. What is adaptive query execution (AQE)?

Spark adjusts the plan during execution using real statistics: it can coalesce small shuffle partitions, switch join strategies, and split skewed partitions. It's on by default in recent versions.

16. What's the difference between repartition and coalesce?

repartition can increase or decrease partitions and does a full shuffle. coalesce only decreases partitions and avoids a full shuffle, so it's cheaper but can produce uneven partitions.

17. When should you cache a DataFrame?

When you reuse the same DataFrame several times in a job and it's expensive to recompute. Don't cache things used once, and release the cache when done.

18. Why prefer built-in functions to Python UDFs?

Built-in functions run inside the JVM and can be optimized. Python UDFs move data between the JVM and Python and are opaque to the optimizer, so they're slower. Use built-ins or pandas UDFs when possible.

Pipelines

19. What is the medallion architecture?

A layered design: bronze holds raw ingested data, silver holds cleaned and conformed data, and gold holds business-level aggregates ready for reporting. It makes it clear where quality checks happen and lets you rebuild later layers from earlier ones.

20. What is Auto Loader?

A feature that incrementally and efficiently ingests new files from cloud storage as they arrive, tracking which files it has processed, with schema inference and evolution support.

21. How does Structured Streaming handle failures and exactly-once?

It uses checkpoints that store progress and state, so a restarted query continues where it stopped. With replayable sources and idempotent or transactional sinks such as Delta, you get end-to-end exactly-once results.

22. How do you make a pipeline safe to re-run?

Make writes idempotent: use MERGE on a key, overwrite a specific partition, or use a run ID to avoid duplicates. Keep the job's input range explicit and log row counts so you can check results.

23. How do you do data quality checks?

Use expectations or constraints in the pipeline (for example Delta constraints or declarative pipeline expectations), validate row counts, nulls, and uniqueness, and quarantine bad rows rather than failing the whole load, depending on the contract with downstream users.

Scenarios

24. A job that ran in 20 minutes now takes three hours. What do you do?

1. Compare the Spark UI for both runs: which stage got slower?
2. Check input size and whether a table grew or lost its layout, for example many small files.
3. Look for skew in the slow stage, and for spilling to disk.
4. Check cluster changes, such as fewer workers or a changed instance type.
5. Check for changes in the code or Spark settings, and for a new join or UDF.
6. Fix the cause (compaction, broadcast, skew handling, or sizing) and add monitoring.

25. Design a pipeline that ingests daily JSON files and serves a dashboard.

"Auto Loader reads new files into a bronze Delta table with the raw data and metadata. A job parses and cleans them into silver, deduplicating by key with MERGE. A gold table aggregates for the dashboard, which reads from a SQL warehouse. Unity Catalog manages access, expectations catch bad data, and the job runs on a job cluster with alerts on failure and on row-count anomalies."

How to prepare

Run through a notebook that reads a file, joins to another table, writes to Delta, and uses MERGE. Open the Spark UI and find a shuffle and a skewed task. Prepare one story about speeding up or fixing a pipeline, with numbers. For related topics, see our guides to Kafka, Snowflake, and Talend.

Say your answers aloud, especially the performance ones, where candidates tend to list tuning tips without a diagnosis. A mock interview transcript shows where. Follow the employer's rules on AI tools in any live interview.