10 free Databricks Data Engineer Professional practice questions with the correct answer and a full explanation for each, taken from the CertStash pack of 339 questions. Work through them, then open each answer to check your reasoning.
Get all 339 questions (US$39) · Download these 10 as a PDF
Question 1
An upstream system has been configured to pass the date for a given batch of data to the Databricks Jobs API as a parameter. The notebook to be scheduled will use this parameter to load data with the following code:
df = spark.read.format("parquet").load(f"/mnt/source/(date)")
Which code block should be used to create the date Python variable used in the above code block?
Show answer and explanation
Correct answer: E. dbutils.widgets.text("date", "null") date = dbutils.widgets.get("date")
When a job passes parameters to a notebook via the Databricks Jobs API, the notebook must use dbutils.widgets to retrieve them. The dbutils.widgets.text() call creates a widget with a default value, and dbutils.widgets.get() retrieves the parameter value passed by the upstream system. This is the standard Databricks pattern for job parameters in notebooks.
Why the other options are wrong
- A. spark.conf.get() retrieves Spark configuration values, not job parameters passed by the Jobs API.
- B. input() is for interactive user input and cannot access job parameters from the API.
- C. sys.argv accesses command-line arguments, which are not used by the Jobs API for parameter passing in Databricks.
- D. dbutils.notebooks.getParam() is used to pass parameters between notebooks, not from the Jobs API to a notebook.
Question 2
The Databricks workspace administrator has configured interactive clusters for each of the data engineering groups. To control costs, clusters are set to terminate after 30 minutes of inactivity. Each user should be able to execute workloads against their assigned clusters at any time of the day.
Assuming users have been added to a workspace but not granted any permissions, which of the following describes the minimal permissions a user would need to start and attach to an already configured cluster.
Show answer and explanation
Correct answer: D. "Can Restart" privileges on the required cluster
A user needs minimal permissions to start and attach to an already-configured cluster. The 'Can Restart' privilege allows a user to restart a terminated cluster and attach to it, which is exactly what's needed when a cluster auto-terminates after 30 minutes of inactivity. This permission is sufficient for the stated use case without requiring cluster creation or management capabilities.
Why the other options are wrong
- A. 'Can Manage' privileges grant far more access than needed and violate the principle of least privilege.
- B. Workspace Admin privileges and cluster creation allowed are excessive for simply attaching to an existing cluster.
- C. Cluster creation allowed is unnecessary since the cluster is already configured; only restart and attach capabilities are needed.
- E. Cluster creation allowed is not required when the cluster already exists and is pr-onfigured.
Question 3
When scheduling Structured Streaming jobs for production, which configuration automatically recovers from query failures and keeps costs low?
Show answer and explanation
Correct answer: D. Cluster: New Job Cluster; Retries: Unlimited; Maximum Concurrent Runs: 1
Maximum Concurrent Runs: 1 For production Structured Streaming jobs, using a new job cluster with unlimited retries and a maximum of 1 concurrent run is optimal. The new job cluster is automatically created and terminated with each run, minimizing costs. Unlimited retries ensure recovery from transient failures, and limiting to 1 concurrent run prevents duplicate processing and ensures exactly-once semantics.
Why the other options are wrong
- A. Unlimited concurrent runs can cause duplicate processing and higher costs without benefit.
- B. No retries means the job fails permanently on any failure, which is unreliable for production.
- C. All-purpose clusters continue running between jobs, consuming resources and costs even when idle.
- E. All-purpose clusters continue running between jobs wasting resources, and no retries leaves the job vulnerable to transient failures.
Question 4
The data engineering team has configured a Databricks SQL query and alert to monitor the values in a Delta Lake table. The recent_sensor_recordings table contains an identifying sensor_id alongside the timestamp and temperature for the most recent 5 minutes of recordings.
The below query is used to create the alert:
The query is set to refresh each minute and always completes in less than 10 seconds. The alert is set to trigger when mean (temperature) > 120.
Notifications are triggered to be sent at most every 1 minute.
If this alert raises notifications for 3 consecutive minutes and then stops, which statement must be true?
Show answer and explanation
Correct answer: E. The average temperature recordings for at least one sensor exceeded 120 on three consecutive executions of the query
The SQL query groups by sensor_id and calculates MEAN(temperature) for each sensor. The alert is configured to trigger when mean(temperature) > 120, meaning it fires when the average temperature for at least one sensor exceeds this threshold. If notifications are sent for 3 consecutive minutes, this means the alert condition was met on 3 consecutive query executions, indicating that for at least one sensor, the mean temperature exceeded 120 across those three runs. The query groups by sensor_id, so the alert evaluates the mean per sensor, not a global average across all sensors.
Why the other options are wrong
- A. The alert evaluates mean temperature per sensor (GROUP BY sensor_id), not the total average across all sensors.
- B. If the table were unresponsive, the query would fail or return no results; it wouldn't trigger an alert based on temperature thresholds.
- C. The query completes in less than 10 seconds and refreshes every minute; there is no evidence of query failures, and if it failed it wouldn't produce temperature-based alert conditions.
- D. The alert is based on MEAN(temperature), not MAX(temperature); exceeding 120 on the maximum does not guarantee the mean exceeds 120.
Question 5
A junior developer complains that the code in their notebook isn't producing the correct results in the development environment. A shared screenshot reveals that while they're using a notebook versioned with Databricks Repos, they're using a personal branch that contains old logic.
The desired branch named dev-2.3.9 is not available from the branch selection dropdown.
Which approach will allow this developer to review the current logic for this notebook?
Show answer and explanation
Correct answer: B. Use Repos to pull changes from the remote Git repository and select the dev-2.3.9 branch.
When a desired branch is not visible in the Databricks Repos branch selection dropdown, it means the local Repos clone does not have that branch information. Using Repos to pull changes from the remote Git repository will fetch all branches from the remote, making dev-2.3.9 available for selection and checkout in the dropdown.
Why the other options are wrong
- A. A pull request does not fetch branches from the remote repository; the REST API cannot update branches in this context.
- C. Checking out a branch that doesn't exist in the local Repos clone will fail; pulling first is necessary.
- D. Merging and re-cloning is unnecessary; simply pulling changes will fetch all remote branches.
- E. Merging branches and making pull requests does not solve the problem of a missing branch in the dropdown.
Question 6
The security team is exploring whether or not the Databricks secrets module can be leveraged for connecting to an external database.
After testing the code with all Python variables being defined with strings, they upload the password to the secrets module and configure the correct permissions for the currently active user. They then modify their code to the following (leaving all other variables unchanged).
Which statement describes what will happen when the above code is executed?
Show answer and explanation
Correct answer: E. The connection to the external table will succeed; the string "REDACTED" will be printed.
When `dbutils.secrets.get()` retrieves a secret from the Databricks secrets module, it returns the actual secret value that can be used for authentication. The connection to the external database will succeed because the password variable contains the correct credentials retrieved from the secrets store. However, when the password variable is printed, Databricks automatically redacts secret values in output for security purposes, displaying "REDACTED" instead of the plain text password. This prevents accidental exposure of credentials in logs or notebook output while still allowing the authentication to work properly.
Why the other options are wrong
- A. The connection will succeed, not fail, because the secrets module correctly provides the authentication credentials.
- B. The secrets module does not prompt for interactive input; it directly retrieves the pr-tored secret value without user interaction.
- C. While the connection succeeds, the password will not print in plain text; Databricks redacts secret values in output automatically.
- D. The password will not print in plain text; Databricks redacts secrets in output display, showing "REDACTED" instead.
Question 7
The data science team has created and logged a production model using MLflow. The following code correctly imports and applies the production model to output the predictions as a new DataFrame named preds with the schema "customer_id LONG, predictions DOUBLE, date DATE".
The data science team would like predictions saved to a Delta Lake table with the ability to compare all predictions across time. Churn predictions will be made at most once per day.
Which code block accomplishes this task while minimizing potential compute costs?
Show answer and explanation
Correct answer: A. preds.write.mode("append").saveAsTable("churn_preds")
Option A uses `preds.write.mode("append").saveAsTable("churn_preds")`, which creates a managed Delta Lake table with append mode. This is ideal for the requirement to compare predictions across time while minimizing compute costs. The `saveAsTable()` method creates a proper Delta table with ACID properties and schema tracking, and append mode allows daily incremental writes without unnecessary data reorganization. This approach leverages Delta Lake's optimization features like Z-ordering and compaction that occur automatically for managed tables, reducing compute overhead compared to alternatives.
Why the other options are wrong
- B. Uses `save()` with an external path rather than `saveAsTable()`, which doesn't create a managed Delta table and loses Delta Lake's built-in optimizations for time-series data comparisons.
- C. Uses `writeStream` with checkpoint path and `start()`, which is for streaming operations; predictions are batch daily operations, not continuous streams, making this unnecessarily complex and costly.
- D. Uses `write.format("delta").mode("overwrite")`, which completely overwrites the table each time instead of appending, eliminating the ability to compare predictions across time as required.
- E. Uses `writeStream` with append mode to a table, which is streaming syntax inappropriate for batch daily predictions and adds unnecessary streaming overhead and state management costs.
Question 8
An upstream source writes Parquet data as hourly batches to directories named with the current date. A nightly batch job runs the following code to ingest all data from the previous day as indicated by the date variable:
Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order.
If the upstream system is known to occasionally produce duplicate entries for a single order hours apart, which statement is correct?
Show answer and explanation
Correct answer: B. Each write to the orders table will only contain unique records, but newly written records may have duplicates already present in the target table.
The code applies `.dropDuplicates(["customer_id", "order_id"])` to the loaded DataFrame before writing with `.mode("append")`. This deduplication step removes duplicate rows within the newly loaded batch based on the composite key, ensuring each write contains only unique records from that day's data. However, the append mode writes these deduplicated records directly to the target table without any merge logic or upsert capability. This means newly written records may already exist in the target table from previous ingestions, creating duplicates across the entire table. The operation does not fail (eliminating D), does not overwrite existing records (eliminating C), does not prevent duplicates in the target table (eliminating A), and does not perform deduplication across the union of new and existing records (eliminating E).
Why the other options are wrong
- A. Append mode does not check the target table before writing; duplicates can exist in the target table after the write completes.
- C. Append mode writes new records without modifying existing ones; overwrite behavior would require mode("overwrite"), not mode("append").
- D. Append mode does not validate against existing data and will not fail even if duplicates exist in the target table.
- E. The deduplication only occurs within the new batch DataFrame; append mode does not merge or deduplicate against existing target table records.
Question 9
A junior member of the data engineering team is exploring the language interoperability of Databricks notebooks. The intended outcome of the below code is to register a view of all sales that occurred in countries on the continent of Africa that appear in the geo_lookup table.
Before executing the code, running SHOW TABLES on the current database indicates the database contains only two tables: geo_lookup and sales.
Which statement correctly describes the outcome of executing these command cells in order in an interactive notebook?
Show answer and explanation
Correct answer: E. Cmd 1 will succeed and Cmd 2 will fail. countries_af will be a Python variable containing a list of strings.
Cmd 1 executes PySpark code that filters the geo_lookup table for African countries and collects the results into a Python list. The list comprehension `[x[0] for x in …]` extracts the first column (country names) from each row, creating a Python variable `countries_af` containing a list of strings. Cmd 2 then attempts to execute SQL that references `countries_af` as if it were a table or view in the database. However, `countries_af` is a Python variable living in the notebook's Python environment, not a SQL-accessible table or view. SQL queries in Databricks notebooks cannot directly reference Python variables without explicit conversion (such as using temporary views or SQL context). Therefore, Cmd 2 fails with a table not found error, while Cmd 1 succeeds in creating the Python list.
Why the other options are wrong
- A. Cmd 2 fails because countries_af is a Python variable, not a registered SQL view that Cmd 2 can access.
- B. Cmd 2 fails immediately on the current database without searching other databases; countries_af is not registered as a table or view anywhere.
- C. Cmd 1 creates a list of strings, not a PySpark DataFrame, due to the list comprehension and collect() operation that materializes results into Python.
- D. Cmd 1 succeeds; it creates a valid Python variable containing the filtered country list without any errors.
Question 10
A Delta table of weather records is partitioned by date and has the below schema: date DATE, device_id INT, temp FLOAT, latitude FLOAT, longitude FLOAT To find all the records from within the Arctic Circle, you execute a query with the below filter: latitude > 66.3
Which statement describes how the Delta engine identifies which files to load?
Show answer and explanation
Correct answer: D. The Delta log is scanned for min and max statistics for the latitude column
Delta Lake uses the Delta log (transaction log) to store metadata including min and max statistics for all columns in data files. When a query filters on the latitude column, the Delta engine scans these statistics in the Delta log to perform predicate pushdown and identify which Parquet files contain records that could match the filter condition, avoiding unnecessary full-file reads.
Why the other options are wrong
- A. Caching all records to an operational database defeats the purpose of file pruning and is not how Delta Lake operates.
- B. While Parquet files do contain footers with statistics, Delta Lake uses the more efficient Delta log metadata rather than scanning individual file footers.
- C. Caching all records to storage and then filtering is inefficient and not the Delta Lake optimization approach.
- E. The Hive metastore stores table schema and partition information but not the min/max statistics used for file pruning.
That was 10 of 339.
The full Databricks Data Engineer Professional pack has all 339 questions, each with the answer, the explanation and why the other options are wrong, plus a questions-only copy for timed runs. US$39, paid once, with free monthly updates and a pass-or-your-money-back guarantee.