85 free Databricks Associate Developer for Apache Spark practice questions from the CertStash pack of 342, each with the correct answer and a full explanation. The rule is the same for every exam: ten free on the product page, 85 free here, all 342 in the pack, and each step includes the one before it.
Get all 342 questions (US$39) · Download the first 10 as a PDF
Question 1
Which of the following operations can be used to create a DataFrame with a subset of columns from DataFrame storesDF that are specified by name?
Show answer and explanation
Correct answer: B. storesDF.select()
The select() method is the standard PySpark operation for creating a DataFrame containing a subset of columns specified by name. It accepts column names as arguments and returns a new DataFrame with only those columns.
Why the other options are wrong
- A. subset() is not a valid DataFrame method in PySpark.
- C. selectColumn() does not exist in the PySpark API.
- D. filter() is used to subset rows based on conditions, not to select columns.
- E. drop() removes columns rather than selecting specific ones.
Question 2
Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000?
Show answer and explanation
Correct answer: E. storesDF.filter(col("sqft") <= 25000)
The filter() method with col() function is the correct syntax for filtering rows in PySpark. The col() function references a column by name, and col("sqft") <= 25000 creates a proper boolean condition for filtering.
Why the other options are wrong
- A. Strings cannot be used directly with comparison operators in filter conditions; col() is required.
- B. Unquoted column names like sqft are treated as undefined variables, not column references.
- C. The bracket syntax storesDF[sqft] is invalid; where() also requires col() for proper column reference.
- D. The operator > filters for values greater than, not less than or equal to as required.
Question 3
Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30?
Show answer and explanation
Correct answer: E. storesDF.filter((col("sqft") <= 25000) | (col("customerSatisfaction") >= 30))
The filter() method with col() functions and the pipe operator (|) for OR logic is the correct syntax. Both col("sqft") and col("customerSatisfaction") properly reference columns, and the | operator combines conditions with OR logic. Parentheses ensure clarity in the compound condition.
Why the other options are wrong
- A. While the syntax is mostly correct, omitting parentheses around the compound condition is less clear and may cause precedence issues.
- B. Python's "or" keyword is not used in PySpark column expressions; the pipe operator | is required.
- C. Unquoted column names and the Python "or" keyword are both invalid in PySpark filter conditions.
- D. The col() function requires quoted strings as arguments; col(sqft) without quotes is invalid syntax.
Question 4
Which of the following code blocks returns a new DataFrame from DataFrame storesDF where column storeId is of the type string?
Show answer and explanation
Correct answer: B. storesDF.withColumn("storeId, col("storeId").cast(StringType()))
The withColumn() method with col().cast(StringType()) is the correct syntax for casting a column to a new type in PySpark. The col() function references the column, and cast() with StringType() performs the type conversion.
Why the other options are wrong
- A. The cast() function is called as a method on col(), not as a standalone function with two arguments.
- C. The as() method is not used for casting in PySpark; cast() is the correct method, and missing closing parenthesis is a syntax error.
- D. Missing quotes around the column name in col(storeId) and incomplete StringType syntax without parentheses.
- E. The cast() function requires col() to reference the column; cast("storeId") alone is invalid syntax.
Question 5
Which of the following code blocks returns a new DataFrame with a new column employeesPerSqft that is the quotient of column numberOfEmployees and column sqft, both of which are from DataFrame storesDF? Note that column employeesPerSqft is not in the original DataFrame storesDF.
Show answer and explanation
Correct answer: A. storesDF.withColumn("employeesPerSqft", col("numberOfEmployees") / col("sqft"))
The withColumn() method is used to add new columns to a DataFrame. The first argument is the new column name as a string, and the second argument is the expression col("numberOfEmployees") / col("sqft") that defines the column's values using col() references and arithmetic operations.
Why the other options are wrong
- B. String literals cannot be used directly in arithmetic expressions; col() is required to reference columns.
- C. The select() method chooses existing columns, not creates new derived columns; withColumn() is the correct method.
- D. The select() method is incorrect for creating new columns, and the expression syntax is incomplete.
- E. The withColumn() method takes a string column name as the first argument, not col("employeesPerSqft").
Question 6
Which of the following code blocks returns a new DataFrame where column division from DataFrame storesDF has been replaced and renamed to column state and column managerName from DataFrame storesDF has been replaced and renamed to column managerFullName?
Show answer and explanation
Correct answer: E. (storesDF.withColumnRenamed("division", "state") .withColumnRenamed("managerName", "managerFullName"))
The withColumnRenamed() method renames a single column at a time, taking the old column name and new column name as arguments. The correct syntax chains two withColumnRenamed() calls: the first renames "division" to "state", and the second renames "managerName" to "managerFullName".
Why the other options are wrong
- A. The withColumnRenamed() method does not accept arrays of names; it must be called once per column rename.
- B. The withColumn() method adds or replaces columns but does not rename them; it also preserves the old column.
- C. The withColumn() method without col() references is invalid, and it adds columns rather than renaming them.
- D. The argument order is reversed; withColumnRenamed(oldName, newName) requires the old name first, then the new name.
Question 7
Which of the following describes the Spark driver?
Show answer and explanation
Correct answer: D. The Spark driver is the program space in which the Spark application’s main method runs coordinating the Spark entire application.
The Spark driver is the program space where the main method of a Spark application runs. It is responsible for coordinating the entire Spark application by creating the SparkContext, defining transformations and actions, and orchestrating execution across the cluster. The driver itself does not perform all execution (that occurs on executors), is not fault-tolerant by design, and is not horizontally scaled for throughput.
Why the other options are wrong
- A. The driver coordinates but does not perform all execution; executors handle actual task execution.
- B. The driver is a single point of failure and cannot recover if it fails.
- C. While the driver is the highest level, 'synonymous with the Spark application' is less precise than describing its role as the coordination program space.
- E. The driver is not horizontally scaled; it runs as a single process on one machine.
Question 8
Which of the following describes the relationship between nodes and executors?
Show answer and explanation
Correct answer: C. An executor is a processing engine running on a node.
An executor is a JVM process running on a node that performs actual computation by executing tasks. A node is a machine in the cluster, and executors run on nodes as the processing engines. Multiple executors can run on a single node, and nodes can exist without executors. This relationship is one of containment: executors are hosted by nodes.
Why the other options are wrong
- A. Executors and nodes are directly related; executors run on nodes.
- B. This is backwards; a node does not run on an executor.
- D. There is no requirement for equal numbers of executors and nodes.
- E. There is no requirement that there be more nodes than executors.
Question 9
Which of the following will occur if there are more slots than there are tasks?
Show answer and explanation
Correct answer: A. The Spark job will likely not run as efficiently as possible.
Slots represent available task execution capacity on executors. When there are more slots than tasks, not all available parallelism is utilized, leading to underutilization of resources and suboptimal job efficiency. The application will still run successfully, but it will not leverage all available processing capacity, resulting in lower throughput than possible.
Why the other options are wrong
- B. Spark does not require equal or greater tasks than slots; jobs run successfully with idle slots.
- C. Executors do not automatically shut down when slots exceed tasks.
- D. Spark does not automatically generate tasks to fill slots.
- E. Spark will use multiple slots and tasks when available, not revert to a single slot.
Question 10
Which of the following is the most granular level of the Spark execution hierarchy?
Show answer and explanation
Correct answer: A. Task
The Spark execution hierarchy from coarsest to finest is: Job, Stage, Task, Slot. A task is the smallest unit of work and represents the execution of a specific operation on a partition of data by a single executor. Tasks cannot be subdivided further in the execution hierarchy.
Why the other options are wrong
- B. An executor contains multiple tasks; it is not the most granular level.
- C. A node is even coarser and contains multiple executors and tasks.
- D. A job is the coarsest level, containing multiple stages and tasks.
- E. A slot is a container for executing a task, but the task itself is the unit of work being executed.
Question 11
Which of the following operations performs a cross join on two DataFrames?
Show answer and explanation
Correct answer: D. DataFrame.crossJoin()
The crossJoin() method on a DataFrame performs a cross join (Cartesian product) without requiring a join condition. This is the standard Spark API method for cross joins. The method is called as DataFrame.crossJoin(otherDataFrame).
Why the other options are wrong
- A. DataFrame.join() requires a join condition and is not used for cross joins.
- B. There is no standalone join() function for cross joins in Spark.
- C. There is no standalone crossJoin() function; it must be called on a DataFrame object.
- E. DataFrame.merge() does not exist in Spark's DataFrame API.
Question 12
Which of the following operations fails to return a DataFrame with no duplicate rows?
Show answer and explanation
Correct answer: E. DataFrame.drop_duplicates(subset = "all")
The drop_duplicates() method with subset="all" fails to remove duplicates because the subset parameter expects either None (default, considers all columns), a string column name, or a list of column names. The string value "all" is not a valid column name and will not match any actual columns in the DataFrame, resulting in no rows being dropped. Options A and B are valid PySpark/Scala methods that remove duplicates. Option C is the correct Python pandas syntax. Option D with subset=None correctly defaults to considering all columns for deduplication.
Why the other options are wrong
- A. dropDuplicates() is a valid Scala/PySpark method that returns a DataFrame with no duplicate rows.
- B. distinct() is a valid PySpark method that returns a DataFrame with unique rows.
- C. drop_duplicates() with no arguments is valid Python pandas syntax that removes duplicate rows.
- D. drop_duplicates(subset=None) correctly defaults to considering all columns and removes duplicates.
Question 13
Which of the following cluster configurations will induce the least network traffic during a shuffle operation? Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.
Show answer and explanation
Correct answer: C. Scenario 1
During shuffle operations in Spark, data is redistributed across executors based on partition keys. Network traffic is minimized when data movement stays within nodes rather than crossing the network. Scenario #1 has a single executor node, meaning all shuffle operations occur within that single machine's memory and storage, eliminating inter-node network traffic entirely. Scenarios #4, #5, and #6 distribute executors across multiple nodes, forcing shuffle data to traverse the network. Although all scenarios have equivalent total compute resources (100GB RAM and 200 cores), Scenario #1's single-node architecture ensures that shuffle operations never require data to be transmitted over the network, making it the optimal choice for minimizing shuffle network traffic.
Why the other options are wrong
- A. The number of partitions affects shuffle granularity but not the fundamental advantage of keeping data local within a single node.
- B. Scenario #5 with 4 nodes requires inter-node network communication during shuffle operations, generating network traffic.
- D. Scenario #4 with 2 nodes still requires network traffic between nodes during shuffle, unlike the single-node architecture of Scenario #1.
- E. Scenario #6 with 8 nodes maximizes data distribution across the network, resulting in the most inter-node shuffle traffic of all options.
Question 14
Which of the following identifies multiple narrow operations that are executed in sequence?
Show answer and explanation
Correct answer: C. Stage
A stage identifies multiple narrow operations that are executed in sequence. Stages are separated by shuffle boundaries; all operations within a stage are narrow transformations that can be pipelined together without shuffling. A stage is a group of tasks that can be executed together on partitions without data movement between partitions. Tasks are individual units of work, jobs are collections of stages, slots are execution resources, and executors are worker processes.
Why the other options are wrong
- A. A slot is a resource allocation unit for executing tasks, not a sequence of operations.
- B. A job is a sequence of stages triggered by an action, not a sequence of narrow operations.
- D. A task is a single unit of work on one partition, not multiple operations in sequence.
- E. An executor is a process that runs tasks, not a logical grouping of operations.
Question 15
Which of the following code blocks writes DataFrame storesDF to file path filePath as parquet and partitions by values in column division?
Show answer and explanation
Correct answer: D. storesDF.write.partitionBy("division").parquet(filePath)
The correct syntax for writing a DataFrame as parquet with partitioning is storesDF.write.partitionBy('division').parquet(filePath). The partitionBy() method specifies the partitioning column as a string, and parquet() is then called with the file path. This chains methods on the DataFrameWriter object returned by write.
Why the other options are wrong
- A. There is no .path() method in the Spark API; the format method (parquet) directly takes the file path.
- B. Parquet format should be specified with .parquet(), not as an option; there is no .path() method.
- C. While partitionBy and parquet syntax are correct, there is no .path() method.
- E. write is a property, not a method, so write() with parentheses is incorrect syntax.
Question 16
In what order should the below lines of code be run in order to write DataFrame storesDF to file path filePath as parquet and partition by values in column division? Lines of code: 1. .write() \ 2. .partitionBy("division") \ 3. .parquet(filePath) 4. .storesDF \ 5. .repartition("division") 6. .write \ 7. .path(filePath, "parquet")
Show answer and explanation
Correct answer: C. 4, 6, 2, 3
The correct sequence is 4, 6, 2, 3: storesDF.write.partitionBy("division").parquet(filePath). Line 4 references the DataFrame, line 6 initializes the write operation, line 2 specifies the partition column, and line 3 specifies the format and file path. This follows the fluent API pattern of Spark DataFrames.
Why the other options are wrong
- A. Line 1 (.write()) is incorrect because write is a property that returns a DataFrameWriter, not a method to be called with parentheses.
- B. Line 5 (repartition()) is unnecessary and incorrect in this context; partitionBy() handles the partitioning for the write operation itself.
- D. Line 1 (.write()) is syntactically incorrect; write should not have parentheses when accessed as a property.
- E. Line 7 (.path(filePath, "parquet")) does not exist as a valid method for specifying both path and format; the format-specific method parquet() should be used.
Question 17
Which of the following Spark properties is used to configure whether DataFrame partitions that do not meet a minimum size threshold are automatically coalesced into larger partitions during a shuffle?
Show answer and explanation
Correct answer: E. spark.sql.adaptive.coalescePartitions.enabled
The spark.sql.adaptive.coalescePartitions.enabled property controls whether partitions that are below a minimum size threshold are automatically coalesced together during shuffle operations. This is part of Adaptive Query Execution in Spark 3.x.
Why the other options are wrong
- A. spark.sql.shuffle.partitions sets the target number of partitions for shuffles, not automatic coalescing.
- B. spark.sql.autoBroadcastJoinThreshold configures the size threshold for broadcast joins.
- C. spark.sql.adaptive.skewJoin.enabled handles skewed join optimization, not partition coalescing.
- D. spark.sql.inMemoryColumnarStorage.batchSize controls batch sizes for columnar caching, not shuffle coalescing.
Question 18
Which of the following DataFrame operations is classified as a wide transformation?
Show answer and explanation
Correct answer: B. DataFrame.join()
DataFrame.join() is a wide transformation because it requires data from multiple partitions to be shuffled and redistributed across the cluster based on join keys. Wide transformations involve shuffling data between partitions, whereas narrow transformations operate within partitions independently.
Why the other options are wrong
- A. DataFrame.filter() is a narrow transformation that operates independently on each partition.
- C. DataFrame.select() is a narrow transformation that projects columns without requiring data movement between partitions.
- D. DataFrame.drop() is a narrow transformation that removes columns without shuffling data.
- E. DataFrame.union() is technically a narrow transformation as it only requires combining partition data without redistributing by keys.
Question 19
The default value of spark.sql.shuffle.partitions is 200. Which of the following describes what that means?
Show answer and explanation
Correct answer: E. By default, DataFrames will be split into 200 unique partitions when data is being shuffled.
The spark.sql.shuffle.partitions parameter specifically controls the number of partitions used when data is shuffled during operations like joins, aggregations, and other wide transformations. The default value of 200 means that whenever a shuffle occurs, the resulting data will be split into 200 partitions for parallelization, regardless of prior partitioning or DataFrame source.
Why the other options are wrong
- A. The parameter does not relate to filling executor memory.
- B. The parameter applies during shuffles, not just for new DataFrames.
- C. The parameter does not limit reading to first 200 partitions.
- D. The parameter specifically applies during shuffle operations, not to all DataFrames universally.
Question 20
Which of the following code blocks returns a new DataFrame from DataFrame storesDF where column modality is the constant string "PHYSICAL"? Assume DataFrame storesDF is the only defined language variable.
Show answer and explanation
Correct answer: C. storesDF.withColumn("modality", lit("PHYSICAL"))
The lit() function creates a literal column value in Spark. Option C correctly uses storesDF.withColumn("modality", lit("PHYSICAL")) to add a column with a constant string value. The lit() function is required to distinguish literal values from column references. Without lit(), the value would be interpreted as a column reference; col() expects an existing column name, not a constant string.
Why the other options are wrong
- A. PHYSICAL without quotes and without col() is treated as a variable reference, not a literal string.
- B. col("PHYSICAL") treats the string as a column name reference, not a literal constant value.
- D. StringType() is a type constructor, not a function for creating literal values.
- E. A bare string without lit() is not converted to a literal Column object; lit() is required.
Question 21
Which of the following code blocks writes DataFrame storesDF to file path filePath as parquet overwriting any existing files in that location?
Show answer and explanation
Correct answer: C. storesDF.write.mode(“overwrite”).parquet(filePath)
The correct syntax chains write (property), mode() (method), and parquet() (method) without parentheses after write. The pattern is storesDF.write.mode("overwrite").parquet(filePath), where write is accessed as a property without calling it, then methods are chained to specify the save mode and format.
Why the other options are wrong
- A. write() is called as a property, not a method, and the syntax doesn't support positional arguments like this.
- B. write() cannot be called with parentheses; it's a property, not a method.
- D. option() is used for configuration options, not format specification; parquet() is the correct format method.
- E. This syntax is missing the required format specification; parquet() must be called to specify the output format.
Question 22
Which of the following statements about the Spark DataFrame is true?
Show answer and explanation
Correct answer: D. A Spark DataFrame is a tabular data structure that is the most common Structured API in Spark.
A Spark DataFrame is a tabular data structure representing distributed data organized into named columns. It is the most commonly used Structured API in Spark for data processing. DataFrames are immutable (not mutable as stated in option A), are heavily used throughout Spark workflows (not rarely used), can be partitioned across a cluster, and while conceptually similar to Python/R data frames, they are optimized for distributed computing with different behavior and capabilities.
Why the other options are wrong
- A. Spark DataFrames are immutable regardless of collection status; immutability is a core design principle.
- B. DataFrames are the primary abstraction used throughout Spark for transformations and analysis, not just for import/export.
- C. Spark DataFrames are inherently distributed and partitioned across cluster nodes for parallel processing.
- E. While conceptually similar, Spark DataFrames differ significantly in implementation, optimization, and distributed behavior from Python or R data frames.
Question 23
Which of the following cluster configurations is most likely to experience an out-of-memory error in response to data skew in a single partition? Note: each configuration has roughly the same compute power using 100 GB of RAM and 200 cores.
Show answer and explanation
Correct answer: C. Scenario #6
Scenario #6 is most vulnerable to out-of-memory errors from data skew because it has 8 executor nodes with only 12.5 GB of RAM per executor. When a single partition becomes skewed and is processed by one executor, that executor must fit all the skewed data into its 12.5 GB memory allocation. In contrast, Scenario #1 has a single executor with 100 GB of RAM, providing much larger headroom for handling skewed partitions. Scenarios #4 and #5 have intermediate memory per executor (50 GB and 25 GB respectively), but still provide more per-executor memory than Scenario #6's 12.5 GB. Since data skew concentrates data in fewer partitions processed by individual executors, the configuration with the smallest per-executor memory (12.5 GB in Scenario #6) will be most likely to exhaust available memory and trigger an out-of-memory error.
Why the other options are wrong
- A. Scenario #4 has 50 GB per executor, which is 4 times larger than Scenario #6's 12.5 GB, making it far less susceptible to memory errors from skewed partitions.
- B. Scenario #5 has 25 GB per executor, which is twice as large as Scenario #6's 12.5 GB, providing more buffer against data skew-induced OOM errors.
- E. Scenario #1 has a single executor with 100 GB of RAM, the largest memory allocation per executor of all options, making it most resilient to data skew.
- D. Sufficient information exists in the exhibit to determine which configuration is most vulnerable based on per-executor memory constraints and data skew characteristics.
Question 24
Which of the following operations is most likely to result in a shuffle?
Show answer and explanation
Correct answer: A. DataFrame.join()
A join operation requires matching rows from two DataFrames based on join keys, which typically necessitates redistributing data across the network so that rows with the same key are co-located on the same executor. This redistribution is a shuffle. Filter, where, union, and drop operations are narrow transformations that do not require shuffling data between partitions.
Why the other options are wrong
- B. Filter is a narrow transformation that does not require shuffling.
- C. Union concatenates DataFrames without redistributing data.
- D. Where is equivalent to filter and is a narrow transformation without shuffling.
- E. Drop removes columns and is a narrow transformation without shuffling.
Question 25
Which of the following code blocks returns a new Data Frame from DataFrame storesDF with no duplicate rows?
Show answer and explanation
Correct answer: E. storesDF.dropDuplicates()
The correct Spark DataFrame method to remove duplicate rows is dropDuplicates(). This method returns a new DataFrame with duplicate rows removed while preserving the original data structure.
Why the other options are wrong
- A. removeDuplicates() is not a valid Spark DataFrame method.
- B. getDistinct() is not a valid Spark DataFrame method.
- C. duplicates.drop() is not valid syntax; duplicates is not an attribute.
- D. duplicates() is not a valid Spark DataFrame method.
Question 26
Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000?
Show answer and explanation
Correct answer: D. storesDF.filter(col("sqft") <= 25000)
The correct syntax uses filter() or where() with col() function to reference column names and comparison operators. Option D uses the correct syntax: storesDF.filter(col("sqft") <= 25000). The col() function is required to create a Column object for evaluation. Options A and E lack proper column reference syntax, option B uses unquoted column names without col(), and option C uses a string literal which would not evaluate the comparison correctly.
Why the other options are wrong
- A. Square bracket notation with unquoted column names is invalid; col() is required.
- B. Column names must be quoted strings when used without col(); unquoted names cause NameError.
- C. String literals are not evaluated as column references; col() must wrap the column name.
- E. While where() is functionally equivalent to filter(), the unquoted column names without col() make this invalid.
Question 27
Which of the following describes the difference between DataFrame.repartition(n) and DataFrame.coalesce(n)?
Show answer and explanation
Correct answer: A. DataFrame.repartition(n) will split a DataFrame into n number of new partitions with data distributed evenly. DataFrame.coalesce(n) will more quickly combine the existing partitions of a DataFrame but might result in an uneven distribution of data across the new partitions.
DataFrame.repartition(n) redistributes data across n partitions using a full shuffle, ensuring even distribution. DataFrame.coalesce(n) combines existing partitions more efficiently by avoiding a full shuffle when possible, but may result in uneven data distribution. Option A accurately describes both methods and their key trade-off between efficiency and data distribution uniformity.
Why the other options are wrong
- B. Incorrectly states repartition is more efficient; coalesce is actually more efficient due to minimal shuffling.
- C. Reverses the shuffling behavior; repartition uses full shuffle, coalesce minimizes shuffling.
- D. Incorrectly states repartition is less efficient; repartition is more thorough but coalesce is faster.
- E. Completely reverses the functions; repartition splits into partitions, coalesce combines them.
Question 28
Which of the following statements about Spark’s stability is incorrect?
Show answer and explanation
Correct answer: E. Spark will reassign the driver to a worker node if the driver’s node fails.
Spark does not automatically reassign the driver to a worker node if the driver node fails. The driver is a critical single point of failure in Spark applications. While Spark can handle worker node failures by rerunning tasks and recomputing cached data, driver failure results in application failure. The other statements accurately describe Spark's fault tolerance mechanisms.
Why the other options are wrong
- A. Spark is designed to support loss of any set of worker nodes through RDD lineage and task recomputation.
- B. Spark automatically reruns failed tasks on healthy nodes when worker nodes fail.
- C. Spark recomputes data that was cached on failed nodes using the original RDD lineage.
- D. Spark spills excess data to disk when memory capacity is exceeded, preventing out-of-memory errors.
Question 29
Which of the following code blocks fails to return a new DataFrame that is the result of an inner join between DataFrame storesDF and DataFrame employeesDF on column storeId and column employeeId?
Show answer and explanation
Correct answer: C. storesDF.join(employeesDF, storesDF("storeId") === employeesDF("storeId") and storesDF("employeeId") === employeesDF("employeeId"))
Option C fails because it uses the 'and' keyword in a Scala boolean expression, which is syntactically incorrect. Spark requires the '&&' operator for combining conditions in join expressions. All other options correctly join on multiple columns using either Seq() with column names or proper boolean expressions with && operators.
Why the other options are wrong
- A. This correctly joins on multiple columns using Seq() with col() expressions.
- B. This correctly joins on multiple columns by passing column names as strings in Seq().
- D. This correctly joins on multiple columns using Seq() with column names and explicit inner join type.
- E. This correctly joins on multiple columns using aliased DataFrames with proper && operator syntax.
Question 30
Which of the following code blocks returns a DataFrame sorted alphabetically based on column division?
Show answer and explanation
Correct answer: A. storesDF.sort("division")
The sort method with a column name as a string argument sorts the DataFrame alphabetically (ascending) by default. This is the simplest and most direct way to sort alphabetically by a column. Both sort and orderBy are valid methods, but sort with just a column name defaults to ascending order.
Why the other options are wrong
- B. orderBy with desc() sorts in descending order, not alphabetically ascending.
- C. This uses desc() which produces descending order rather than ascending alphabetical order.
- D. The syntax is incorrect; ascending - true is not valid Python or Scala syntax for Spark.
- E. This uses desc() which sorts in descending order rather than ascending alphabetical order.
Question 31
Which of the following code blocks returns the number of rows in DataFrame storesDF for each distinct combination of values in column division and column storeCategory?
Show answer and explanation
Correct answer: C. storesDF.groupBy(“division”, “storeCategory”).count()
The correct syntax for grouping by multiple columns in Spark is groupBy("division", "storeCategory").count(). This accepts string column names as separate arguments and returns the count for each distinct combination.
Why the other options are wrong
- A. Using Seq(col(...), col(...)) is valid but less idiomatic than passing string names directly.
- B. Using unquoted identifiers division and storeCategory without Seq or string wrapping is incorrect syntax.
- D. Chaining multiple groupBy calls does not group by multiple columns; the second groupBy overwrites the first.
- E. While Seq("division", "storeCategory") is syntactically valid, the idiomatic and most commonly used approach is option C.
Question 32
The below code block contains a logical error resulting in inefficiency. The code block is intended to efficiently perform a broadcast join of DataFrame storesDF and the much larger DataFrame employeesDF using key column storeId. Identify the logical error. Code block: storesDF.join(broadcast(employeesDF), "storeId")
Show answer and explanation
Correct answer: A. The larger DataFrame employeesDF is being broadcasted rather than the smaller DataFrame storesDF.
A broadcast join is efficient when the smaller DataFrame is broadcasted to all executor nodes. The code incorrectly broadcasts employeesDF, which is described as the larger DataFrame. The smaller DataFrame storesDF should be broadcasted instead to minimize network transfer and memory usage.
Why the other options are wrong
- B. Explicit broadcast() calls remain useful in Spark 3; automatic broadcasting relies on size thresholds that may not always apply.
- C. The broadcast() operation is correctly applied to the DataFrame itself, not the entire join expression.
- D. The spark.sql.autoBroadcastJoinThreshold property works by default; explicit broadcast() calls are additional optimization hints.
- E. Broadcasting both DataFrames is unnecessary and inefficient; only the smaller DataFrame should be broadcasted.
Question 33
Which of the following Spark properties is used to configure whether DataFrames found to be below a certain size threshold at runtime will be automatically broadcasted?
Show answer and explanation
Correct answer: B. spark.sql.autoBroadcastJoinThreshold
The spark.sql.autoBroadcastJoinThreshold property controls the maximum size in bytes for a table that will be broadcast to all worker nodes during a join operation. When a DataFrame is found to be below this threshold at runtime, Spark automatically broadcasts it rather than shuffling data, which improves join performance. This is a critical optimization in Spark SQL execution.
Why the other options are wrong
- A. spark.sql.broadcastTimeout controls the timeout duration for broadcast operations, not the size threshold for automatic broadcasting.
- C. spark.sql.shuffle.partitions sets the number of partitions to use when shuffling data, unrelated to broadcast thresholds.
- D. spark.sql.inMemoryColumnarStorage.batchSize controls batch sizing for columnar storage, not broadcast behavior.
- E. spark.sql.adaptive.localShuffleReader.enabled is related to adaptive query execution for local shuffle readers, not broadcasting.
Question 34
A Spark application has a 128 GB DataFrame A and a 1 GB DataFrame B. If a broadcast join were to be performed on these two DataFrames, which of the following describes which DataFrame should be broadcasted and why?
Show answer and explanation
Correct answer: D. DataFrame B should be broadcasted because it is smaller and will eliminate the need for the shuffling of DataFrame A.
DataFrame B (1 GB) should be broadcasted because it is smaller and can be efficiently copied to all executor nodes. Broadcasting the smaller DataFrame eliminates the need to shuffle the larger DataFrame A (128 GB) across the network. This minimizes data movement and network I/O, making the join operation significantly faster.
Why the other options are wrong
- A. The DataFrames are vastly different in size; broadcasting the larger one would be extremely inefficient.
- B. While B should be broadcasted, the reasoning is incomplete; it eliminates shuffling of A, not just itself.
- C. A should not be broadcasted; broadcasting a 128 GB DataFrame is impractical and defeats the purpose of a broadcast join.
- E. A is larger, not smaller, and broadcasting it would require excessive memory on all executor nodes.
Question 35
Which of the following is the most complete description of lazy evaluation?
Show answer and explanation
Correct answer: B. A process is lazily evaluated if its execution does not start until it is put into action by some type of trigger
Lazy evaluation is a programming model where computation is deferred until explicitly triggered. In Spark, transformations on DataFrames are lazily evaluated, they are not executed when defined but only when an action (like collect(), write(), or show()) forces the computation to run. This allows Spark to optimize the execution plan before running it.
Why the other options are wrong
- A. Option B provides a correct description.
- C. While displaying results is one trigger, lazy evaluation is more broadly triggered by any action, not just user display.
- D. Lazy evaluation is not tied to scheduled execution at specific times.
- E. Lazy evaluation is independent of compilation; it relates to runtime execution deferral.
Question 36
Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without inducing a shuffle?
Show answer and explanation
Correct answer: D. storesDF.coalesce(1)
coalesce(1) reduces the number of partitions by merging existing partitions, which is a narrow transformation and does not trigger a shuffle. It returns a new DataFrame with a single partition.
Why the other options are wrong
- A. intersect() must compare rows across partitions, which requires a shuffle, and it also needs another DataFrame as an argument.
- B. repartition() always performs a full shuffle to redistribute the data.
- C. union() requires another DataFrame as an argument, so storesDF.union() as written is not valid code.
- E. getNumPartitions() returns an integer count of partitions, not a new DataFrame.
Question 37
Which of the following statements about the Spark driver is true?
Show answer and explanation
Correct answer: D. Spark driver is responsible for scheduling the execution of data by various worker nodes in cluster mode.
The Spark driver is responsible for scheduling and coordinating the execution of tasks across worker nodes in a cluster. It converts the user's application into tasks and distributes them to executors on worker nodes. The driver is not horizontally scaled (it's a single point), not the coarsest level of execution hierarchy (the application is), not fault-tolerant in the sense described (failure of the driver fails the entire application), and works with multiple cluster managers, not just one.
Why the other options are wrong
- A. The driver cannot be horizontally scaled; it is a single coordinator process.
- B. The application is the coarsest level; the driver is within the execution hierarchy below it.
- C. The driver is not fault-tolerant; if it fails, the entire Spark application fails.
- E. Spark supports multiple cluster managers including YARN, Kubernetes, and Mesos, not just one.
Question 38
Which of the following operations can perform an outer join on two DataFrames?
Show answer and explanation
Correct answer: D. DataFrame.join()
DataFrame.join() supports multiple join types including outer joins by specifying the join type parameter. The method signature allows passing 'outer', 'left_outer', 'right_outer', or 'full_outer' as the join type argument to perform outer joins.
Why the other options are wrong
- A. crossJoin() performs cross joins, not outer joins.
- B. The standalone join() function is not the primary Spark API for joins; DataFrame.join() is standard.
- C. There is no outerJoin() method; join() is used with a join type parameter instead.
- E. DataFrame.merge() does not exist in Spark; join() is the correct operation.
Question 39
Which of the following code blocks returns a new DataFrame where column division from DataFrame storesDF has been replaced and renamed to column state and column managerName from DataFrame storesDF has been replaced and renamed to column managerFullName?
Show answer and explanation
Correct answer: A. storesDF.withColumnRenamed("division", "state") .withColumnRenamed("managerName", "managerFullName")
The withColumnRenamed() method renames a column by taking the old name and new name as arguments. Option A correctly chains two withColumnRenamed() calls to rename division to state and managerName to managerFullName. Options B and C use withColumn() which would replace values, not rename columns. Option D uses incorrect Seq syntax. Option E reverses the argument order, renaming in the wrong direction.
Why the other options are wrong
- B. withColumn() replaces column values, not rename columns; it would create wrong transformations.
- C. withColumn() with col() copies values rather than renaming; the original column names persist.
- D. withColumnRenamed() does not accept Seq collections; it requires individual old/new name pairs.
- E. The arguments are reversed; this would rename state to division and managerFullName to managerName, opposite of the requirement.
Question 40
Which of the following code blocks extracts the value for column sqft from the first row of DataFrame storesDF?
Show answer and explanation
Correct answer: E. storesDF.first().sqft
The first() method returns the first Row object from a DataFrame. In PySpark, you can access column values from a Row using dot notation with the column name as an attribute. The syntax storesDF.first().sqft retrieves the sqft value from the first row. This is the most direct and Pythonic way to access a single column value from a single row.
Why the other options are wrong
- A. first() returns a Row object, but indexing with col() is not valid syntax; use bracket notation or dot notation.
- B. DataFrame indexing with [0] is not supported in PySpark; you must use first() or collect().
- C. collect() returns a list requiring [0] indexing, and the syntax collect(l) with lowercase 'l' is a typo.
- D. first is a method and must be called with parentheses as first(), not accessed as a property.
Question 41
Which of the following code blocks returns a 10 percent sample of rows from DataFrame storesDF with replacement?
Show answer and explanation
Correct answer: B. storesDF.sample(true, fraction = 0.1)
The sample method takes a boolean for sampling with replacement and a fraction parameter. Setting the first parameter to true enables sampling with replacement, and fraction = 0.1 specifies a 10 percent sample. This returns approximately 10 percent of the rows from the original DataFrame.
Why the other options are wrong
- A. This samples with replacement but doesn't specify a fraction, so it doesn't guarantee a 10 percent sample.
- C. A fraction of 0.15 returns a 15 percent sample, not 10 percent.
- D. sampleBy is a stratified sampling method used with a column specification, not a simple random sample.
- E. Setting the first parameter to false disables sampling with replacement, contrary to the requirement.
Question 42
Which of the following code blocks writes DataFrame storesDF to file path filePath as JSON?
Show answer and explanation
Correct answer: B. storesDF.write.json(filePath)
The correct syntax for writing a DataFrame to JSON is storesDF.write.json(filePath). The write property returns a DataFrameWriter object, and json() is a method that accepts the file path as an argument and performs the write operation.
Why the other options are wrong
- A. The option() method is used for format-specific options, not to specify the format itself; the format should be specified via a dedicated method like json().
- C. The path() method is not the correct way to write DataFrames; the format-specific methods like json() handle both format and path.
- D. write is a property that returns a DataFrameWriter, not a callable method; parentheses should not follow it directly.
- E. write returns a DataFrameWriter object directly and should not have parentheses; calling json() on the writer is the correct approach.
Question 43
Which of the following code blocks will always return a new 4-partition DataFrame from the 8-partition DataFrame storesDF without inducing a shuffle?
Show answer and explanation
Correct answer: C. storesDF.coalesce(4)
The coalesce() method reduces the number of partitions without inducing a shuffle. It merges existing partitions without redistributing data across the cluster. Going from 8 partitions to 4 partitions using coalesce(4) simply combines adjacent partitions efficiently.
Why the other options are wrong
- A. repartition() with a column argument forces a shuffle operation to redistribute data.
- B. repartition() without arguments is invalid syntax.
- D. repartition(4) always induces a shuffle, even when reducing partitions.
- E. coalesce without arguments is invalid syntax; a partition count must be specified.
Question 44
Which of the following best describes the similarities and differences between the MEMORY_ONLY storage level and the MEMORY_AND_DISK storage level?
Show answer and explanation
Correct answer: D. The MEMORY_ONLY storage level will store as much data as possible in memory and will recompute any data that does not fit in memory as it's called. The MEMORY_AND_DISK storage level will store as much data as possible in memory and will store any data that does on fit in memory on disk and read it as it's called.
MEMORY_ONLY stores as much data as possible in memory and recomputes any data that does not fit when needed, while MEMORY_AND_DISK stores as much data as possible in memory and writes overflow to disk, reading it back when needed. The key difference is that MEMORY_ONLY uses recomputation for data that exceeds memory, whereas MEMORY_AND_DISK uses disk storage for overflow. Both operate on the local executor's memory and disk, not across multiple cluster nodes by default.
Why the other options are wrong
- A. This reverses the behavior of the two storage levels.
- B. Storage levels do not inherently involve two cluster nodes; they operate on individual executors.
- C. This also reverses the behavior and incorrectly specifies two cluster nodes.
- E. MEMORY_AND_DISK does not split data 50/50 between memory and disk; it stores all data in memory first, then spills remainder to disk.
Question 45
In what order should the below lines of code be run in order to read a JSON file at the file path filePath into a DataFrame with the specified schema schema? Lines of code: 1. .json(filePath, schema = schema) 2. .storesDF 3. .spark \ 4. .read() \ 5. .read \ 6. .json(filePath, format = schema)
Show answer and explanation
Correct answer: C. 3, 5, 1
The correct sequence is 3, 5, 1: spark.read.json(filePath, schema = schema). Line 3 references the Spark session, line 5 initializes the read operation (without parentheses, as read is a property), and line 1 specifies the JSON format with the schema parameter. This follows the standard Spark DataFrame read API.
Why the other options are wrong
- A. Line 6 (.json(filePath, format = schema)) uses an incorrect parameter name; the parameter should be schema, not format.
- B. Line 2 references storesDF which is the target variable name, not part of the read operation; line 4 incorrectly adds parentheses to read.
- D. Line 2 is incorrect as it references the variable assignment target; line 4 incorrectly calls read as a method with parentheses.
- E. Line 4 (.read()) is incorrect because read is a property, not a method, and should not have parentheses.
Question 46
Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 AND the value in column customerSatisfaction is greater than or equal to 30?
Show answer and explanation
Correct answer: D. storesDF.filter(col("sqft") <= 25000 & col("customerSatisfaction") >= 30)
In PySpark, the filter() method requires boolean expressions using bitwise operators. The ampersand (&) is the correct operator for AND conditions between Column objects. Option D correctly uses col() to reference columns and & to combine conditions. Python's 'and' keyword (Option A) does not work with Column expressions; 'or' (Option B) is the wrong logical operator; Option C has syntax errors with missing parenthesis; Option E has malformed syntax with an incorrect opening parenthesis.
Why the other options are wrong
- A. Python 'and' keyword cannot be used with Spark Column objects; only bitwise operators work.
- B. Uses 'or' instead of 'and', which would return rows matching either condition rather than both.
- C. Contains syntax errors with mismatched parentheses and missing col() wrapper.
- E. Has malformed syntax with an incorrect opening parenthesis and missing proper Column wrapping.
Question 47
Which of the following will cause a Spark job to fail?
Show answer and explanation
Correct answer: E. A failed driver node.
A failed driver node will cause a Spark job to fail because the driver is responsible for coordinating the entire application. If the driver fails, it cannot schedule tasks, track their progress, or manage the application state, resulting in complete application failure. Pulling data onto the driver does not inherently cause failure (though it can cause memory issues), caching data larger than executor memory can be handled through spilling, and data spilling to disk is a normal operation. Even a failed worker node can be recovered through task recomputation.
Why the other options are wrong
- A. Never pulling data onto the driver does not cause a job to fail; in fact, it is often the desired pattern.
- B. Caching data larger than executor memory does not fail the job; Spark handles this through storage levels and recomputation.
- C. Data spilling from memory to disk is a normal and expected operation in Spark, not a failure condition.
- D. A failed worker node does not necessarily fail the job; Spark can recompute lost partitions on other nodes.
Question 48
Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?
Show answer and explanation
Correct answer: D. [assessPerformance(row) for row in storesDF.collect()]
The collect() method materializes all rows from the DataFrame into a Python list, which can then be iterated over in a list comprehension with the assessPerformance() function applied to each row. Option A uses take(3) which only processes 3 rows, B fails to pass row as an argument, C uses incorrect syntax with apply() and lambda, and E attempts to iterate directly over the DataFrame which is not iterable.
Why the other options are wrong
- A. take(3) only processes the first 3 rows, not all rows
- B. assessPerformance() is called without passing the row argument
- C. apply() is a pandas method not available on Spark DataFrames, and lambda syntax is incorrect
- E. DataFrames are not directly iterable; you must call collect() first to get a Python list
Question 49
Which of the following code blocks returns a DataFrame containing only the rows from DataFrame storesDF where the value in column sqft is less than or equal to 25,000 OR the value in column customerSatisfaction is greater than or equal to 30?
Show answer and explanation
Correct answer: B. storesDF.filter(col("sqft") <= 25000 | col("customerSatisfaction") >= 30)
In Spark SQL, the pipe operator (|) represents OR logic for combining conditions in filter expressions. Option B correctly uses col() to reference columns, quotes column names, and employs the | operator for OR logic. Python's 'or' keyword cannot be used in Spark Column expressions, and 'and' is replaced by & for AND operations. Unquoted column names without col() are invalid syntax.
Why the other options are wrong
- A. Python's 'and' keyword is not valid in Spark Column expressions; use & for AND operations.
- C. Column names must be quoted strings within col(); unquoted names cause NameError.
- D. Column names without col() and quotes are not valid; they lack proper Column object construction.
- E. Python's 'or' keyword cannot be used in Spark Column expressions; use | for OR operations.
Question 50
Which of the following sets of DataFrame methods will both return a new DataFrame only containing rows that meet a specified logical condition?
Show answer and explanation
Correct answer: C. filter(), where()
In Spark DataFrames, both filter() and where() are equivalent methods that return a new DataFrame containing only rows meeting a specified logical condition. They are aliases for the same functionality. Option C correctly identifies these two methods, while other options include methods that either drop rows (drop), select specific columns (select), rather than filter based on logical conditions.
Why the other options are wrong
- A. drop() removes columns or rows by position, not by logical condition; it does not return filtered rows based on conditions.
- B. select() returns specific columns from a DataFrame, not rows filtered by logical conditions.
- D. select() returns specific columns, not filtered rows; where() is correct but paired with the wrong method.
- E. drop() is used for removing data, not filtering by logical conditions; it does not return conditionally filtered rows.
Question 51
Which of the following describes the difference between cluster and client execution modes?
Show answer and explanation
Correct answer: A. The cluster execution mode runs the driver on a worker node within a cluster, while the client execution mode runs the driver on the client machine (also known as a gateway machine or edge node).
In cluster mode, the driver runs on a worker node within the cluster and manages the application remotely. In client mode, the driver runs on the client machine (an external gateway or edge node) and communicates with the cluster. This distinction determines where the driver process executes and how it interacts with worker executors.
Why the other options are wrong
- B. Both modes can run on local or cloud clusters; the distinction is not about local vs cloud infrastructure.
- C. Both modes distribute executors across worker nodes; the difference is specifically about driver placement.
- D. This reverses the correct placement: cluster mode runs the driver on a worker node, not client mode.
- E. Client mode does not require the cluster to be remote or unconfigurable; it only means the driver is external to the cluster.
Question 52
Which of the following types of processes induces a stage boundary?
Show answer and explanation
Correct answer: A. Shuffle
A shuffle operation induces a stage boundary in Spark. Shuffles occur when data must be rearranged across partitions (such as in joins, groupBy, or aggregations) and require breaking the execution into separate stages. The stage after a shuffle cannot be pipelined with the stage before it. Caching, executor failure, job delegation, and application failure do not define stage boundaries.
Why the other options are wrong
- B. Caching does not induce a stage boundary; it allows data to be reused across actions.
- C. Executor failure is a runtime failure, not an execution planning concept that creates stages.
- D. Job delegation is not a Spark execution concept related to stages.
- E. Application failure terminates execution but does not define stage boundaries.
Question 53
Which of the following code blocks returns the first 3 rows of DataFrame storesDF?
Show answer and explanation
Correct answer: C. storesDF.take(3)
The take() method returns the first n rows of a DataFrame as an array. It is the correct Spark API for retrieving a specific number of initial rows. head() also works but take() is the primary method for this operation in Spark DataFrames.
Why the other options are wrong
- A. top_n() is not a valid DataFrame method in Spark.
- B. n() is not a valid DataFrame method in Spark.
- D. While head(3) does return the first 3 rows, take(3) is the more standard Spark API for this operation.
- E. collect() retrieves all rows into memory; collect(3) is not valid syntax.
Question 54
Which of the following Spark properties is used to configure whether skewed partitions are automatically detected and subdivided into smaller partitions when joining two DataFrames together?
Show answer and explanation
Correct answer: A. spark.sql.adaptive.skewedJoin.enabled
spark.sql.adaptive.skewedJoin.enabled controls whether Spark automatically detects skewed partitions during join operations and subdivides them into smaller partitions to improve performance. This adaptive optimization helps prevent slow nodes from becoming bottlenecks.
Why the other options are wrong
- B. spark.sql.adaptive.coalescePartitions.enable controls whether small partitions are automatically coalesced together, not whether skewed partitions are subdivided.
- C. spark.sql.adaptive.skewHints.enabled is not a standard Spark property for controlling skew detection in joins.
- D. spark.sql.shuffle.partitions sets the default number of partitions for shuffle operations, not skew detection behavior.
- E. spark.sql.shuffle.skewHints.enabled is not a standard Spark property; the correct property uses the adaptive prefix.
Question 55
Which of the following code blocks applies the function assessPerformance() to each row of DataFrame storesDF?
Show answer and explanation
Correct answer: E. storesDF.collect.foreach(row => assessPerformance(row))
The foreach() method iterates over each row in a collection and applies a function without returning a value. The syntax storesDF.collect.foreach(row => assessPerformance(row)) correctly collects all rows and applies the function to each one using the arrow function syntax.
Why the other options are wrong
- A. This syntax is incorrect; assessPerformance(row) would be evaluated immediately rather than passed as a lambda function.
- B. apply() is not the correct method for iteration; foreach is required for side effects.
- C. The correct method is foreach, not apply, and the syntax is incomplete.
- D. map() returns a transformed collection; foreach is needed when the function has side effects and no return value is needed.
Question 56
Which of the following DataFrame operations is classified as a transformation?
Show answer and explanation
Correct answer: A. DataFrame.select()
DataFrame.select() is a transformation because it returns a new DataFrame without triggering computation. It is lazy and defines a step in the execution plan. Options B, C, D, and E are all actions that trigger evaluation and return results to the driver or console rather than returning a new DataFrame.
Why the other options are wrong
- B. count() is an action that returns a single integer value, triggering full evaluation.
- C. show() is an action that displays results to the console, triggering evaluation.
- D. first() is an action that returns the first row, triggering evaluation.
- E. collect() is an action that returns all rows to the driver, triggering full evaluation.
Question 57
Which of the following code blocks writes DataFrame storesDF to file path filePath as CSV?
Show answer and explanation
Correct answer: C. storesDF.write.csv(filePath)
The correct syntax for writing a DataFrame to CSV is storesDF.write.csv(filePath). The write property returns a DataFrameWriter object, and csv() is called as a method (not with parentheses like a function call). This is the standard Spark API for writing DataFrames.
Why the other options are wrong
- A. write is a property, not a method, so write() with parentheses is incorrect syntax.
- B. write.csv(filePath) is the correct pattern; passing filePath to write() is incorrect.
- D. CSV format should be specified with .csv(), not as an option; there is no .path() method.
- E. There is no .path() method in DataFrameWriter; the format method (csv, parquet, etc.) includes the file path.
Question 58
Which of the following code blocks returns all the rows from DataFrame storesDF?
Show answer and explanation
Correct answer: B. storesDF.collect()
The collect() method returns all rows from a DataFrame as an array to the driver program. head() returns only the first few rows, count() returns the number of rows as an integer, take() returns a limited number of rows, and show() displays rows in a formatted table without returning them as a complete collection.
Why the other options are wrong
- A. head() returns only the first few rows (default 20), not all rows
- C. count() returns an integer representing the row count, not the rows themselves
- D. take() returns only a limited number of rows specified as an argument
- E. show() prints rows to the console in formatted output but does not return them
Question 59
Which of the following operations can be used to create a new DataFrame that has 12 partitions from an original DataFrame df that has 8 partitions?
Show answer and explanation
Correct answer: A. df.repartition(12)
df.repartition(12) increases the number of partitions from 8 to 12 by performing a full shuffle of the data across the cluster. Repartition can increase or decrease the number of partitions and redistributes data evenly across the new partition count.
Why the other options are wrong
- B. df.cache() stores a DataFrame in memory but does not change the number of partitions.
- C. df.partitionBy(1.5) is not a valid method; partitionBy expects column names, not numeric multipliers.
- D. df.coalesce(12) reduces partitions when the argument is smaller than the current count; it cannot increase from 8 to 12 partitions efficiently.
- E. df.partitionBy(12) is used in write operations to organize output files, not to repartition an existing DataFrame.
Question 60
Which of the following storage levels should be used to store as much data as possible in memory on two cluster nodes while storing any data that does not fit in memory on disk to be read in when needed?
Show answer and explanation
Correct answer: D. MEMORY_AND_DISK_2
MEMORY_AND_DISK_2 stores data in memory with replication factor 2, and any data exceeding available memory is automatically spilled to disk. The "_2" suffix indicates two-node replication, while MEMORY_AND_DISK provides the memory-with-overflow-to-disk functionality required.
Why the other options are wrong
- A. MEMORY_ONLY_2 replicates data across two nodes but does not spill to disk; data that doesn't fit in memory is lost.
- B. MEMORY_AND_DISK_SER uses serialization and spills to disk, but does not specify the two-node replication requirement.
- C. MEMORY_AND_DISK provides memory with disk spillover but without the two-node replication specified in the requirement.
- E. MEMORY_ONLY keeps data only in memory without disk spillover; data exceeding memory capacity is lost.
Question 61
Which of the following statements about slots is incorrect?
Show answer and explanation
Correct answer: E. There must be at least as many slots as there are executors.
The incorrect statement is that there must be at least as many slots as executors. The relationship is actually the reverse: each executor contains multiple slots, so there are typically more slots than executors. A single executor can have multiple slots for parallel task execution within that executor.
Why the other options are wrong
- A. Slots are indeed the most granular level of execution in the hierarchy.
- B. Slots are correctly resources for parallelization within an executor.
- C. Tasks are correctly assigned to slots for computation.
- D. There can correctly be more slots than tasks when not all slots are in use.
Question 62
Which of the following Spark properties is used to configure the maximum size of an automatically broadcasted DataFrame when performing a join?
Show answer and explanation
Correct answer: B. spark.sql.autoBroadcastJoinThreshold
spark.sql.autoBroadcastJoinThreshold configures the maximum size in bytes of a DataFrame that will be automatically broadcast to all nodes during a join operation. When a DataFrame is smaller than this threshold, Spark automatically broadcasts it rather than performing a shuffle join.
Why the other options are wrong
- A. spark.sql.broadcastTimeout controls the timeout duration for broadcast operations, not the size threshold for automatic broadcasting.
- C. spark.sql.shuffle.partitions configures the number of partitions used in shuffle operations, not the broadcast size threshold.
- D. spark.sql.inMemoryColumnarStorage.batchSize configures batch sizes for columnar storage, not broadcast thresholds.
- E. spark.sql.adaptive.skewedJoin.enabled controls whether skewed joins are detected and handled, not broadcast sizing.
Question 63
Which of the following operations returns a GroupedData object?
Show answer and explanation
Correct answer: D. DataFrame.groupBy()
The groupBy() method (with lowercase 'B' and camelCase) is the correct PySpark DataFrame method that returns a GroupedData object. This object represents a grouped DataFrame and enables subsequent aggregation operations. The method name is case-sensitive, and the correct form is groupBy(), not GroupBy() or other variations.
Why the other options are wrong
- A. GroupBy() with capital 'G' is not the correct method name; PySpark uses camelCase with lowercase first letter.
- B. cubed() is not a valid PySpark method related to grouping.
- C. group() is not a valid PySpark DataFrame method.
- E. grouping_id() is a window function for computing grouping IDs in complex grouping scenarios, not the main grouping method.
Question 64
Which of the following describes a partition?
Show answer and explanation
Correct answer: E. A partition is a collection of rows of data that fit on a single machine in a cluster.
A partition is a collection of rows of data that fit on a single machine in a cluster. Partitions are the fundamental unit of parallelism in Spark; each partition can be processed independently by a separate task. While partitions are stored and processed on executors within worker nodes, the definition emphasizes that a partition is a logical collection of rows that resides on one machine. The size of a partition is not automatically determined by executor memory or node capacity alone.
Why the other options are wrong
- A. Partitions are not sized by what fits in a single executor; executors can process multiple partitions or portions thereof.
- B. Partitions are not automatically sized; they are explicitly created and their size depends on configuration and data distribution.
- C. While partitions reside on worker nodes, this is less precise than describing them as collections of rows on a single machine.
- D. Partitions are not portions of a Spark application made up of jobs; they are units of data distribution.
Question 65
Which of the following statements about Spark jobs is incorrect?
Show answer and explanation
Correct answer: D. There is no way to monitor the progress of a job.
Jobs can be monitored for progress through the Spark UI, logs, and programmatic APIs. Jobs are broken into stages based on shuffle boundaries, contain multiple tasks when DataFrames have multiple partitions, and are triggered by action calls on DataFrames. The statement that there is no way to monitor job progress is factually incorrect.
Why the other options are wrong
- A. Jobs are correctly broken down into stages.
- B. Multiple tasks within a job are correctly created for multi-partition DataFrames.
- C. Jobs are correctly divided based on action calls.
- E. While language variables may influence job definitions, the primary division is by action calls and shuffle points.
Question 66
Which of the following code blocks will most quickly return an approximation for the number of distinct values in column division in DataFrame storesDF?
Show answer and explanation
Correct answer: C. storesDF.agg(approx_count_distinct(col("division"), 0.15).alias("divisionDistinct"))
The approx_count_distinct() function accepts an accuracy parameter (second argument) that controls the accuracy-speed tradeoff. A larger accuracy parameter value (closer to 1.0) means faster execution but less accuracy, while smaller values improve accuracy at the cost of speed. With accuracy set to 0.15, the function achieves a good balance for quick approximation. Option A uses the default accuracy. Option B uses 0.01 for very high accuracy but slower performance. Option D with 0.0 is invalid. Option E uses 0.05 for higher accuracy than needed.
Why the other options are wrong
- A. Uses default accuracy parameter, providing moderate speed but not the fastest approximation.
- B. Uses 0.01 accuracy, which prioritizes correctness over speed, making it slower than necessary.
- D. An accuracy value of 0.0 is not valid for the approx_count_distinct function.
- E. Uses 0.05 accuracy, which provides higher accuracy than the fastest approximation option.
Question 67
Which of the following operations can be used to return a DataFrame with no duplicate rows? Please select the most complete answer.
Show answer and explanation
Correct answer: E. DataFrame.dropDuplicates(), DataFrame.distinct() and DataFrame.drop_duplicates()
Spark provides three equivalent methods for removing duplicate rows: distinct(), dropDuplicates(), and drop_duplicates(). All three return a new DataFrame with no duplicate rows. The most complete answer includes all available options across Spark's different API patterns, including the Python-style snake_case naming convention.
Why the other options are wrong
- A. distinct() works but is incomplete; other equivalent methods exist.
- B. Both methods work but drop_duplicates() is omitted from this answer.
- C. dropDuplicates() works but distinct() and drop_duplicates() are omitted.
- D. drop_duplicates() works but distinct() and dropDuplicates() are omitted; this is incomplete.
Question 68
Of the following, which is the coarsest level in the Spark execution hierarchy?
Show answer and explanation
Correct answer: B. Job
The Spark execution hierarchy from coarsest to finest is: Job → Stage → Task → Slot. A job is the coarsest level, representing the entire action triggered by a Spark operation. Jobs are divided into stages, stages into tasks, and tasks are assigned to slots within executors.
Why the other options are wrong
- A. Slots are the finest (most granular) level, not the coarsest.
- C. Tasks are finer-grained than jobs; multiple tasks compose stages which compose jobs.
- D. Stages are finer-grained than jobs; a job contains multiple stages.
- E. Executors are infrastructure components that contain slots, not part of the logical execution hierarchy.
Question 69
Which of the following cluster configurations is most likely to experience delays due to garbage collection of a large Dataframe? Note: each configuration has roughly the same compute power using 100GB of RAM and 200 cores.
Show answer and explanation
Correct answer: D. Scenario #1
Scenario #1 has a single executor with 100 GB of memory managing a large DataFrame. When garbage collection runs on a large heap, it causes significant pause times that delay task execution. A single large executor is more susceptible to these pauses because the entire GC event affects all task processing on that one executor. In contrast, Scenarios #4, #5, and #6 distribute the workload across multiple smaller executors (2, 4, and 8 respectively), allowing garbage collection events to be parallelized and localized to individual executors with smaller heaps. Smaller heap sizes result in shorter GC pause times, and having multiple executors means other executors continue processing while one is undergoing GC. Therefore, the centralized single-executor architecture of Scenario #1 will experience the most severe delays from DataFrame garbage collection.
Why the other options are wrong
- A. The answer can be determined from the cluster configurations shown; all have comparable compute power but different executor distribution.
- B. Scenario #5 distributes memory across 4 executors (25 GB each), resulting in smaller heaps and faster GC cycles compared to Scenario #1's single 100 GB heap.
- C. Scenario #4 uses 2 executors with 50 GB each, still resulting in smaller individual heaps and faster garbage collection than the single 100 GB executor.
- E. Scenario #2 is not shown in the options; however, if referring to a similar multi-executor configuration, it would have better GC performance than Scenario #1.
Question 70
Which of the following code blocks uses SQL to return a new DataFrame containing column storeId and column managerName from a table created from DataFrame storesDF?
Show answer and explanation
Correct answer: D. storesDF.createOrReplaceTempView("stores") spark.sql("SELECT storeId, managerName FROM stores")
To use SQL with a DataFrame, you must first create a temporary view using createOrReplaceTempView() with a string argument specifying the view name. The method createOrReplaceTempView("stores") registers storesDF as a temporary table named "stores". Then spark.sql() is called to execute the SQL query against that registered view. The spark.sql() method returns a new DataFrame containing the selected columns storeId and managerName.
Why the other options are wrong
- A. createOrReplaceTempView() requires a string argument for the view name and is missing that argument; additionally, spark.sql() call is not captured or assigned.
- B. DataFrames do not have a query() method; SQL queries must be executed through spark.sql().
- C. spark.createOrReplaceTempView() is incorrect syntax; the method belongs to the DataFrame object, not the spark session, and storesDF.sql() does not exist.
- E. DataFrames do not have a query() method; only spark.sql() can execute SQL queries in Spark.
Question 71
Which of the following code blocks returns a collection of summary statistics for all columns in DataFrame storesDF?
Show answer and explanation
Correct answer: E. storesDF.describe()
The describe() method without arguments returns a DataFrame containing summary statistics (count, mean, stddev, min, max) for all numeric columns in the DataFrame. This is the standard approach for obtaining a comprehensive statistical summary of all columns. The summary() method does not exist in PySpark, and the other options use incorrect method names or invalid parameter syntax.
Why the other options are wrong
- A. The summary() method is not a standard PySpark DataFrame method.
- B. The describe() method does not accept an all=True parameter; it returns all statistics by default.
- C. While describe() accepts string arguments, the correct invocation without arguments returns all statistics.
- D. The summary() method is not a valid PySpark DataFrame method.
Question 72
Of the following situations, in which will it be most advantageous to store DataFrame df at the MEMORY_AND_DISK storage level rather than the MEMORY_ONLY storage level?
Show answer and explanation
Correct answer: D. When it’s faster to read all the computed data in DataFrame df that cannot fit into memory from disk rather than recompute it based on its logical plan.
MEMORY_AND_DISK is advantageous when reading spilled data from disk is faster than recomputing it from the logical plan. When DataFrame data exceeds available memory, MEMORY_AND_DISK caches what fits in memory and spills the rest to disk, avoiding expensive recomputation. This is most beneficial when disk I/O is faster than the computational cost of regenerating the data.
Why the other options are wrong
- A. If all data fits in memory, MEMORY_ONLY is preferable and sufficient; MEMORY_AND_DISK adds no benefit.
- B. When it's faster to recompute than read from disk, caching to disk is not advantageous and MEMORY_ONLY is better.
- C. This describes a scenario where caching is inefficient; recomputation would be the better strategy without caching.
- E. MEMORY_ONLY is faster for data that fits in memory, but MEMORY_AND_DISK is superior when data exceeds memory capacity.
Question 73
QUESTION NO: 75 Which of the following code blocks returns a DataFrame where column divisionDistinct is the approximate number of distinct values in column division from DataFrame storesDF?
Show answer and explanation
Correct answer: A. storesDF.withColumn("divisionDistinct", approx_count_distinct(col("division")))
The approx_count_distinct function is used as an aggregation function within withColumn to add a new column containing the approximate count of distinct values. Option A correctly applies approx_count_distinct(col("division")) as a column expression. Option C uses agg which returns a single row aggregation rather than adding a column to each row. Options B, D, and E use incorrect syntax or method chaining that doesn't work with the approx_count_distinct function.
Why the other options are wrong
- B. The agg method doesn't accept approx_count_distinct as a column method in this syntax.
- C. Using agg returns an aggregated result with a single row, not a DataFrame with an added column.
- D. approx_count_distinct() cannot be called as a method on a column object.
- E. The syntax is incorrect; approx_count_distinct is not a column method.
Question 74
Which of the following code blocks returns a 15 percent sample of rows from DataFrame storesDF without replacement?
Show answer and explanation
Correct answer: B. storesDF.sample(fraction = 0.15)
The sample() method returns a random sample of rows from a DataFrame. By default, sampling is done without replacement. The syntax storesDF.sample(fraction=0.15) returns approximately 15 percent of rows without replacement. When withReplacement is not specified, it defaults to False, which means sampling without replacement.
Why the other options are wrong
- A. sample(True, fraction=0.15) would enable sampling with replacement, not without.
- C. sampleBy() is a method for stratified sampling by a specific column, not for simple random sampling.
- D. sample(fraction=0.10) returns a 10 percent sample, not 15 percent.
- E. sample() with no arguments returns an arbitrary sample, not a specific 15 percent fraction.
Question 75
Which of the following object types cannot be contained within a column of a Spark DataFrame?
Show answer and explanation
Correct answer: A. DataFrame
A DataFrame cannot contain another DataFrame as a column value. Spark DataFrames support primitive types, strings, arrays, maps, structs, null values, and complex nested types, but not nested DataFrames. Columns contain individual values or collections of values, not tabular data structures.
Why the other options are wrong
- B. String is a basic supported data type in Spark DataFrames.
- C. Array is a supported complex data type that can be stored in DataFrame columns.
- D. null is a valid value in DataFrame columns, representing missing or undefined data.
- E. Vector is a supported data type in Spark MLlib and can be stored in DataFrame columns.
Question 76
Which of the following code blocks returns a 15 percent sample of rows from DataFrame storesDF without replacement?
Show answer and explanation
Correct answer: E. storesDF.sample(fraction = 0.15)
The sample() method with fraction=0.15 returns a 15 percent sample of rows from the DataFrame without replacement by default. The fraction parameter specifies the sampling ratio, and withReplacement defaults to False (without replacement). Option A uses 0.10 (10% instead of 15%). Option B uses a non-existent sampleBy() method. Option C includes withReplacement=True as the first positional argument, enabling sampling with replacement. Option D calls sample() without parameters, returning a random sample with unspecified size.
Why the other options are wrong
- A. Uses fraction=0.10, which returns only 10 percent of rows, not 15 percent.
- B. The sampleBy() method does not exist in PySpark's DataFrame API.
- C. The first argument True enables sampling with replacement, which violates the requirement for sampling without replacement.
- D. Calling sample() without parameters returns a random sample of unspecified size, not a precise 15 percent sample.
Question 77
Which of the following pairs of arguments cannot be used in DataFrame.join() to perform an inner join on two DataFrames, named and aliased with "a" and "b" respectively, to specify two key columns?
Show answer and explanation
Correct answer: B. on = [col("column1"), col("column2")]
Option B with on = [col("column1"), col("column2")] cannot be used because it specifies only column names without establishing the relationship between the two DataFrames' columns. The join operation needs to know which columns from each DataFrame correspond to each other; passing unqualified column names ambiguously references columns that may exist in both DataFrames.
Why the other options are wrong
- A. Explicit equality expressions like a.column1 == b.column1 properly specify multi-column join keys.
- C. Qualified column references with DataFrame aliases work correctly for multi-column joins.
- D. This option is incorrect; Option B cannot be used for the reasons stated above.
- E. String column names work when they unambiguously exist in both DataFrames with the same names.
Question 78
Which of the following operations can be used to return a new DataFrame from DataFrame storesDF without columns that are specified by name?
Show answer and explanation
Correct answer: C. storesDF.drop()
The drop() method removes specified columns by name and returns a new DataFrame. The filter() and where() methods filter rows based on conditions, not remove columns. The subset() and dropColumn() methods do not exist in Spark's DataFrame API.
Why the other options are wrong
- A. filter() removes rows based on conditions, not columns.
- B. select() chooses specific columns to keep; it doesn't remove named columns.
- D. subset() is not a valid Spark DataFrame method.
- E. dropColumn() is not a valid Spark DataFrame method; drop() is the correct method.
Question 79
Which of the following code blocks fails to return a DataFrame reverse sorted alphabetically based on column division?
Show answer and explanation
Correct answer: C. storesDF.orderBy(col("division").asc())
The code storesDF.orderBy(col("division").asc()) fails to return reverse sorted (descending) results because it explicitly uses the asc() method, which sorts in ascending order. For reverse alphabetical sorting, asc() must be replaced with desc(). The other options correctly use syntax that achieves descending/reverse alphabetical order, using either ascending=False parameter or the desc() function.
Why the other options are wrong
- A. orderBy() with ascending=False correctly sorts in descending order.
- B. orderBy() with ascending=[0] (where 0 represents False) correctly sorts in descending order.
- D. sort() with ascending=False correctly sorts in reverse alphabetical order.
- E. sort(desc("division")) correctly applies the desc() function for descending sort.
Question 80
Which of the following code blocks returns the number of rows in DataFrame storesDF for each unique value in column division?
Show answer and explanation
Correct answer: E. storesDF.groupBy("division").count()
The groupBy method groups rows by the specified column, and count() returns the number of rows in each group. This produces a DataFrame with two columns: division and count. Option E correctly chains these methods together to get the row count per division.
Why the other options are wrong
- A. The count() function requires a column argument when used within agg, but here it's called without arguments.
- B. The groupBy method cannot be used inside agg in this manner; it must be called first on the DataFrame.
- C. The syntax is incorrect; groupby is not callable as a property and the method chaining is wrong.
- D. Calling groupBy() with no arguments doesn't group by any column, making the operation ineffective.
Question 81
Which of the following pairs of arguments cannot be used in DataFrame.join() to perform an inner join on two DataFrames, named and aliased with "a" and "b" respectively, to specify two key columns column1 and column2?
Show answer and explanation
Correct answer: B. usingColumns = Seq(col(“column1”), col(“column2”))
The usingColumns parameter accepts a Seq of strings representing column names, not a Seq of col() expressions. Option B uses col() wrapped inside Seq(), which is invalid syntax for the usingColumns parameter. The usingColumns parameter requires Seq("column1", "column2") with plain strings, not col() objects.
Why the other options are wrong
- A. joinExprs with multiple conditions combined using && (and) can correctly specify multiple key columns.
- C. Not all options are valid; Option B specifically cannot be used.
- D. joinExprs with DataFrame references and the === operator correctly specifies multiple join conditions.
- E. usingColumns with Seq("column1", "column2") is the correct string-based syntax for specifying multiple key columns.
Question 82
Which of the following operations will fail to trigger evaluation?
Show answer and explanation
Correct answer: D. DataFrame.join()
DataFrame.join() is a transformation that returns a new DataFrame without triggering evaluation. It is lazy and only defines how data should be combined. Options A, B, C, and E are all actions that immediately trigger computation and return results.
Why the other options are wrong
- A. collect() is an action that triggers full evaluation and returns all rows.
- B. count() is an action that triggers evaluation and returns the row count.
- C. first() is an action that triggers evaluation and returns the first row.
- E. take() is an action that triggers evaluation and returns the specified number of rows.
Question 83
Which of the following code blocks returns a new DataFrame with a new column customerSatisfactionAbs that is the absolute value of column customerSatisfaction in DataFrame storesDF? Note that column customerSatisfactionAbs is not in the original DataFrame storesDF.
Show answer and explanation
Correct answer: A. storesDF.withColumn(“customerSatisfactionAbs”, abs(col(“customerSatisfaction”)))
The withColumn() method creates a new DataFrame with an additional column. The correct syntax is withColumn("columnName", expression). Option A correctly uses withColumn with a string column name and the abs() function applied to a Column object returned by col(). The abs() function computes the absolute value.
Why the other options are wrong
- B. withColumnRenamed() renames existing columns, not creates new ones with computed values.
- C. Has incorrect syntax with col() wrapping a string instead of a Column expression.
- D. Missing quotes around the column name in col(); col() requires string arguments.
- E. Passes a string directly to abs() instead of a Column object from col().
Question 84
Which of the following code blocks returns a new DataFrame where column division is the first two characters of column division in DataFrame storesDF?
Show answer and explanation
Correct answer: D. storesDF.withColumn(“division”, col(“division”).substr(0, 2))
The substr() method on a Column object takes a starting position (1-indexed, not 0-indexed) and length as parameters. To get the first two characters, you call col("division").substr(1, 2), where 1 is the starting position and 2 is the length. Option D uses the correct method syntax with correct parameters for 1-indexed substring extraction.
Why the other options are wrong
- A. substr() with 0 as the starting position is incorrect; Spark uses 1-based indexing, not 0-based.
- B. susbtr is a typo and the function does not exist; the correct function is substr.
- C. substr(0, 3) uses 0-based indexing which is incorrect, and would extract 3 characters instead of 2.
- E. The 'l' character appears to be a typo for the number '1', making this option unreadable and syntactically invalid.
Question 85
Which of the following operations can be used to return the number of rows in a DataFrame?
Show answer and explanation
Correct answer: D. DataFrame.count()
The count() method is the standard PySpark DataFrame method that returns the total number of rows in a DataFrame as a long integer. It performs an action that triggers computation of the DataFrame. The other options are either non-existent methods or serve different purposes entirely.
Why the other options are wrong
- A. numberOfRows() is not a valid PySpark DataFrame method.
- B. n() is not a valid PySpark DataFrame method; this is common in other languages but not PySpark.
- C. sum() is an aggregation function used to sum numeric column values, not to count rows.
- E. countDistinct() counts distinct values within a column during aggregation, not total row count.
That was 85 of 342.
The full Databricks Associate Developer for Apache Spark pack has all 342 questions, each with the answer, the explanation and why the other options are wrong, plus a questions-only copy for timed runs. US$39, paid once, with free monthly updates and a pass-or-your-money-back guarantee.
Not ready yet? Get the first 143 free as one PDF by email. Get the Half Pack →