Apache Spark Spark SQL and DataFrames 3

Question 1

What is a Dataset in Spark compared to a DataFrame?

Accepted Answer

A Dataset is a strongly typed, object-oriented API; a DataFrame is an alias for Dataset[Row]

Answer

Dataset is a strongly typed distributed collection available in Scala/Java, while DataFrame is Dataset[Row] with untyped column access.

Question 2

Which Spark SQL function fills null values in a DataFrame with a specified value?

Accepted Answer

df.na.fill()

Answer

df.na.fill() replaces null values with a specified value and can handle different column types.

Question 3

What does the explode() function do in Spark SQL?

Accepted Answer

Converts a column of arrays or maps into multiple rows

Answer

explode() takes a column containing an array or map and creates a new row for each element.

Question 4

Which format is recommended for reading structured data in Spark SQL when schema inference is needed?

Accepted Answer

JSON

Answer

JSON allows Spark to automatically infer the schema during read, though Parquet is preferred for performance in production.

Question 5

What is the function of withColumn() in Spark DataFrames?

Accepted Answer

Adds a new column or replaces an existing column with the given column expression

Answer

withColumn() returns a new DataFrame by adding or replacing a column using the provided expression.

Apache Spark Practice Test