Apache Spark Spark SQL and DataFrames 3

Question 1

What is a Dataset in Spark compared to a DataFrame?

Accepted Answer

A Dataset is a strongly typed, object-oriented API; a DataFrame is an alias for Dataset[Row]

Answer

A Dataset is an untyped distributed collection; a DataFrame is strongly typed

Answer

A Dataset is only available in Python; a DataFrame is for Scala and Java

Answer

A Dataset supports only batch processing; a DataFrame supports streaming

Question 2

Which Spark SQL function fills null values in a DataFrame with a specified value?

Accepted Answer

df.na.fill()

Answer

df.replaceNull()

Answer

df.fillNull()

Answer

df.impute()

Question 3

What does the explode() function do in Spark SQL?

Accepted Answer

Converts a column of arrays or maps into multiple rows

Answer

Raises an exception if data is malformed

Answer

Splits a string column into multiple columns

Answer

Increases the number of partitions in the DataFrame

Question 4

Which format is recommended for reading structured data in Spark SQL when schema inference is needed?

Accepted Answer

JSON

Answer

CSV

Answer

Parquet

Answer

ORC

Question 5

What is the function of withColumn() in Spark DataFrames?

Accepted Answer

Adds a new column or replaces an existing column with the given column expression

Answer

Selects only the specified column

Answer

Renames a column

Answer

Removes a column from the DataFrame

Question 6

What does the repartition() function do to a Spark DataFrame?

Accepted Answer

Increases or decreases the number of partitions by performing a full shuffle

Answer

Reduces partitions without shuffling

Answer

Saves the DataFrame with custom partition keys

Answer

Merges small files into larger ones