Apache Spark Spark SQL and DataFrames 2

Question 1

Which Spark SQL function is used to perform an inner join between two DataFrames?

Accepted Answer

df1.join(df2, 'key')

Answer

df1.join(df2, 'key') performs a join between two DataFrames; the default join type is inner.

Question 2

What is the purpose of the groupBy() function in Spark DataFrames?

Accepted Answer

Groups rows by specified columns for aggregation

Answer

groupBy() groups DataFrame rows by one or more columns and is typically followed by an aggregation function like agg(), count(), or sum().

Question 3

Which function converts a DataFrame column to a different data type in Spark SQL?

Accepted Answer

df.col('x').cast()

Answer

cast() is used to convert a column to a specified data type, e.g., col('age').cast('integer').

Question 4

What is Catalyst in Apache Spark?

Accepted Answer

The query optimizer for Spark SQL

Answer

Catalyst is Spark SQL's extensible query optimizer that transforms logical plans into optimized physical plans.

Question 5

Which Spark SQL function removes duplicate rows from a DataFrame?

Accepted Answer

dropDuplicates()

Answer

dropDuplicates() removes duplicate rows, optionally considering only a subset of columns.

Apache Spark Practice Test