Apache Spark Spark SQL and DataFrames 2

Question 1

Which Spark SQL function is used to perform an inner join between two DataFrames?

Accepted Answer

df1.join(df2, 'key')

Answer

df1.merge(df2, on='key')

Answer

df1.combine(df2, 'key')

Answer

df1.link(df2, 'key')

Question 2

What is the purpose of the groupBy() function in Spark DataFrames?

Accepted Answer

Groups rows by specified columns for aggregation

Answer

Sorts the DataFrame by specified columns

Answer

Filters rows that belong to the same group

Answer

Partitions the DataFrame by specified columns

Question 3

Which function converts a DataFrame column to a different data type in Spark SQL?

Accepted Answer

df.col('x').cast()

Answer

df.col('x').convert()

Answer

df.col('x').asType()

Answer

df.col('x').transform()

Question 4

What is Catalyst in Apache Spark?

Accepted Answer

The query optimizer for Spark SQL

Answer

A machine learning library

Answer

A streaming engine

Answer

A storage manager for DataFrames

Question 5

Which Spark SQL function removes duplicate rows from a DataFrame?

Accepted Answer

dropDuplicates()

Answer

deduplicate()

Answer

unique()

Answer

removeDups()

Question 6

How do you write a Spark DataFrame to a Parquet file?

Accepted Answer

df.write.parquet('path')

Answer

df.export('path', format='parquet')

Answer

df.save('path', 'parquet')

Answer

df.to_parquet('path')