Apache Spark Spark SQL and DataFrames 2 — Questions and Answers
Question 1: Which Spark SQL function is used to perform an inner join between two DataFrames?
- df1.merge(df2, on='key')
- df1.join(df2, 'key') (Correct answer)
- df1.combine(df2, 'key')
- df1.link(df2, 'key')
Correct answer: df1.join(df2, 'key')
df1.join(df2, 'key') performs a join between two DataFrames; the default join type is inner.
Question 2: What is the purpose of the groupBy() function in Spark DataFrames?
- Sorts the DataFrame by specified columns
- Groups rows by specified columns for aggregation (Correct answer)
- Filters rows that belong to the same group
- Partitions the DataFrame by specified columns
Correct answer: Groups rows by specified columns for aggregation
groupBy() groups DataFrame rows by one or more columns and is typically followed by an aggregation function like agg(), count(), or sum().
Question 3: Which function converts a DataFrame column to a different data type in Spark SQL?
- df.col('x').convert()
- df.col('x').cast() (Correct answer)
- df.col('x').asType()
- df.col('x').transform()
Correct answer: df.col('x').cast()
cast() is used to convert a column to a specified data type, e.g., col('age').cast('integer').
Question 4: What is Catalyst in Apache Spark?
- A machine learning library
- A streaming engine
- The query optimizer for Spark SQL (Correct answer)
- A storage manager for DataFrames
Correct answer: The query optimizer for Spark SQL
Catalyst is Spark SQL's extensible query optimizer that transforms logical plans into optimized physical plans.
Question 5: Which Spark SQL function removes duplicate rows from a DataFrame?
- deduplicate()
- dropDuplicates() (Correct answer)
- unique()
- removeDups()
Correct answer: dropDuplicates()
dropDuplicates() removes duplicate rows, optionally considering only a subset of columns.
Question 6: How do you write a Spark DataFrame to a Parquet file?
- df.export('path', format='parquet')
- df.write.parquet('path') (Correct answer)
- df.save('path', 'parquet')
- df.to_parquet('path')
Correct answer: df.write.parquet('path')
df.write.parquet('path') saves the DataFrame to the specified path in Parquet format.
Apache Spark Spark SQL and DataFrames 2
Which Spark SQL function is used to perform an inner join between two DataFrames?