Apache Spark Spark SQL and DataFrames 1 — Questions and Answers
Question 1: What is the entry point for Spark SQL functionality in Spark 2.x and later?
- SparkContext
- SQLContext
- SparkSession (Correct answer)
- HiveContext
Correct answer: SparkSession
SparkSession is the unified entry point for Spark SQL, DataFrame, and Dataset APIs introduced in Spark 2.0.
Question 2: Which method is used to run a SQL query on a registered temporary view in Spark SQL?
- spark.execute()
- spark.sql() (Correct answer)
- spark.query()
- spark.runSQL()
Correct answer: spark.sql()
spark.sql() executes a SQL query and returns the results as a DataFrame.
Question 3: What is a DataFrame in Apache Spark?
- An RDD of Row objects with a named column schema (Correct answer)
- A distributed key-value store
- An immutable list of serialized Java objects
- A streaming data source
Correct answer: An RDD of Row objects with a named column schema
A DataFrame is a distributed collection of data organized into named columns, conceptually equivalent to a database table.
Question 4: How do you register a DataFrame as a temporary SQL view in Spark?
- df.registerTable("name")
- df.createOrReplaceTempView("name") (Correct answer)
- df.toTable("name")
- spark.registerView(df, "name")
Correct answer: df.createOrReplaceTempView("name")
createOrReplaceTempView() registers a DataFrame as a temporary view scoped to the current SparkSession.
Question 5: Which DataFrame function is used to select specific columns?
- filter()
- where()
- select() (Correct answer)
- pick()
Correct answer: select()
select() returns a new DataFrame with only the specified columns.
Question 6: What does the printSchema() method do in Spark DataFrames?
- Prints all rows of the DataFrame
- Displays the first 20 rows
- Prints the schema of the DataFrame in a tree format (Correct answer)
- Exports the schema to a file
Correct answer: Prints the schema of the DataFrame in a tree format
printSchema() prints the schema of the DataFrame as a tree showing column names, types, and nullability.
Apache Spark Spark SQL and DataFrames 1
What is the entry point for Spark SQL functionality in Spark 2.x and later?